Speech synthesis, the use of computers to generate realistic human speech, is rapidly entering the ‘uncanny valley’ – creepily almost-realistic.
Recent approaches have used neural networks, trained using only speech examples and text transcripts, to generate human-like text-to-speech synthesis. In the example embedded above, the late Christopher George Latore Wallace (May 21, 1972 – March 9, 1997), aka The Notorious B.I.G., raps The Book of Genesis.
The ‘voice’ voice was computer-generated, using a text-to-speech model trained on the speech patterns of The Notorious B.I.G. In a nutshell, the approach uses an AI to ‘learn’ how an audio file of an individual’s speech compares to a text transcript. Once trained, the model can synthesize speech from text that conforms to the ‘learned’ speech patterns.
The Vocal Synthesis channel on Youtube features a wide range of examples that demonstrate what’s currently possible. The introduction to the channel features the voices of six presidents:
Bill Clinton reads Baby Got Back by Sir Mix-A-Lot, embedded below, is an example that highlights both the entertaining and troubling potential of the technology:
“In an evaluation where we asked human listeners to rate the naturalness of the generated speech, we obtained a score that was comparable to that of professional recordings.
While our samples sound great, there are still some difficult problems to be tackled. For example, our system has difficulties pronouncing complex words (such as “decorum” and “merlot”), and in extreme cases it can even randomly generate strange noises. Also, our system cannot yet generate audio in realtime. Furthermore, we cannot yet control the generated speech, such as directing it to sound happy or sad. Each of these is an interesting research problem on its own.”
What Will Biggie Rap Next?
Biggie has been gone for more than two decades, but this technology is being used to make him rap from beyond the grave. The results are realistic enough that artists and agents are taking notice.
For example, in April 2020, Youtube temporarily took down this video of Jay-Z rapping the To Be, Or Not To Be soliloquy from Hamlet:
Jay-Z’s agency, Roc Nation LLC, claimed that the video “unlawfully uses an AI to impersonate our client’s voice.”
The legality of unauthorized speech synthesis is still to be determined, and could depend on how the audio is used. Some uses may fall into the category of fair use and others may infringe on copyright or an artist’s publicity rights.
The anonymous creator of the Vocal Synthesis channel recognizes that the channel is in ‘legally uncharted waters’, but suggests that it is important to explore the creative possibilities of AI-generated speech:
“I wanted to show that synthetic media doesn’t have to be exclusively made for malicious/evil purposes, and I think there’s currently massive amounts of untapped potential in terms of fun/entertaining uses of the technology.
I think the scariness of deepfakes and synthetic media is being overblown by the media, and I’m not at all convinced that the net impact will be negative, so I hoped that my channel could be a counterexample to that narrative.”
Check out the examples and share your thoughts on AI speech synthesis, and its creative possibilities, in the comments!
10 thoughts on “Speech Synthesis Enters The Uncanny Valley, Or ‘What Will Biggie Rap Next?’”
If it can make Justin Bieber sing Tool I’d buy that!
Wow, and I already had a hard time believing anything I hear!
“Enters the uncanny valley”? This skipped right past. Sure, the Jay-Z sibilance was too consistent but I don’t many people would get an “uncanny valley” felling if they heard this and didn’t know.
Are we sure the Clinton one isn’t real? Seems on brand for him,
Just what society needs.
Wow, the BIG “I Am The Very Model of a Modern Major General” is literally moving.
Wow that’s amazing. They can do so much these days with our advanced technology.
Interesting, I was always wondering where all those “unleased” 2 pac and Biggie tapes came from..
My Father, God rest his soul, predicted this technology to me about 50 years ago. I have to question the legalities though. Do you have a RIGHT to impersonate someone so closely that is is virtually impossible to differentiate the real vs the synthesized? I think this will be a new subject matter for the patent and trademarks office. Just watch!
RE: Jay-Z/WS video… Humanity has been trying to modernize Shakespeare since, well, Shakespeare + 1 day. This “deep fake” does a better job of bringing Billie’s words into modern times than any performance I’ve ever seen. I’ve seen many and been a part of a few. Maybe not as artistic as some but very much more to the point than any I’m aware of.
Most interesting to me at the moment: were Jay tasked with interpreting+spitting Shakespeare verses, he’d prolly not do as good a job as the AI did. And I have deep respect for him as a rapper.