VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

Puyuan Peng1, Po-Yao Huang2, Abdelrahman Mohamed3, David Harwath1,
1The University of Texas at Austin 2FAIR 3Rembrand
Paper Code&Model

TL;DR

VoiceCraft is a token infilling neural codec language model, that achieves state-of-the-art performance on both speech editing and zero-shot text-to-speech (TTS) on in-the-wild data including audiobooks, internet videos, and podcasts.

To clone an unseen voice or edit a recording, VoiceCraft needs only a few seconds of the voice.

Speech Editing with VoiceCraft

Interpolate start reference image.

Table of Contents

Guess which part is synthesized!

Edited Transcript VoiceCraft Edited Audio
what do I really wanna do, I wanna talk to the audience, which is always the best part of my day
Cause a year from now, when you kickin' it in the Caribbean as an independent researcher, you gonna say to yourself, "my advisor was right"
The path of the righteous man is beset on all sides by the inequities of the selfish and the tyranny of evil men. Yet, he walks with unwavering resolve, guided by a compass of moral clarity and a heart unyielding in the face of adversity.
I saw it, I saw it, and it was beautiful. Who said that I didn't see it? Did Jim say that I didn't see it? He just envies me, I saw it!
I don't like to write down things that I don't mean, and when I was writing wrecking ball I really poured my heart into it
Go Back to Table of Contents

Speech Editing: VoiceCraft v.s. FluentSpeech

All speakers are unseen during training. Utterances are from our RealEdit evaluation set, comprises audiobooks, YouTube videos, and Spotify podcasts

Original Transcript Original Recording Target Transcript: VoiceCraft (Ours) FluentSpeech
Fast cars, that had the nice clothes, that had the money, they was criminals. Fast cars, that had the nice clothes, that had expensive gold watches, that had the money, they was criminals.
We would just be open and willing to adopt whatever child God brought to her life. We would just be excited to welcome whatever child God brought into her life.
that's a bomb and that's a good sign from him. he got fully extended on it. knew it as soon as that ball left. that's a bomb and that's a good sign from him. he clearly signalled and made the play happen as soon as that ball left.
economic development remains one of the most effective ways to increase the capacity to adapt to climate change. economic development remains one of the most promising options that we have left on the table to increase the capacity to adapt to climate change.
So if you've been following my story, you will remember that I said earlier in this podcast that the Grammy nominations came out. So if you've been following my story, you will remember that I said earlier that this week we had super exciting stuff to talk about because Grammy nominations came out.
because we can include so many other characters if we just expand the definitions to any sword wielder, who's a little spicy. because we can include so many other participants if we are brave enough to expand the definitions to any blade wielder, who's a little spicy.
It was a glance of inquiry, ending in a look of chagrin, with some muttered phrases that rendered it more emphatic. It was a look of disgust followed by a curled lip, with some muttered phrases that rendered it more emphatic.
some times i really feel that the world around us continues to be more hectic and more complicated and so many of us are truly craving to find simplicity. some times i really feel that the world around us continues to be more hectic, more impersonal, and more uncaring and so many of us are truly craving to find simplicity.
For making the title though because I need to get my numbers way up before I get there, but I'm gonna get there title of Iceland is definitely going to sign me and um, yeah. For making the title though because I need to get my numbers way up before I get there, but I'm gonna get there title of Iceland is going to sign me and um, yeah.
they knew that governments don't control things. a government can't control the economy without controlling people. they knew that governments don't control money directly. a government can't control the economy without controlling people.
No words just lightning breaking darkness and crashing into the Earth with brilliant presence. No words just lightning breaking darkness and crashing into the surface of the Earth with brilliant presence.
that schedule is one per week and it will probably be like a Wednesday night thing because I plan on doing one to two videos per week. that schedule is one per week and you will start to see a lot more content arriving because I plan on doing one to two videos per week.
in a case like this, i probably wouldn't spend any more time looking at the deal if i was only interested in the cash flow. in a case like this, i probably wouldn't spend any more time looking at all the details and the fine print if i was only interested in the cash flow.
but the renaissance broke their monopoly on knowledge, one of the most important bastions of the church. but the renaissance broke their monopoly on knowledge, with it's free movement of research and endless scientific inquiry, one of the most important bastions of the church.
Go Back to Table of Contents

Speech Editing: VoiceCraft v.s. Voicebox

The original recordings and Voicebox's output are taken from the official demo page of Voicebox at here.

Original Transcript Original Recording Target Transcript: VoiceCraft (Ours) Voicebox
will find himself completely at a loss on occasions of common and constant recurrence speculative ability is one thing and practical ability is another will find himself completely at a loss on rare and unpredictable circumstances speculative ability is one thing and practical ability is another
in zero weather in mid-winter when the earth is frozen to a great depth below the surface when in driving over the unpaved country roads they give forth a hard metallic ring in zero weather in mid-winter when jack frost has cast his icy spell upon the land when in driving over the unpaved country roads they give forth a hard metallic ring
and especially as i am not very much up in latin myself he said the suit was on an insurance policy that he was defending on the ground of misinterpretations and especially as i am not very much up in latin myself he said the suit was on a classified treasure map that he was defending on the ground of misinterpretations
yet these petty operations incessantly continued in time surmount the greatest difficulties mountains are elevated and oceans bounded by the slender force of human beings yet these petty operations incessantly continued in time surmount the greatest difficulties vast challenges emerge and unexplored frontiers beckon by the slender force of human beings
and the carlsruhe professor had to devise an ingenious apparatus which enabled him to bring the preparation at the required temperature on to the very plate of the microscope and the inventive professor had to devise an ingenious apparatus which enabled him to bring the preparation at the required temperature on to the very plate of the microscope
this was george steers the son of a british naval captain and ship modeler who had become an american naval officer and was the first man to take charge of the washington navy yard this was george steers the son of a british naval captain and ship modeler who had become an american naval officer and was entrusted with the prestigious role of overseeing the operations at the renowned naval headquarters
Go Back to Table of Contents

Zero-Shot TTS: VoiceCraft v.s. Prior SotA

All speakers are unseen during training. Only the first 3 seconds of the Voice Prompt are given to the models

Traget Transcript Voice Prompt (only the first 3 seconds is used) VoiceCraft (Ours) VALLE XTTS v2 FluentSpeech YourTTS Ground Truth
i don't wanna make it as a thing where i'm absolving myself of any responsibility.
and the only way to do great work is to love what you do. if you haven't found it yet, keep looking.
i'm pleading the fifth about my shoeless state at the moment. but what if you learned about yourselves in this time and and what has changed about the way that you're working?
we voted it out that was a standard that people used till eleven or little bit past eleven um and it was an international standard.
hey you all, my name is corey ash and, and i know that you have been working really hard to try to figure out
Do you know I was so foolish that I thought every flash of lightning must descend on your head.
Quox did not have much to say until the conversation was ended, but then he turned to Kaliko and asked:
you know it's one thing to kind of be home alone but it's another to know that you have, you know zoom, yoga classes with friend of yours.
at its base, raider one is a four wheel drive polaris, MVRS 700, that controllers can operate nearly one thousand yards away.
We clambered up to the front seat and jolted off past the little pond and along the road that climbed to the big cornfield.
We were the only ones who did go afoot, however, although the corrals were not more than two hundred yards' distant.
blank space because i'd written a lot of lines down already in the year preceding the session.
Go Back to Table of Contents

BibTeX

@article{peng2024voicecraft,
  author    = {Peng, Puyuan and Huang, Po-Yao and Mohamed, Abdelrahman and Harwath, David},
  title     = {VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild},
  journal   = {arXiv},
  year      = {2024},
}