VALL-E: Microsoft clones a voice from 3 seconds of audio using in-context learning
In one sentence VALL-E imitates an unseen speaker's voice from 3 seconds of audio, without fine-tuning, treating speech synthesis as language modeling over EnCodec tokens. It beats the previous zero-shot state of the art in naturalness and speaker similarity.
VALL-E is a speech synthesis system that works a bit like a language model, but for voices: you play it 3 seconds of someone it never heard in training, and it reads any text you want in that voice. No retraining and no long recordings: those 3 seconds act as an "audio prompt". The trick is treating voice as sequences of numeric codes (tokens from the EnCodec audio codec) and training the model on about 60,000 hours of English speech. According to the authors, the output also keeps the emotion and acoustic environment of the source recording.
Companies
Microsoft
Tools
VALL-E, EnCodec
Tags
Sources