CosyVoice: Alibaba's open-source multilingual zero-shot speech synthesis
In one sentence Alibaba's Speech Lab publishes CosyVoice, zero-shot speech synthesis in Chinese, English, Japanese, Korean and Cantonese: an LLM generates semantic speech tokens, a flow matching model turns them into audio. 300M models and code are open source under Apache 2.0, as part of the FunAudioLLM family.
CosyVoice is a text-to-speech system developed by Alibaba's Speech Lab and released as open source in early July 2024, together with the SenseVoice speech recognition model, under the shared name FunAudioLLM.
It does "zero-shot" voice cloning: given a short audio sample of a person, it can speak new text in that voice without specific training. The languages covered are Mandarin Chinese, English, Japanese, Korean and Cantonese.
The architecture has two stages. A language model turns text into discrete speech "tokens"; a second model, based on flow matching, converts those tokens into the spectrogram from which the audio is produced. The paper's novelty lies in the tokens: instead of learning them without supervision, it derives them from a speech recognition model, so they stay tied to the spoken content. According to the authors this improves both content accuracy and similarity to the cloned voice.
Besides the base model there is an "instruct" variant that accepts instructions on speaker identity, style and emotions.
Companies
Alibaba
Tools
—
Tags
Sources