F5-TTS: zero-shot speech synthesis with flow matching, open code and models
In one sentence A Shanghai Jiao Tong University team releases F5-TTS: a non-autoregressive TTS based on flow matching and a Diffusion Transformer, with no duration model or phoneme alignment. Trained on 100K hours, RTF 0.15; MIT code, CC-BY-NC weights.
F5-TTS is a speech synthesis system that, given a short reference audio and a text, generates the sentence in the reference voice (zero-shot), with no extra training on that voice. Compared with many earlier TTS systems it is simpler: it has no separate model for sound duration and no alignment between phonemes and audio. The text is padded with filler tokens up to the speech length and the model generates audio by "denoising". The authors released code and checkpoints: the code is MIT-licensed, the pre-trained models are CC-BY-NC (so not for commercial use) because of the Emilia dataset used in training.
Companies
Shanghai Jiao Tong University, Geely
Tools
F5-TTS, E2 TTS
Tags
Sources