VITS: end-to-end TTS with variational autoencoder
In one sentence VITS merges acoustic model and vocoder into a single end-to-end model: on LJSpeech it reaches a MOS of 4.43, close to real recordings (4.46) and above Tacotron 2 + HiFi-GAN (3.77), with synthesis about 2.4x faster than Glow-TTS + HiFi-GAN.
Before VITS, a quality speech synthesis system was made of two separate pieces: one turned text into an intermediate representation (the spectrogram), a second one (the vocoder) converted it into audio. It is like having a translator who writes the translation on paper and another person who then reads it aloud.
VITS puts everything into a single model: text goes in, audio comes out directly. The model is trained in one go, end to end.
The technical core is a conditional variational autoencoder, combined with GAN-style "adversarial" training. A stochastic duration predictor lets the same sentence be read with different rhythms, as a person would.
In the authors' listening test on an English dataset (LJSpeech), VITS voices scored almost as high as real recordings. The code was released on GitHub under the MIT license.
Companies
Kakao Enterprise
Tools
VITS
Tags
Sources