SoundStorm: Google generates 30 seconds of audio in half a second
In one sentence SoundStorm generates SoundStream codec tokens in parallel from AudioLM semantic tokens: 30 seconds of audio in about 0.5 seconds on a TPU-v4, with AudioLM-level quality and more consistent voice.
Models like AudioLM generate audio tokens one at a time, left to right: for long clips this makes them slow. SoundStorm instead uses a parallel approach inspired by MaskGIT: it starts with all audio tokens masked and fills them in over a few iterations, many at a time. According to the paper it generates 30 seconds of audio in about half a second on a TPU-v4, roughly a hundred times faster than AudioLM's acoustic stage, with equivalent quality and more consistent speaker voice. Combined with SPEAR-TTS it also produces multi-voice dialogue from an annotated transcript and short voice samples.
Companies
Tools
SoundStorm, SoundStream, AudioLM, SPEAR-TTS
Tags
Sources