AudioPaLM: Google fuses PaLM-2 and AudioLM into a model that listens and speaks
In one sentence Google presents AudioPaLM, which merges the PaLM-2 text model and the AudioLM speech model into a shared vocabulary of text and audio tokens: state-of-the-art speech translation results and speech-to-speech translation that keeps the speaker's voice.
Language models are excellent with text, but speech carries more: the voice, the intonation, the identity of the speaker. The usual approach chains separate systems (speech recognition, translation, synthesis), and every step loses information.
Google's AudioPaLM takes PaLM-2, the company's large text model, and AudioLM, a model that generates speech, and fuses them into a single model. Speech is turned into discrete audio tokens added to the text vocabulary: for the model, text and audio become sequences of the same kind.
This lets AudioPaLM transcribe, translate speech into text or directly into speech in another language, keeping the speaker's voice from a short sample. The model is not released: Google publishes the paper and some audio examples.
Companies
Tools
—
Tags
Sources