Unified-IO (AI2): one sequence-to-sequence model for vision, language and multimodal tasks
In one sentence On June 17, 2022 AI2 and the University of Washington present Unified-IO: a single encoder-decoder Transformer that converts the inputs and outputs of vision, language and multimodal tasks into sequences of discrete tokens, trained jointly on over 90 datasets. It is the first model to perform all 7 tasks of the GRIT benchmark.
Usually each computer vision job needs a different model: one to find objects, one to estimate depth, one to describe an image, one to answer questions. Each has its own input and output format.
Unified-IO tries to do it all with a single model. The trick is turning everything into a sequence of "words" from a shared vocabulary: text stays text, images and maps (depth, masks) become sequences of visual tokens, boxes around objects become coordinates written as tokens. Then a single Transformer learns to read and produce all these sequences.
Trained jointly on over 90 datasets, the model handles very different tasks, from object detection to image generation to text question answering, without task-specific adaptation. Audio and video arrive only with its successor Unified-IO 2, in late 2023.
Companies
AI2, University of Washington
Tools
—
Tags
Sources