LLaVA-1.5: open-source vision-language, SOTA on 11 benchmarks with 1.2M public data points
In one sentence LLaVA-1.5 combines CLIP ViT-L/14 at 336px, an MLP projector and Vicuna v1.5: state-of-the-art results on 11 benchmarks with about 1.2 million public data points and one day of training on a single 8-A100 node.
Teaching a language model to "see" is complicated: you need to connect the visual world with the textual world so the model can answer questions about images, describe scenes, or read text in photographs.
LLaVA-1.5 does this in a surprisingly economical way: it takes a pre-trained visual encoder (CLIP), connects it to a text LLM (Vicuna) through a small projection layer, and trains the whole thing with about 1.2 million public examples, in roughly one day on a single node with 8 A100 GPUs. No enormous datasets, no complex architectures.
The result is an open-source model with state-of-the-art results on 11 benchmarks. It has become a reference baseline for research in this field.
Companies
University of Wisconsin-Madison, Microsoft Research
Tools
LLaVA-1.5, CLIP ViT-L/14-336px, Vicuna v1.5
Tags
Sources