InternVL: a 6-billion-parameter vision encoder aligned with LLMs
In one sentence The InternVL paper (arXiv, 21 December 2023) scales the vision foundation model to 6B parameters with InternViT-6B and progressively aligns it with an LLM, claiming state-of-the-art results on 32 visual-linguistic benchmarks.
In multimodal models the language side had grown to tens of billions of parameters, while the image encoder stayed much smaller. InternVL tries to rebalance: it scales the vision encoder, InternViT-6B, to 6 billion parameters.
The encoder is then aligned step by step with a language model, using billions of image-text pairs from the web. The result can do image and video classification, image-text retrieval and, connected to an LLM, multimodal dialogue.
The work was accepted as an oral at CVPR 2024 and is the basis of the InternVL family that followed (1.5, 2, 2.5).
Companies
Shanghai AI Laboratory
Tools
InternVL, InternViT-6B
Tags
Sources