Molmo: Ai2's open VLMs that point at objects
In one sentence On September 25, 2024 Ai2 releases Molmo, a family of open vision-language models (1B, 7B, 72B) that can point at precise locations in images. The 72B beats Claude 3.5 Sonnet and Gemini 1.5 Pro on academic benchmarks and ranks second only to GPT-4o in human evaluation.
Molmo is a family of vision-language models from the Allen Institute for AI (Ai2). Besides describing an image, Molmo can show where something is: it answers with the coordinates of a point, can count objects by pointing at them one by one, and can use the point as a visual explanation of its answer.
The training data (PixMo) was collected in an unusual way: annotators described images by speaking for 60-90 seconds instead of typing, and the transcripts were then refined with language models. No external VLM was used to generate the data.
At launch Ai2 published four models, inference code, a technical report and a demo; the PixMo datasets and training code were announced for a second phase.
Companies
Allen Institute for AI
Tools
Molmo, MolmoE-1B, Molmo-7B-O, Molmo-7B-D, Molmo-72B, PixMo
Tags
Sources