OpenVLA: an open-source 7B Vision-Language-Action model for robotics
In one sentence On June 13, 2024 researchers from Stanford, UC Berkeley, Toyota Research Institute, Google DeepMind, Physical Intelligence and MIT publish OpenVLA: a 7B model trained on 970,000 robot demonstrations that beats the closed RT-2-X (55B) by 16.5% in absolute success rate across 29 tasks.
OpenVLA is a model that takes a robot's camera image and a natural-language instruction and outputs the actions to perform. The best-known models of this kind, such as RT-2-X, could not be downloaded: OpenVLA releases checkpoints, fine-tuning notebooks and training code.
It has 7 billion parameters and is trained on about 970,000 real-robot demonstrations from the Open X-Embodiment dataset, which spans many tasks, scenes and robot types.
According to the authors, OpenVLA outperforms RT-2-X, a closed 55-billion-parameter model, while being 7 times smaller. It can be adapted to a new robot even on consumer GPUs with techniques such as LoRA, and it runs quantized without losing success rate.
Companies
Stanford University, UC Berkeley, Toyota Research Institute, Google DeepMind, Physical Intelligence, MIT
Tools
OpenVLA, Prismatic VLM, Llama 2
Tags
Sources