MiniGPT-4 (KAUST): open-source visual chatbot with a single projection layer
In one sentence KAUST researchers connect a frozen BLIP-2-style visual encoder to the Vicuna language model, also frozen, training only a projection layer. The result describes images in detail and generates a website from a hand-drawn sketch, after a few hours of training.
After GPT-4 showed it could understand images, many wondered where that ability came from. On April 20, 2023 a group from King Abdullah University of Science and Technology (KAUST), in Saudi Arabia, publishes MiniGPT-4 to test a hypothesis: most of the credit goes to the language model, not the visual part.
The experiment is simple: they take an already trained visual encoder (BLIP-2's) and an already trained language model (Vicuna), keep both frozen, and train only a projection layer that translates the visual output into the model's language.
With this minimal link, MiniGPT-4 describes images in detail, writes stories and poems inspired by a photo, explains how to cook a pictured dish and generates website code from a hand-drawn sketch. It was not the first work of this kind (LLaVA had come out a few days earlier), but it showed how little training was needed to get a usable visual chatbot.
Companies
KAUST
Tools
MiniGPT-4
Tags
Sources