CogVLM: a visual expert inside every layer to keep language skills intact
In one sentence The THUDM group (Tsinghua / Zhipu AI) releases CogVLM-17B: 10 billion visual plus 7 billion language parameters, with a visual expert module in the attention and FFN layers and SOTA results on 10 classic cross-modal benchmarks.
CogVLM is a vision-language model from the THUDM group (Tsinghua University and Zhipu AI). It tackles a common problem in multimodal models: when image understanding is added, the language model's text abilities often degrade.
CogVLM's idea is to add parameters dedicated to images, a "visual expert", while keeping the original language model frozen. According to the authors, the model learns to reason about images without sacrificing performance on text-only tasks.
Companies
Tsinghua University, Zhipu AI
Tools
CogVLM
Tags
Sources