LLaVA-NeXT-Video: an image-trained model that also understands video
In one sentence On April 30, 2024 the LLaVA team (ByteDance, UW-Madison, NTU) publishes LLaVA-NeXT-Video: LLaVA-NeXT, trained only on images, shows zero-shot video understanding; with video fine-tuning and DPO from LLM feedback, the 7B and 34B models beat the video-specific open LMMs they are compared with.
LLaVA-NeXT is an open model that answers questions about images. The team behind it finds that, although it was trained only on text-image pairs, it understands video frames surprisingly well when given them: "zero-shot modality transfer".
Building on this, the team releases LLaVA-NeXT-Video: the same model, then fine-tuned on video data and optimized with preferences generated by another language model. It answers questions about video clips and describes them in detail, in 7- and 34-billion-parameter versions.
Companies
ByteDance, University of Wisconsin-Madison, Nanyang Technological University
Tools
LLaVA-NeXT-Video
Tags
Sources