AnimateDiff: bringing motion to personalized Stable Diffusion models
In one sentence Shanghai AI Lab, CUHK and Stanford present AnimateDiff: a motion module trained once on video and plugged into personalized models derived from Stable Diffusion (DreamBooth, LoRA), which can then generate short animations without model-specific tuning.
In 2023 the Stable Diffusion ecosystem is full of personalized models, downloaded from CivitAI or Hugging Face or trained by users with DreamBooth and LoRA: they produce images in the desired style, but only still images.
AnimateDiff proposes an extra component, the motion module, inserted into the Stable Diffusion network while leaving the original weights untouched. The module is trained once on video clips and learns motion; it can then be plugged into any personalized model derived from the same base.
The result: an anime-style checkpoint or a photorealistic one can generate short animated clips while keeping their style, without dedicated training for each.
Companies
Shanghai AI Lab, CUHK, Stanford
Tools
—
Tags
Sources