Follow-Your-Emoji
Fine-Controllable and Expressive Freestyle Portrait Animation

Yue Ma*, Hongyu Liu*, Hongfa Wang*, Heng Pan*, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu and Qifeng Chen

is Accpeted by Siggraph Asia 2024

📣 Updates

• [2025.04.16] 🔥 We propose Taylor-Interpolated Cache (TIC) for faster generation up to 2.4× lossless acceleration, codes are released!
• [2025.03.21] 🔥 If you want to generate longer length, please use the forward_long function!
• [2024.07.31] 🔥 Release OpenXLab, thanks for keyhsw development!
• [2024.07.21] 🔥 Release Colab, thanks for daswer123!
• [2024.07.18] 🔥 Release inference code, config and checkpoints!
• [2024.06.07] 🔥 Release Paper and Project page!

🤪 Gallery

We present Follow-Your-Emoji, a diffusion-based framework for portrait animation, which animates a reference portrait with target landmark sequences.

🤪 Getting Started

1. Clone the code and prepare the environment

pip install -r requirements.txt

2. Download pretrained weights

[FollowYourEmoji] We also provide our pretrained checkpoints in Huggingface. you could download them and put them into checkpoints folder to inference our model.

• base model: lambdalabs/sd-image-variations-diffusers
• vae: sd-vae-ft-mse
• AnimateDiff: AnimateDiff

Finally, these pretrained models should be organized as follows:

pretrained_models
├── AnimateDiff
│   └── mm_sd_v15_v2.ckpt
├── follow-your-emoji
│   ├── lmk_guider.pth
│   ├── referencenet.pth
│   └── unet.pth
├── sd-image-variations-diffusers
│   ├── alias-montage.jpg
│   ├── default-montage.jpg
│   ├── earring.jpg
│   ├── feature_extractor
│   │   └── preprocessor_config.json
│   ├── image_encoder
│   │   ├── config.json
│   │   └── pytorch_model.bin
│   ├── inputs.jpg
│   ├── model_index.json
│   ├── README.md
│   ├── safety_checker
│   │   ├── config.json
│   │   └── pytorch_model.bin
│   ├── scheduler
│   │   └── scheduler_config.json
│   ├── unet
│   │   ├── config.json
│   │   └── diffusion_pytorch_model.bin
│   ├── v1-montage.jpg
│   ├── v2-montage.jpg
│   └── vae
│       ├── config.json
│       └── diffusion_pytorch_model.bin
└── sd-vae-ft-mse
    ├── config.json
    ├── diffusion_pytorch_model.bin
    ├── diffusion_pytorch_model.safetensors
    └── README.md

3. Inference 🚀

bash infer.sh

CUDA_VISIBLE_DEVICES=0 python3 -m torch.distributed.run \
    --nnodes 1 \
    --master_addr $LOCAL_IP \
    --master_port 12345 \
    --node_rank 0 \
    --nproc_per_node 1 \
    infer.py \
    --config ./configs/infer.yaml \
    --model_path /path/to/model \
    --input_path your_own_images_path \
    --lmk_path ./inference_temple/test_temple.npy  \
    --output_path your_own_output_path

4. Efficient Inference with TIC🚀

We present Taylor-Interpolated Cache, a cache based accelerating method that can accelerate original pipeline losslessly. It is compatible with existing pipeline and very easy to use! We have deployed TIC on 4090(24G), 32 frames per case and 512 × 512 resolution.

bash infer.sh

CUDA_VISIBLE_DEVICES=0 python3 -m torch.distributed.run \
    --nnodes 1 \
    --master_addr $LOCAL_IP \
    --master_port 12345 \
    --node_rank 0 \
    --nproc_per_node 1 \
    infer.py \
    --config ./configs/infer.yaml \
    --model_path /path/to/model \
    --input_path your_own_images_path \
    --lmk_path ./inference_temple/test_temple.npy  \
    --output_path your_own_output_path \
    --TIC

TIC is developed based on codebase from DeepCache and Taylor Seer. Thanks to their exellent work!

🤪 Make Your Emoji

You can make your own emoji using our model. First, you need to make your emoji temple using MediaPipe. We provide the script in make_temple.ipynb. You can replace the video path with your own emoji video and generate the .npy file.

CUDA_VISIBLE_DEVICES=0 python3 -m torch.distributed.run \
    --nnodes 1 \
    --master_addr $LOCAL_IP \
    --master_port 12345 \
    --node_rank 0 \
    --nproc_per_node 1 \
    infer.py \
    --config ./configs/infer.yaml \
    --model_path /path/to/model \
    --input_path your_own_images_path \
    --lmk_path  your_own_temple_path \
    --output_path your_own_output_path

👨‍👩‍👧‍👦 Follow Family

Follow-Your-Pose: Pose-Guided text-to-Video Generation.

Follow-Your-Click: Open-domain Regional image animation via Short Prompts.

Follow-Your-Handle: Controllable Video Editing via Control Handle Transformations.

Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation.

Citation 💖

If you find Follow-Your-Emoji useful for your research, welcome to 🌟 this repo and cite our work using the following BibTeX:

@article{ma2024follow,
  title={Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation},
  author={Ma, Yue and Liu, Hongyu and Wang, Hongfa and Pan, Heng and He, Yingqing and Yuan, Junkun and Zeng, Ailing and Cai, Chengfei and Shum, Heung-Yeung and Liu, Wei and others},
  journal={arXiv preprint arXiv:2406.01900},
  year={2024}
}