快速开始
环境要求
安装
克隆仓库：
git clone https://github.com/robbyant/lingbot-world.git
cd lingbot-world
安装依赖：
# 请确保 torch >= 2.4.0
pip install -r requirements.txt
安装 flash_attn：
pip install flash-attn --no-build-isolation
下载模型
根据你需要的控制方式选择模型：
| 模型 | 控制信号 | 分辨率 | 下载链接 |
|---|---|---|---|
| LingBot-World-Base (Cam) | 相机位姿 | 480P & 720P | �HuggingFace · �ModelScope |
| LingBot-World-Base (Act) | 动作 | 480P & 720P | �HuggingFace |
| LingBot-World-Fast | 相机位姿 | 480P & 720P | �HuggingFace |
使用 huggingface-cli 下载：
pip install "huggingface_hub[cli]"
huggingface-cli download robbyant/lingbot-world-base-cam --local-dir ./lingbot-world-base-cam
或使用 modelscope-cli 下载：
pip install modelscope
modelscope download robbyant/lingbot-world-base-cam --local_dir ./lingbot-world-base-cam
第一条推理
运行前准备好以下输入：
- 输入图像：作为视频首帧的图片
- 文本提示词（prompt）：描述画面与运动
- 控制信号（可选）：可用 ViPE 从已有视频生成
intrinsics.npy：形状[num_frames, 4]，4 个值为[fx, fy, cx, cy]poses.npy：形状[num_frames, 4, 4]，每个[4, 4]为 OpenCV 坐标系下的变换矩阵
相机位姿控制（Cam）
480P：
torchrun --nproc_per_node=8 generate.py --task i2v-A14B --size 480*832 --ckpt_dir lingbot-world-base-cam --image examples/00/image.jpg --action_path examples/00 --dit_fsdp --t5_fsdp --ulysses_size 8 --frame_num 161 --prompt "The video presents a soaring journey through a fantasy jungle. The wind whips past the rider's blue hands gripping the reins, causing the leather straps to vibrate. The ancient gothic castle approaches steadily, its stone details becoming clearer against the backdrop of floating islands and distant waterfalls."
720P（把 --size 480*832 换成 --size 720*1280）：
torchrun --nproc_per_node=8 generate.py --task i2v-A14B --size 720*1280 --ckpt_dir lingbot-world-base-cam --image examples/00/image.jpg --action_path examples/00 --dit_fsdp --t5_fsdp --ulysses_size 8 --frame_num 161 --prompt "The video presents a soaring journey through a fantasy jungle. The wind whips past the rider's blue hands gripping the reins, causing the leather straps to vibrate. The ancient gothic castle approaches steadily, its stone details becoming clearer against the backdrop of floating islands and distant waterfalls."
也可以不使用控制信号运行（去掉 --action_path）：
torchrun --nproc_per_node=8 generate.py --task i2v-A14B --size 480*832 --ckpt_dir lingbot-world-base-cam --image examples/00/image.jpg --dit_fsdp --t5_fsdp --ulysses_size 8 --frame_num 161 --prompt "The video presents a soaring journey through a fantasy jungle. The wind whips past the rider's blue hands gripping the reins, causing the leather straps to vibrate. The ancient gothic castle approaches steadily, its stone details becoming clearer against the backdrop of floating islands and distant waterfalls."
动作控制（Act）
480P：
torchrun --nproc_per_node=8 generate.py --task i2v-A14B --size 480*832 --ckpt_dir lingbot-world-base-cam --image examples/05/image.jpg --action_path examples/05 --allow_act2cam --sample_steps 20 --dit_fsdp --t5_fsdp --ulysses_size 8 --frame_num 121 --prompt "The video presents a soaring journey through a fantasy jungle. The wind whips past the rider's blue hands gripping the reins, causing the leather straps to vibrate. The ancient gothic castle approaches steadily, its stone details becoming clearer against the backdrop of floating islands and distant waterfalls."
480P，使用更易用的动作字符串控制（如前进、左转、跳跃等）：
torchrun --nproc_per_node=8 generate.py --task i2v-A14B --size 480*832 --ckpt_dir lingbot-world-base-cam --image examples/05/image.jpg --action_path examples/05 --action_string "w-10,a-10,d-10,iw-15,none-10,j-10,l-10,s-15" --allow_act2cam --sample_steps 20 --dit_fsdp --t5_fsdp --ulysses_size 8 --prompt "The video presents a soaring journey through a fantasy jungle. The wind whips past the rider's blue hands gripping the reins, causing the leather straps to vibrate. The ancient gothic castle approaches steadily, its stone details becoming clearer against the backdrop of floating islands and distant waterfalls."
动作字符串格式为
按键-持续帧数，多个动作用逗号分隔。例如w-10表示前进 10 帧。