FrameForge: Unified Image Generation Using the Prior Knowledge of Video Foundation Models

Problem

Existing methods that use video models for image generation always place the condition image and the target image at fixed frame positions. The authors show that the difference between condition and target varies widely across tasks. Based on LPIPS distributions, editing involves small changes, while subject-driven generation and edge-guided generation involve large changes. The difference also varies widely between samples within the same task, so with a fixed gap it is hard for one shared adapter to handle several tasks together.

Method

The HunyuanVideo1.5 backbone and VAE are frozen, and only a single LoRA is trained. (1) MoTE (Mixture of Temporal Experts) uses several temporal RoPE Experts, each corresponding to a different virtual frame gap (17, 69 and 121 frames). A router that looks at the text, the condition and the noisy target image picks the Top-1 Expert for each sample. Training uses a straight-through estimator and a load-balancing loss. (2) CoTR (Chain of Temporal Reasoning) inserts noise-initialized intermediate latents between the condition and the target. In the middle training stage, an attention mask is applied stochastically so that target tokens cannot directly see the text and condition tokens, pushing information to pass through this intermediate path. Training has three stages: train only the router → turn on the mask and train the router and LoRA together → finish with full attention.

Results

• The authors report higher performance than the task-specific and unified baselines they compared against on DreamBench (subject-driven generation, GPT-4o-based VIEScore), the COCO2017 validation set (edge-guided generation, FID, SSIM, CLIP-T) and the GEdit-Bench English set (editing, GPT-4.1-based scores). The provided text is cut off before the tables, so the specific numbers could not be checked.
• Instead of a separate adapter per task, a single LoRA handles all three tasks (trained on 170,000 images per task, 510,000 in total).
• The authors state that ablation experiments confirmed that MoTE and CoTR each contribute to performance.

Limitations

Only three tasks were evaluated, and resolution was fixed at 512×512. Some of the main metrics rely on LLM-judge scores from models such as GPT-4o and GPT-4.1. Experiments used only one backbone, HunyuanVideo1.5. Because the provided text is cut off, numerical comparisons and inference cost analysis could not be checked.

Why it matters

It links the 'change over time' that video models learn to the size of change between image tasks. This shows a way to offer several image generation and editing capabilities together by attaching only a lightweight adapter to an existing video model, without separately training a unified model on large datasets.

Key terms

RoPE(Rotary Positional Embedding)

A positional encoding that reflects token positions in attention through rotations. This paper changes the temporal RoPE values to make two images appear to be a certain number of frames apart.

MoTE(Mixture of Temporal Experts)

A routing structure that keeps several RoPE settings, each representing a different virtual time gap, and picks one depending on the input.

Reasoning-bottleneck attention mask

A mask that blocks direct attention between target tokens and condition/text tokens, so information passes only through the intermediate latents.

LoRA

A parameter-efficient fine-tuning technique that keeps the original weights frozen and adapts the model by training only small low-rank matrices.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Project
• Hugging Face

Authors: Yijing Lin, Zhendong Mao, Mengqi Huang, Yuxin Wang, Yongdong Zhang · Posted 2026-10-27 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.