Reference Image
"The white lion opens its mouth and roars."
"The flower is blooming."
"This man gives a thumbs-up."
"Bigfoot is walking through the woods."
"Mouse falls to the ground."
"Camera zooms in. A giant striped planet rotates."
"Pouring water into the glass."
"A young woman adjusts her gaze and hand while holding a leafy branch."
The image-to-video (I2V) generation is conditioned on the static image, which has been enhanced recently by the motion intensity as an additional control signal. These motion-aware models are appealing to generate diverse motion patterns, yet there lacks a reliable motion estimator for training such models on large-scale video set in the wild. Traditional metrics, e.g., SSIM or optical flow, are hard to generalize to arbitrary videos, while, it is very tough for human annotators to label the abstract motion intensity neither. Furthermore, the motion intensity shall reveal both local object motion and global camera movement, which has not been studied before. This paper addresses the challenge with a new motion estimator, capable of measuring the decoupled motion intensities of objects and cameras in video. We leverage the contrastive learning on randomly paired videos and distinguish the video with greater motion intensity. Such a paradigm is friendly for annotation and easy to scale up to achieve stable performance on motion estimation. We then present a new I2V model, named MotionStone, developed with the decoupled motion estimator. Experimental results demonstrate the stability of the proposed motion estimator and the state-of-the-art performance of MotionStone on I2V generation. These advantages warrant the decoupled motion estimator to serve as a general plug-in enhancer for both data processing and video generation training.
Our unique feature is the precise camera or object motion control through text instructions. Furthermore, users have the ability to customize these motions by setting different "motion intensities".
"Camera pans right. A turtle is swimming."
Camera Motion Intensity: 2
Camera Motion Intensity: 5
Camera Motion Intensity: 8
“Camera zooms in. A warm campfire burning brightly at night, with flames flickering and sparks flying upward into the dark sky.”
“A teddy bear is dancing in the snow.”
Object Motion Intensity: 2
Object Motion Intensity: 5
Object Motion Intensity: 8
“Wind blows the tree.”
We compare MotionStone with a recent Image to-Video (I2V) method CogVideoX. Notably, MotionStone demonstrates superior performance in text-guided motion control.
"A vast, luminous spiral galaxy with a warm yellow core and colorful arms slowly rotates against a dark backdrop, with stars scattered throughout, as the camera pushes in towards the bright core."
CogVideoX
MotionStone
"A fair-skinned, blonde person sits slightly to the left, while another individual with a darker complexion, holding a small black makeup pencil, carefully applies makeup around their eyes. The seated person remains still throughout the process."
"A person, wearing a red shirt and a helmet, rides a mountain Example 4 bike along a rocky path. The terrain is rugged, with large rock formations and a dirt path."
"A giraffe with a light brown and white patterned coat stands in a lush environment, moving its head from a frontal view to the right and slightly lowering its head as it eats leaves from nearby plants."
The framework of our model is shown below. The model takes a reference image, a text prompt, and two disentangled motion intensities predicted by a motion intensity estimator as inputs. During training, we first extract the first frame from the input video to use as a conditioning reference for generation. The trained motion intensity estimator then predicts the camera and object motion intensities of the input video, providing two motion scores that guide the video generation process. During inference, users can specify the desired motion intensities for the object and camera, if available, to customize the generated video. The model takes a latent z and concatenates the first frame latent of the video along the channel dimension to guide video generation. For frames beyond the first in the video sequence, zeros are padded in place. Subsequently, the model uses a text encoder to extract the features of the text prompt, which are then concatenated with the latent and fed into the diffusion transformer. Meanwhile, the two motion intensities predicted by the motion estimator are mapped to high-dimensional embeddings by MLP, then concatenated and added to the time step t. This combined representation serves as a modulation condition for the vision and text features, enabling fine-grained control over the motion of video generation.
@inproceedings{shi2025motionstone, title={Motionstone: Decoupled motion intensity modulation with diffusion transformer for image-to-video generation}, author={Shi, Shuwei and Gong, Biao and Chen, Xi and Zheng, Dandan and Tan, Shuai and Yang, Zizheng and Li, Yuyuan and He, Jingwen and Zheng, Kecheng and Chen, Jingdong and others}, booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference}, pages={22864--22874}, year={2025} }