DreamLoop: Controllable Cinemagraph Generation from a Single Photograph

Aniruddha Mahapatra, Long Mai, Cusuh Ham, Feng Liu

ADOBE RESEARCH

Controllable General Domain Cinemagraphs - Our Results More Results

Cinemagraph Creation from Single Photograph with Motion Control

DreamLoop enables users to generates cinemagraphs from a single photograph with intuitive and easy-to-use, but precise motion control.

Prompt: A close-up shot of a person holding a DSLR camera. The person is trying to adjust the lens of the camera by rotating it. Everything else remains static in place. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end."

Prompt: A cat wiggling its tail while sitting still on a window counter top. The cat does not move its body and remains still in one place. The camera is completely static. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end."

Prompt: A makeup artist is applying makeup with a brush on the face of a female celebrity. The scene is static, the makeup artist and celebrity do not move, they remain still in their place. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end."

Prompt: A close-up shot of a table with a burger and a girl holding her phone. She remains in the same posture, static, only moving her thumb to swipe on her phone. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end."

Prompt: A plastic toy sunflower in a field. The black and white checkered petals of the sunflower are rotating. Everything else remains static in its place. The shot is taken with a static camera. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end."

Prompt: A girl is sitting in a cafe, stirring her coffee in a glass. She remains stationary in her place and does not move. The camera is also stationary. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end."

Controllable General Domain Cinemagraphs - Comparisons More Results

Comparison of Our method with Baselines for Controllable General Domain Cinemagraphs

We compare our method (both with internal model and Wan2.2-5B) with baselines which include: I) Image-to-Video Models - CogVideoX and Wan2.2-5B (pretrained), II) Motion Controllable Image-to-Video Models - DragAnything, III) Our method with only first-last-frame conditioning. For I) and II), we use post-processing to make the videos looping.

Input

CogVideoX

DragAnything

Wan2.2-5B (Pretrained)

Ours (Wan2.2-5B) [first-last]

Ours (internal) [first-last]

Ours (Wan2.2-5B)

Ours (internal)

Timing Control for Cinemagraphs More Results

Importance of Timing Control over Object Motion

DreamLoop provides additional control over the timing of object motion in cinemagraphs, precisely letting users define the duration of the motion for each action.

Prompt: “A metallic ball is oscillating in a car hanging from the rear-view mirror with a thread. Everything else in the scene is completely static. The shot is taken with a static camera..”

Explanation: The metallic ball is oscillating in a Simple Harmonic Motion (SHM). The speed of motion near the equilibrium position is faster than the speed of motion near the extreme positions. We can achieve this effect by controlling the timing of the motion.

Reference frame for timed motion control

w/o Timing Control

w/ Timing Control

Full vs Partial Control over Object Motion More Results

Full vs Partial Control over Object Motion

DreamLoop also works in cases where user does to provide the full motion path for all objects in the scene, but just the intial motion hint. The rest is inferred by the model.

Prompt: A scene of two empty swings oscillating due to the winds blowing. The background is covered in snow, and it is a morning scene. The shot is with a static camera. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end.

Full Input Reference frame for partial control

Generated (Full)

Partial Input Reference frame for partial control

Generated (Partial)

Limitations of DreamLoop

Despite the capabilities of DreamLoop and its ability to generate cinemagraphs from a single photograph with intuitive and easy-to-use, but precise motion control, it still has some limitations.

Limitation: The hand of the girl gets deformed in the video. This can happen because of limitations of the base model involving human motion.

Limitation: Only part of the petal moves for which motion path was defined. But the results is not as expected intuitively.

Limitation: The buttons on the blazer do not move in the video with the hand. This is likely the limitation of the base model involving object motion.

Teaser

Teaser Figure

Video results for the teaser figure in the main paper.

Ptompt (top): A girl holding a rabbit in her lap. Both are rabbit and the girl are completely stationary. Only the earrnig of the girl is oscillating very slowly. The shot is taken with a static camera. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end.

Prompt (bottom): A girl petting the rabbit which is sitting on her lap. Both are rabbit and the girl are completely stationary. The girl only moves her arm to pet the rabbit. The rabbit is nibbling. The shot is taken with a static camera. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end.

Reference frame for timing control scene 1

Quantitative Comparison - General Domain Cinemagraphs

Performance on VBench

DreamLoop outperforms baselines on custom domain cinemagraphs on controllable generation for general domain. across all VBench metrics.

Method	Aesthetic Quality ↑	Imaging Quality ↑	Temporal Flickering ↑	Motion Smoothness ↑	Background Consistency ↑	Subject Consistency ↑
CogVideoX*	0.5353	0.6419	0.9888	0.9904	0.9741	0.9787
Wan2.2-5B* (pretrained)	0.5442	0.6018	0.9881	0.9922	0.9598	0.9605
DragAnything*	0.4986	0.5535	0.9674	0.9789	0.9613	0.9474
Ours (Wan2.2-5B) [only first-last]	0.5897	0.6756	0.9950	0.9959	0.9746	0.9941
Ours (internal) [only first-last]	0.5872	0.6615	0.9899	0.9929	0.9685	0.9566
Ours (Wan2.2-5B)	0.5998	0.6762	0.9955	0.9964	0.9800	0.9868
Ours (internal)	0.5997	0.6771	0.9965	0.9959	0.9849	0.9858

User Study - General Domain Cinemagraphs

Human Preference Study

User Study with Amazon Mechanical Turk involving 50 participants confirms DreamLoop achieves higher user preference for controllable cinemagraph generation compared with contemporary open-source methods. Each comparison is done by 3 participants.