Abstract

Recent rapid advancements in text-to-video (T2V) generation, such as SoRA and Kling, have shown great potential for building world simulators. However, current T2V models struggle to grasp abstract physical principles and generate videos that adhere to physical laws. This challenge arises primarily from a lack of clear guidance on physical information due to a significant gap between abstract physical principles and generation models. To this end, we introduce the World Simulator Assistant (WISA), an effective framework for decomposing and incorporating physical principles into T2V models. Specifically, WISA decomposes physical principles into textual physical descriptions, qualitative physical categories, and quantitative physical properties. To effectively embed these physical attributes into the generation process, WISA incorporates several key designs, including Mixture-of-Physical-Experts Attention (MoPA) and a Physical Classifier, enhancing the model's physics awareness. Furthermore, most existing datasets feature videos where physical phenomena are either weakly represented or entangled with multiple co-occurring processes, limiting their suitability as dedicated resources for learning explicit physical principles. We propose a novel video dataset, WISA-32K, collected based on qualitative physical categories. It consists of 32,000 videos, representing 17 physical laws across three domains of physics: Dynamics, Thermodynamics, and Optics. Experimental results demonstrate that WISA can effectively enhance the compatibility of T2V models with real-world physical laws, achieving a considerable improvement on the VideoPhy benchmark.

More Visual Comparison


An apple falls into a vat of cider, sending up a spray

Cosmos
CogVideoX
PhyT2V
WISA

The rope supports a wooden swing

Cosmos
CogVideoX
PhyT2V
WISA

Coin spins rapidly on a wooden table

Cosmos
CogVideoX
PhyT2V
WISA

Pouring milk into still tea

Cosmos
CogVideoX
PhyT2V
WISA

The eraser rubs against the paper, removing pencil marks

Cosmos
CogVideoX
PhyT2V
WISA

More Samples Generated by WISA

An antique clock's pendulum hangs inside its ornate wooden casing, with the camera focusing on its rhythmic motion. The metal rod remains straight as it swings, and the spherical bob at its end shines with a polished gleam, reflecting faint images of the wood grain within the clock. The pendulum moves with a steady frequency, producing a soft 'tick-tock' sound at each turn, harmonizing with the faint hum of the clock's internal gears. Throughout the motion, the amplitude gradually diminishes, but the central balance point stays constant. Shadows inside the wooden case shift gently with the pendulum's rhythm, enhancing the serene atmosphere of the scene. The background remains static, directing all attention to the pendulum's graceful and consistent movement, showcasing the stability of rigid body motion.
On a cold winter day, the outside temperature is frigid while the indoors remain cozy and warm. The inside surface of the window, chilled by the outdoor air, causes water vapor in the room to condense into a fine mist on the glass. Over time, the tiny droplets coalesce into larger liquid beads. These droplets slowly trickle down the glass, leaving clear trails behind. Occasionally, some droplets merge and slide down in a sudden motion. The view outside appears slightly blurred through the condensation, adding a serene and uniquely wintry atmosphere to the scene.
On a sunny afternoon, a small soap bubble floats gently in the air. The surface of the bubble, due to the structure of the thin film, causes light to interfere, creating beautiful rainbow patterns. As the bubble rotates and changes angle, the interference effect of the light shifts, with golden, blue, green, and purple light spots blending together, like a dynamic painting. The surface of the bubble reflects the surrounding light, and the delicate film of the soap bubble creates a soft halo in the sunlight.
a solitary lantern swings gently in the night, casting flickering shadows on the cobblestone. its warm glow disrupts the cool darkness around, creating dancing shapes that morph and twist, dissolving against the stillness of the empty street. the motion of the lantern defines the rhythm of the light.
a vibrant red rubber ball bounces steadily on a smooth, sunlit sidewalk. each impact compresses the rubber, then releases, sending it soaring back into the air. the rhythm of its rise and fall showcases the perfect elasticity, each rebound a testament to the energy exchanged with the concrete.
a bright green rubber band stretches as fingers pull it taut, revealing its thin, glossy surface. tiny creases form along its length, it demonstrates the elasticity of materials through clear deformation.
a single firecracker bursts in mid-air, releasing a brilliant flash of light and a shockwave that radiates outward. fragments of colored paper scatter wildly, spiraling away like confetti, while the roar of the explosion resonates, momentarily punctuating the calm night with vibrant chaos.
a solitary ice cube rests on a warm granite countertop. sunlight streams through the window, illuminating its glossy surface. slowly, drops of water form and trickle down the sides, pooling beneath, as the cube shrinks and contorts, surrendering to the heat around it.
a lone pot of water simmers on a hot stove. tiny bubbles form at the bottom, dance upwards, and burst at the surface. steam curls gracefully into the air, shimmering as it dissipates, blurring the edge of reality, revealing the dynamic transformation from liquid to gas.
a solitary red balloon floats silently against a still pond. its vibrant surface mirrors the sky's blue and soft clouds, creating a perfect reflection. as a gentle breeze stirs, the balloon sways slightly, causing ripples that distort its mirrored image, blending reality with illusion.
a solitary glass marble rolls across a polished wooden table. as it glides, sunlight filters through a nearby window, casting a bright halo around it. the marble's surface perfectly reflects the warm glow and the faint outlines of the room, creating a captivating interplay of light and shape.
a series of raindrops trickles from the edge of the leaf, suspended momentarily before breaking free. as it plummets, it elongates, stretching into a delicate teardrop shape. upon hitting the puddle below, it splashes outward, ripples radiating in concentric circles, dancing across the water's surface.

WISA-32K


Collision
Rigid Body Motion
Elastic Motion
Liquid Motion
Gas Motion
Deformation
Deformation
Melting
Solidification
Vaporization
Liquefaction
Explosion
Combustion
Combustion
Reflection
Refraction
Elastic_Motion
Scattering
Interference and Diffraction
Unnatural Light Sources