VoxelMem: video generation with 4D voxel control and memory

Zhiyuan Zhang1*, Jingbo Zhang2, Zhitong Huang1,3, Qi Sun1,
Haitao Lin2, Shenghao Zhang2, Yuzhen Liu2, Jing Liao1
1CityU, 2Robotics X, Tencent, 3ARC Lab, Tencent
*Work done during an internship at Robotics X, Tencent
Paper Code Supp

Examples synthesized by VoxelMem. Given 4D voxel controls, VoxelMem generates long videos across diverse application scenarios, including filmmaking (e.g., synthesis-to-real transfer and re-rendering), realistic physical simulation, and streaming web-game generation.

Abstract

Long video generation remains limited by weak controllability, spatial drift, and memory loss when objects leave and re-enter the view. We propose VoxelMem, a framework that uses 4D voxels as both controllable primitives and memory carriers. Each voxel stores time-dependent geometry, visibility, segmentation identity, and local appearance, enabling multi-granularity control from object-level layout to part-level motion. To maintain long-term consistency, VoxelMem combines local appearance propagated with voxel motion and global history frames retrieved by future voxel coverage, providing both fine-grained spatial correspondence and holistic scene references. We further extend VoxelMem to streaming generation by distilling the bidirectional model into a causal generator. Finally, we construct 4D video-voxel training data from synthetic rendered scenes and real-world videos to support multi-granularity control and memory learning.

Method

Overview of the VAM framework

Overview of the VoxelMem framework. Given an initial image and 4D voxel controls, VoxelMem generates videos chunk by chunk while updating a voxel cache. For each target chunk, the voxel field provides voxel-indexed global memory for retrieving relevant frames and voxel-carried local memory for propagating patch-level appearance. Both memory sources are injected into video model to improve long-range consistency under camera revisits and dynamic motion.

Data

Overview of the video-voxel data construction pipeline

Overview of the video-voxel data construction pipeline. Synthetic scenes provide accurate 4D voxels through static mesh voxelization and dynamic vertex-space hashing. Real videos are converted into voxel-like primitives by combining 3D tracking, segmentation, and motion-aware clustering. Both sources are unified as paired RGB videos and 4D voxels for training.

Applications

Synthesis-to-Real Transfer

Re-Rendering

Physical Simulation

Streaming Web-Game