What is W.A.L.T, the model that generates videos from images or text?

On December 27, 2023, researchers from Stanford University, Google Research, and the Georgia Institute of Technology introduced the Window Attention Latent Transformer (W.A.L.T) model. Utilizing the transformer neural network architecture and presenting a novel approach for latent video diffusion models (LVDMs), it facilitates the generation of photorealistic videos from static images or textual descriptions.

### The Innovative Approach of the W.A.L.T Team

The researchers employed an autoencoder to map both videos and images into a unified latent space of reduced dimension, enabling learning and generation across modalities. By training W.A.L.T on videos and images concurrently, they provided the model with a deeper understanding of motion from the outset.

Furthermore, a specialized design of transformer blocks allowed them to model latent video diffusion. These blocks alternate between spatial and spatiotemporal self-attention layers, with spatial attention restricted to a window. This design offers significant advantages, including reduced computational demands due to localized windowed attention and the possibility of joint training to process images and video frames independently.

### W.A.L.T’s Strong Performance

The research team states: “Taken together, these design choices allow us to achieve state-of-the-art performance on established video (UCF-101 and Kinetics-600) and image (ImageNet) generation benchmarks without using classifier-free guidance.”

The images and videos are encoded into a shared latent space. The transformer backbone processes these latents with blocks having two layers of window-restricted attention: the spatial layers capture spatial relationships within images and videos, while the spatiotemporal layers model temporal dynamics in videos and transitioning images via the identity attention mask. Text conditioning is done via spatial cross-attention.

The researchers trained a cascade of three models for the text-to-video generation task: the base latent video diffusion model that generates small 128 x 128 pixel clips, upsampled by two super-resolution video diffusion models that produce 3.6-second videos at 8 frames per second, reaching a resolution of 512 x 896 pixels.

W.A.L.T demonstrates robust performance, particularly in terms of video smoothness, and appears to validate the researchers’ assertion that “a unified framework for image and video will bridge the gap between image and video generation.”

### W.A.L.T’s Contributions to the Field of Text-to-Video Generation

W.A.L.T stands as a significant contribution to the field of text-to-video generation, offering several key advantages:

1. **Unified Framework:** By training a single model on both images and videos, W.A.L.T develops a comprehensive understanding of visual content, enabling it to generate videos that are both realistic and coherent.

2. **Windowed Attention:** The use of windowed attention allows W.A.L.T to focus on local regions of the video, reducing computational costs and improving training efficiency.

3. **State-of-the-Art Performance:** W.A.L.T achieves state-of-the-art results on established video and image generation benchmarks, demonstrating its strong performance in generating high-quality videos from text or image prompts.

W.A.L.T’s capabilities open up new possibilities for creative content generation, video editing, and various applications in entertainment, education, and beyond. It represents a significant step forward in the field of AI-powered video synthesis.