Visual State Encoders for World Models

One representation for prediction, instruction, and inspection

World models predict how a scene changes in response to actions and use these predictions to plan. Many of them operate in the latent space of a frozen pretrained visual encoder. The encoder therefore decides what the world model can see. ALPS unifies physical-state information, native language alignment, and pixel reconstruction in a single set of patch tokens.

No compared encoder is better than ALPS on all three.

Two-stage visual state pretraining

We pretrain our encoder in two stages. The first stage establishes language alignment, pixel reconstruction, and spatial features in a shared representation. We train a ViT-B/16 from scratch on image-text pairs with contrastive alignment, captioning, reconstruction, and dense correspondence objectives on the same patch tokens. The second stage then strengthens state information through cross-view self-distillation, training the encoder to predict scene content across changes in crop, scale, and appearance. From an augmented view, the student predicts a frozen first-stage teacher's patch tokens at corresponding locations in a clean view. Both stages train on still images and text alone, without video or interaction supervision.

One encoder for state, language, and pixels

We compare ALPS with fifteen public encoders, all frozen and evaluated under common protocols. ALPS achieves the lowest aggregate LIBERO state-probe error among ViT-B encoders, matching the 3.5× larger V-JEPA 2-L, while retaining strong language alignment and accurate pixel reconstruction.

Table 1. Frozen encoders on state, language, and pixels: ALPS has the lowest state-probe error among the ViT-B encoders. State probe: LIBERO position error relative to ALPS (Stage 1); language: each model's own text tower. Bold: best ViT-B encoder in each column, ALPS (Stage 1) not ranked; gray: ViT-L references; n/a: no released text tower (not zero).

Physical state, read from frozen tokens

Pixels, decoded from frozen tokens

The refinement strengthens the state and keeps language

Cross-view refinement reduces state-probe error by 17% while changing image-text retrieval by at most 0.5 points, and also improves world-model prediction. In contrast, the same refinement lowers ImageNet zero-shot accuracy and image-to-text retrieval by 4–12 points for SigLIP 2 and CLIP.

Control, planning, and prediction on frozen tokens

Frozen ALPS tokens achieve the highest mean policy success in our LIBERO comparison when paired with our native text tower, and support leading planning performance on PushT, Wall, and Granular. Predicted future tokens can also be decoded into images for inspection.

Use ALPS in a few lines

The repository holds the PyTorch inference package, the unified evaluation, and the JAX and PyTorch training code. The weights and the eight ALPS policies of the LIBERO experiments are on Hugging Face.

git clone https://github.com/UCSC-VLAA/ALPS.git && cd ALPS
pip install -e .                 # torch, numpy, pillow

import torch, alps
from PIL import Image

model = alps.load("alps-vitb16")                          # downloads alps_vitb16.safetensors from Hugging Face
x = alps.center_crop_transform(Image.open("image.jpg"))   # [1, 3, 224, 224], normalized

with torch.no_grad():
    out = model.encode_image(x)
    z = out["tokens"]            # [1, 196, 768]  raw patch tokens: the visual state
    img = model.image_embedding(tokens=z)                      # L2-normalized
    txt = model.text_embedding(["a robot arm opens the top drawer", "a photo of a dog"])
    probs = (model.contrastive_scale() * img @ txt.T).softmax(-1)

GitHub: UCSC-VLAA/ALPS  ·  Hugging Face: UCSC-VLAA/ALPS  ·  License: Apache-2.0