OpenVision OpenVision icon

A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning

Xianhang Li* · Yanqing Liu* · Haoqin Tu · Hongru Zhu · Cihang Xie

University of California, Santa Cruz

arXiv arXiv Code HF Model

Abstract

OpenAI's CLIP, released in early 2021, have long been the go-to choice of vision encoder for building multimodal foundation models. Although recent alternatives such as SigLIP have begun to challenge this status quo, to our knowledge none are fully open: their training data remains proprietary and/or their training recipes are not released. This paper fills this gap with OpenVision, a fully-open, cost-effective family of vision encoders that match or surpass the performance of OpenAI’s CLIP when integrated into multimodal frameworks like LLaVA. OpenVision builds on existing works---e.g., CLIPS for training framework and Recap-DataComp-1B for training data---while revealing multiple key insights in enhancing encoder quality and showcasing practical benefits in advancing multimodal models. By releasing vision encoders spanning from 5.9M to 632.1M parameters, OpenVision offers practitioners a flexible trade-off between capacity and efficiency in building multimodal models: larger models deliver enhanced multimodal performance, while smaller versions enable lightweight, edge-ready multimodal deployments.

Key Contributions

  • Fully Open Vision Encoders: Datasets, training recipes, and model checkpoints are entirely public, fostering reproducibility and transparency in multimodal research.
  • Wide Range of Model Scales: A comprehensive family of vision encoders from Tiny (5.9M) to Huge (632.1M) parameters, supporting deployment from edge devices to high-capacity servers.
  • Superior Multimodal Performance: Matches or surpasses proprietary vision encoders (e.g., OpenAI-CLIP, SigLIP) across popular multimodal benchmarks (e.g., LLaVA-1.5, Open-LLaVA-Next).
  • Efficient Progressive Resolution Training: Demonstrates significant efficiency improvements (2×–3× faster) compared to proprietary counterparts through a progressive, multi-stage resolution training strategy.
  • Flexible Patch-Size Configuration: Supports adaptive encoding (8×8, 16×16 patches), allowing detailed visual understanding or efficient processing based on practical needs.

Detailed Comparisons and Efficiency

OpenVision vs. Proprietary Encoders

OpenVision vs Proprietary Encoders

OpenVision encoders match or outperform proprietary models like OpenAI's CLIP and Google's SigLIP across multimodal tasks.

Performance under LLaVA-1.5 Framework

LLaVA-1.5 Performance Comparison

OpenVision demonstrates strong performance improvements over existing CLIP models under the LLaVA-1.5 multimodal framework.

Performance under Open-LLaVA-Next Framework

Open-LLaVA-Next Performance Comparison

Under Open-LLaVA-Next, OpenVision maintains its competitive edge, excelling particularly in document-heavy multimodal tasks.

Efficiency Comparison

Efficiency Comparison

OpenVision achieves superior multimodal performance with significantly reduced training time compared to proprietary alternatives.

Model Zoo (ImageNet-1K)

We report ImageNet-1K Top-1 accuracy across OpenVision variants. All models are available in both JAX and PyTorch formats.

ModelSizePatchResolutionTop-1LinkJAXPyTorch
ViT-Tiny5M1616046.9%HF✓✓
ViT-Tiny5M1622449.6%HF✓✓
ViT-Tiny5M1638451.5%HF✓✓
ViT-Tiny5M816051.9%HF✓✓
ViT-Tiny5M822453.5%HF✓✓
ViT-Tiny5M838453.9%HF✓✓
ViT-Small22M1616063.5%HF✓✓
ViT-Small22M1622465.9%HF✓✓
ViT-Small22M1638467.1%HF✓✓
ViT-Small22M816067.3%HF✓✓
ViT-Small22M822468.6%HF✓✓
ViT-Small22M838468.5%HF✓✓
ViT-Base86M1616072.4%HF✓✓
ViT-Base86M1622473.9%HF✓✓
ViT-Base86M1638474.5%HF✓✓
ViT-Base86M816074.8%HF✓✓
ViT-Base86M822475.4%HF✓✓
ViT-Base86M838475.6%HF✓✓
ViT-Large307M148474.7%HF✓✓
ViT-Large307M1422478.5%HF✓✓
ViT-Large307M1433678.9%HF✓✓
SoViT-400M400M148476.2%HF✓✓
SoViT-400M400M1422479.7%HF✓✓
SoViT-400M400M1438479.9%HF✓✓
ViT-Huge632M148477.4%HF✓✓
ViT-Huge632M1422480.4%HF✓✓

Model Usage

With Our Customized OpenCLIP Tokenizer

⚠️ IMPORTANT: Make sure you're importing from src/convert_upload/open_clip/ in this repo.
The tokenizer implementation here is customized and not yet available in the official OpenCLIP repo or PyPI release. 


import torch
import torch.nn.functional as F
from urllib.request import urlopen
from PIL import Image

# ⚠️ Use our repo's tokenizer implementation at src/convert_upload/open_clip/
from open_clip import create_model_from_pretrained, get_tokenizer

model, preprocess = create_model_from_pretrained('hf-hub:UCSC-VLAA/openvision-vit-large-patch14-224')
tokenizer = get_tokenizer('hf-hub:UCSC-VLAA/openvision-vit-large-patch14-224')

image = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))
image = preprocess(image).unsqueeze(0)

text = tokenizer(["a diagram", "a dog", "a cat", "a beignet"], context_length=model.context_length)

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features = F.normalize(image_features, dim=-1)
    text_features = F.normalize(text_features, dim=-1)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)  # prints: [[0., 0., 0., 1.0]]

Acknowledgement

We would like to thank the TPU Research Cloud (TRC) program and Google Cloud Research Credits program for supporting our computing needs.

BibTeX


@article{li2025openvision,
  title   = {OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning},
  author  = {Li, Xianhang and Liu, Yanqing and Tu, Haoqin and Zhu, Hongru and Xie, Cihang},
  journal = {arXiv preprint arXiv:2505.04601},
  year    = {2025}
}