Vision Language Models (VLMs) are built by finetuning existing text-only language models to process images, not training from scratch. Research labs in 2026 avoid multimodal training from the ground up because it's prohibitively expensive and often yields worse results. Instead, they take pretrained models like BERT and adapt them to handle visual inputs through specialized architectures.

At the core of this approach is the Query-Former (Q-Former), which bridges the gap between pixel-based embeddings from Vision Transformers and text-based representations. The Q-Former uses cross-attention layers to contextualize image features with query embeddings, effectively teaching the model to understand visual content in a text-compatible format. Training typically uses multimodal datasets like Conceptual Captions, with losses such as ITC or ITM to align image and text representations.

The process involves freezing the image backbone (like ViT-Base) to prevent overfitting and reduce costs, then focusing computational resources on the adapter layer and language model. This architectural choice transfers vision-language learning to the adapter and text backbone, making the finetuning process more efficient while preserving the model's original world knowledge.