A new multilingual visual document retrieval model has been released by LlamaIndex, a significant development for industries that rely on complex document search and retrieval. The model, called vdr-2b-multi-v1, is designed to encode document page screenshots into dense single-vector representations, allowing for efficient search and querying of visually rich multilingual documents without the need for OCR or data extraction pipelines.

The model has been trained on a dataset of 500k high-quality samples across five languages: Italian, Spanish, English, French, and German. This extensive training enables vdr-2b-multi-v1 to perform well in both multilingual and cross-lingual retrieval scenarios, with the ability to search for documents in one language using queries in another.

What Happened

LlamaIndex has released its first-ever models focused on Multilingual Visual Document Retrieval. The company's vdr-2b-multi-v1 model is a cutting-edge multilingual embedding model designed for visual document retrieval across various languages and domains. This model encodes document page screenshots into dense single-vector representations, allowing efficient search and querying of visually rich multilingual documents without OCR or data extraction pipelines.

The release includes two models: a multilingual 2B parameter model (vdr-2b-multi-v1) and an English-only 2B parameter model (vdr-2b-v1). Both models are accompanied by the largest open-source multilingual dataset for visual document retrieval, vdr-multilingual-train. This dataset consists of 500k high-quality samples across five languages.

Background and Context

Visual Document Retrieval (VDR) is a task that involves retrieving visually-rich documents by leveraging both visual and textual cues. Traditional text-centric pipelines, which rely purely on OCR-extracted text for indexing, are insufficient for visually complex documents where semantic meaning is co-determined by layout, font, tables, figures, and spatial relationships.

Recent advancements in VDR have introduced models that learn to map document page images—and, when available, queries or supporting text—into embedding spaces that reflect global and local image cues. These models facilitate efficient semantic retrieval at page-, layout-, or even patch-level granularity. Contemporary VDR systems universally employ vision-LLMs (VLMs) or multimodal LLMs (MLLMs), with architectural innovations along several axes, including image encoder backbones.

Why It Matters to the Industry

The release of vdr-2b-multi-v1 and vdr-2b-v1 models is significant for industries that rely on complex document search and retrieval. The multilingual capabilities of these models make them particularly beneficial for regions like Europe, where multilingual documents are prevalent. The performance and efficiency of these models also outperform previous benchmarks in terms of speed, memory efficiency, and retrieval accuracy.

The open-source nature of the vdr-2b-multi-v1 model allows developers to integrate it into their applications, making it a valuable resource for industries that require efficient document search and retrieval. The accompanying dataset, vdr-multilingual-train, is also a significant contribution to the field, providing a large-scale multilingual dataset for visual document retrieval.

What Comes Next

The release of vdr-2b-multi-v1 and vdr-2b-v1 models marks an important milestone in the development of visual document retrieval. The next steps will involve exploring how these models perform when adapted to new and specific domains. Early tests suggest impressive retrieval gains with minimal data and computational resources.

Key Facts

  • The vdr-2b-multi-v1 model is a cutting-edge multilingual embedding model designed for visual document retrieval across various languages and domains.
  • The model has been trained on a dataset of 500k high-quality samples across five languages: Italian, Spanish, English, French, and German.
  • vdr-2b-multi-v1 performs well in both multilingual and cross-lingual retrieval scenarios, with the ability to search for documents in one language using queries in another.
  • The model is accompanied by the largest open-source multilingual dataset for visual document retrieval, vdr-multilingual-train.
  • vdr-2b-v1 is an English-only 2B parameter model that outperforms previous benchmarks in terms of speed, memory efficiency, and retrieval accuracy.