jina-embeddings-v5-omni-small

Jina AI's jina-embeddings-v5-omni-small is a 1.74-billion-parameter multimodal embedding model designed to map text, image, audio, and video inputs into a single, shared 1024-dimensional vector space. Built as a local AI model with 1.74B parameters in 2026, it addresses a core operational bottleneck in multimodal vector search: the historical requirement to rebuild entire vector stores when expanding from text-only retrieval to rich media indexing.

The model achieves this compatibility by extending jina-embeddings-v5-text-small using a frozen-tower composition architecture. The core language model backbone and its underlying LoRA adapters remain locked, while dedicated encoders for vision and audio are aligned to the existing text space via lightweight projection connectors. Because the text geometry is fully preserved, text embeddings are identical to those produced by the text-only predecessor. Teams with existing text vector indexes can incorporate visual and acoustic search without re-indexing a single textual record.

Operating under a CC-BY-NC-4.0 license, jina-embeddings-v5-omni-small positions itself directly against monolithic multimodal models that require extensive VRAM footprint and full retraining. It provides local search engines, retrieval-augmented generation (RAG) pipelines, and multimodal asset management systems with a high-throughput, low-latency embedding layer that fits comfortably on modern consumer-tier desktop GPUs.

The architecture of jina-embeddings-v5-omni-small avoids the performance regressions typical of joint multimodal training by locking its primary components. It operates as a dense 1.74B parameter model comprising three distinct architectural modules:

• Text Backbone: A frozen transformer derived from jina-embeddings-v5-text-small, equipped with task-specific LoRA adapters.
• Vision Encoder: A SigLIP2 vision transformer that ingests images and video frames, mapping pixel representations into the transformer sequence.
• Audio Encoder: A Whisper-large-v3 acoustic tower that processes raw audio spectrograms.

Rather than fine-tuning all 1.74 billion weights, Jina AI trained only the cross-modal connectors, which constitute roughly 0.35% of the total parameter count. This locked-aligned-towers methodology forces the non-text modalities to conform to the established linguistic manifold rather than shifting the vector space.

The model supports an expansive context window of 32,768 tokens. This enables native ingestion of long-form documents, dense technical manuals, extended transcripts, and sequential video frames without forcing aggressive chunking strategies that fragment semantic meaning.

Embeddings are generated via last-token pooling and output at a native width of 1024 dimensions. The model integrates Matryoshka Representation Learning (MRL), allowing engineers to slice vector dimensions down to 768, 512, 256, 128, 64, or 32 dimensions with negligible loss in retrieval recall. This flexibility allows operators to scale vector databases down to minimal RAM footprints during high-scale indexing.

The base release of jina-embeddings-v5-omni-small houses four task-specific adapters, letting practitioners switch adapter targets depending on the operational pipeline:

• Cross-Modal Retrieval (`retrieval`): Optimized for asymmetric search. Operators can query document archives, image catalogs, podcasts, or video clips using plain natural language text queries, or query cross-modally (such as finding audio segments via an image prompt). Raw transformer implementations use Query: and Document: task prefixes to direct the attention mechanisms.
• Zero-Shot Classification (`classification`): Enables direct label assignment across modalities by calculating the cosine similarity between media embeddings and label text vectors.
• Topic Discovery and Clustering (`clustering`): Designed for symmetric vector spaces to group unstructured datasets containing mixed media formats, such as customer support tickets containing screenshots, voice notes, and text logs.
• Symmetric Semantic Matching (`text-matching`): Tuned for semantic textual similarity (STS), paraphrase detection, and near-duplicate deduplication.

Supported input formats include standard text sequences, common image formats (.jpg, .png, .webp, .tiff, .heic, .svg), video files (.mp4, .mov, .mkv, .webm), and raw audio files (.wav, .mp3, .flac, .m4a, .opus).

Running jina-embeddings-v5-omni-small locally offers predictable latency and data privacy for sensitive enterprise records, removing external API dependencies and network latency.

At 1.74B parameters, the model is compact, but local deployment planning must account for the active visual and audio encoders alongside the text transformer.

• FP16 / BF16 Execution: Requires roughly 3.8 GB to 4.5 GB of VRAM. An 8 GB graphics card accommodates batching and moderate context depths.
• Q8_0 Quantization: Requires approximately 2.3 GB of VRAM. This retains near-perfect fidelity across all Matryoshka dimensions and task adapters.
• Q4_K_M Quantization: Drops baseline model weight storage to roughly 1.3 GB to 1.6 GB of VRAM, with total operational VRAM sitting around 2.0 GB during active context processing.

For consumer hardware, the model runs easily across modern setups:

• NVIDIA Desktops: An entry-level RTX 3060 (12GB) or RTX 4060 (8GB) provides more than enough headroom to run high-throughput batch encoding.
• High-End Workstations: An RTX 4090 or professional Ada Generation GPU can dedicate spare capacity to running a secondary generative LLM simultaneously.
• Apple Silicon: Systems equipped with M2, M3, or M4 chips (including base 16GB unified memory configurations) handle inference natively via Metal Performance Shaders (MPS) with zero memory bottlenecks.

The best quantization for jina-embeddings-v5-omni-small depends on your downstream vector index:

• If using full 1024-dimension embeddings, standard Q4_K_M provides substantial memory savings with unnoticeable retrieval degradation.
• If leveraging Matryoshka truncation (down to 128 or 64 dimensions), run the model in BF16 or Q8_0. Low-bit quantizations (such as 4-bit) can degrade the subtle numerical variances in early vector dimensions that Matryoshka heads rely upon for rank accuracy.

On a modern GPU like an RTX 4090, jina-embeddings-v5-omni-small tokens per second throughput routinely exceeds 2,500 to 4,000 tokens per second for raw text encoding. On audio and image processing, throughput is primarily bounded by the forward pass of Whisper and SigLIP2. Audio files are processed substantially faster than real-time (often 20x to 50x playback speed), while single image embeddings process in under 20 milliseconds per frame.

Practitioners can serve the model using Hugging Face transformers or sentence-transformers for native Python environments:

1from sentence_transformers import SentenceTransformer
2
3# Load model locally
4model = SentenceTransformer(
5    "jinaai/jina-embeddings-v5-omni-small",
6    trust_remote_code=True
7)
8
9# Encode text and multimodal inputs directly into the shared space
10embeddings = model.encode([
11    "Query: structural mechanical failure analysis",
12    "path/to/diagram.png",
13    "path/to/audio_recording.wav"
14])

For zero-code microservices or shared local endpoints, running the model through llama.cpp backends or deploying via Ollama provides the quickest way to spin up an OpenAI-compatible /v1/embeddings endpoint across your local network.

Evaluating jina-embeddings-v5-omni-small against contemporary embedding models highlights clear tradeoffs in parameter efficiency and cross-modal flexibility:

• jina-embeddings-v5-omni-small vs. jina-embeddings-v5-omni-nano: The smaller nano variant operates at approximately 0.9B parameters. Choose omni-nano when deploying directly to mobile devices, low-power edge nodes, or minimal compute environments (such as a 4GB Raspberry Pi or low-tier edge boxes). Choose omni-small on any machine housing a discrete GPU or unified memory Mac: the 1.74B configuration provides sharper retrieval accuracy and superior zero-shot clustering stability on out-of-domain technical vocabulary.
• jina-embeddings-v5-omni-small vs. CLIP / SigLIP (Standalone): Standard dual-encoder vision-language models like clip-vit-large-patch14 produce high-quality image-text alignment, but they lack audio capabilities, possess a short context window (typically 77 tokens), and produce brittle text-to-text semantic representations. jina-embeddings-v5-omni-small completely outperforms standard CLIP for complex textual understanding, handles 32K context lengths, and natively processes audio alongside video.
• jina-embeddings-v5-omni-small vs. BGE-M3: Flagship text models like BGE-M3 support extensive multilingual text search and multi-function retrieval (dense, sparse, multi-vector). However, BGE-M3 is completely blind to images, video, and audio. If your retrieval corpus is strictly text, BGE-M3 remains an efficient standard. If your data ecosystem involves documentation with mixed charts, audio notes, or video clips, jina-embeddings-v5-omni-small replaces multiple single-modality pipelines with a single 1.74B model deployment.