Embeddings Explained — How Vector Search Works

Photo: cottonbro CG studio / Pexels
An embedding is a dense numerical vector that represents the semantic meaning of a piece of content — text, image, audio, or code. Two pieces of content with similar meaning produce vectors that are close together in vector space; unrelated content produces vectors that are far apart.
Typical embedding dimensions in 2026 range from 384 (small, fast) to 3072 (high accuracy). Each dimension is a learned feature — not human-readable, but together they encode an extraordinarily rich notion of meaning.
Embeddings are the foundation of semantic search, retrieval-augmented generation (RAG), clustering, recommendation, deduplication, and many other modern AI workflows.
How Embeddings Are Generated
An embedding model is a neural network — typically a Transformer encoder — trained on contrastive learning objectives. The training signal is roughly:
- Pull together vectors of pairs that should mean similar things (e.g., a question and its correct answer, two paraphrases of the same sentence)
- Push apart vectors of pairs that should not (random pairs, hard negatives)
After training, the model maps any input into a fixed-length vector. The resulting space has a remarkable property: cosine similarity (the angle between vectors, scaled to [-1, 1]) correlates with semantic similarity.
import numpy as np
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
A similarity of 0.85+ typically means "closely related"; <0.3 typically means "unrelated."
Top Embedding Models in 2026
| Model | Dimensions | Provider | Notes |
|---|---|---|---|
| text-embedding-3-large | 3072 (truncatable) | OpenAI | High accuracy, supports Matryoshka truncation |
| text-embedding-3-small | 1536 (truncatable) | OpenAI | Cheap, fast, surprisingly strong |
| voyage-3 / voyage-3-large | 1024 | Voyage AI | Specialized for retrieval; recommended by Anthropic |
| embed-v3 (English/Multilingual) | 1024 | Cohere | Excellent multilingual support |
| BGE-M3 | 1024 | BAAI (open source) | Multi-lingual, multi-functional (dense + sparse) |
| Nomic Embed v1.5 | 768 | Nomic (open source) | Matryoshka, fully open weights + data |
| JinaAI embeddings v3 | 1024 | Jina AI | Long context (8K tokens) |
| e5-mistral-7b | 4096 | Microsoft (open) | LLM-derived, top of MTEB |
| gte-large-en-v1.5 | 1024 | Alibaba (open) | Strong open-source baseline |
The MTEB (Massive Text Embedding Benchmark) leaderboard on Hugging Face is the standard reference for comparing models. Specialized domains (code, biomedical, legal) often have dedicated models — e.g., CodeBERT for code, BioBERT for medical text.
What Embeddings Enable
| Application | How It Uses Embeddings |
|---|---|
| Semantic search | Embed query and documents; return nearest neighbors |
| RAG | Retrieve relevant chunks to inject into an LLM prompt |
| Clustering | Group similar items with K-means or HDBSCAN over vectors |
| Classification | Train a small classifier on top of frozen embeddings |
| Deduplication | Detect near-duplicate content above a similarity threshold |
| Recommendation | Recommend items whose vectors are close to user's history |
| Anomaly detection | Flag points far from any cluster |
| Multimodal search | Use shared text+image embedding space (CLIP) to search images with text |
Vector Databases
Once you have millions of vectors, you need a system that supports fast approximate nearest-neighbor (ANN) search. Exact brute-force comparison is O(N) per query — too slow above ~100K vectors.
| Database | Type | Strengths |
|---|---|---|
| Pinecone | Managed cloud | Easiest to use, serverless tier |
| Weaviate | Open source + cloud | GraphQL API, hybrid search, modules |
| Qdrant | Open source + cloud | Fast Rust core, rich filtering |
| Chroma | Open source | Great for prototypes, lightweight Python |
| pgvector | Postgres extension | SQL-native, simplest for existing Postgres apps |
| Milvus / Zilliz | Open source + cloud | Scales to billions of vectors |
| LanceDB | Embedded, open source | Columnar, runs in your process |
| Vespa | Open source | Production search at scale, hybrid retrieval |
| Elasticsearch / OpenSearch | Search engine | Vector + lexical hybrid |
ANN Indexes
Underneath, most vector databases use one of a few index structures:
- HNSW (Hierarchical Navigable Small World) — graph-based, very fast queries, higher memory. Default in most modern DBs.
- IVF (Inverted File Index) — partition vectors into clusters, search only relevant clusters. Used by FAISS.
- ScaNN — Google's optimized index with quantization, very fast on CPU.
- DiskANN — billions of vectors on SSD, used by Microsoft.
You trade off recall (did you find the true nearest neighbors?), latency, memory, and build time with index parameters.
Dimensions Trade-off
Higher dimensions generally mean better accuracy but cost more in storage, memory, and similarity computation.
- 384d vector × 4 bytes (float32) = 1.5 KB per vector
- 3072d vector × 4 bytes = 12 KB per vector
- 1M vectors at 3072d = ~12 GB just for vectors
Workarounds:
- Matryoshka embeddings (Kusupati et al. 2022, used by OpenAI text-embedding-3) — trained so you can truncate to fewer dimensions with graceful degradation
- Quantization (int8, binary) — 4-32x smaller, slight accuracy loss
- Product quantization (PQ) — used by FAISS, IVF-PQ
Multilingual and Multimodal
- Multilingual embeddings (Cohere embed-multilingual-v3, BGE-M3, multilingual-e5) place semantically equivalent text from different languages near each other — enabling cross-lingual search.
- Multimodal embeddings (CLIP from OpenAI 2021, SigLIP from Google) embed images and text into a shared space — you can search photos by text query, or find similar images by image query.
- Audio embeddings (Whisper-based, CLAP) support similar applications for sound and music.
Evaluating Embedding Quality
Don't trust the MTEB leaderboard blindly — evaluate on your own data.
Steps:
- Build a small labeled set: 50-200 queries with known correct documents
- Embed and run retrieval for each query
- Measure recall@K (was the correct document in the top K?) and MRR (mean reciprocal rank)
- Compare candidate models head-to-head
Domain-specific data often inverts the public leaderboard order — a generic top-MTEB model can lose to a smaller, domain-tuned one on, say, legal contracts.
Normalization
Most embedding models output L2-normalized vectors (unit length). When vectors are normalized, cosine similarity is equivalent to dot product and to (negative) Euclidean distance up to a constant. Confirm in the model's documentation, and re-normalize if you concatenate or transform vectors yourself.
Code Example
from openai import OpenAI
client = OpenAI()
def embed(text: str) -> list[float]:
resp = client.embeddings.create(
model="text-embedding-3-large",
input=text,
dimensions=1024 # Matryoshka truncation
)
return resp.data[0].embedding
query_vec = embed("How do I reset my password?")
doc_vecs = [embed(doc) for doc in documents]
import numpy as np
scores = [np.dot(query_vec, dv) for dv in doc_vecs]
top_idx = int(np.argmax(scores))
print(documents[top_idx])
For production, batch the calls and store vectors in a real vector database.
Real-world retrieval augmented generation (RAG) example
The problem: An LLM trained on data up to April 2025 cannot answer questions about events in August 2026 without new knowledge. Hallucination (confident false claims) is common when LLMs lack relevant context.
The RAG solution: Store external documents (internal PDFs, website content, documentation) as embeddings in a vector database. At query time:
- Embed the user's question
- Retrieve the 3-5 most similar document chunks from the vector database
- Inject those chunks into the LLM's context window
- Generate a response grounded in retrieved facts
This gives the LLM access to fresh, domain-specific information without retraining. Examples: ChatGPT's "web search" mode, customer support bots trained on company documentation, medical question-answering on latest research papers.
Source: Lewis et al. (2020), "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," Facebook AI Research.
Embedding fine-tuning and domain adaptation
Generic embeddings trained on general internet text often underperform on specialized domains. A legal embedding model trained on contracts will beat OpenAI's text-embedding-3 on legal query-document pairs.
Fine-tuning approaches:
- Contrastive fine-tuning: Provide pairs of related documents and queries (e.g., 100-200 query-document pairs in your domain) and fine-tune the embedding model to pull these pairs closer together.
- Adapter layers: Add a small trainable projection layer on top of a frozen embedding model.
- Distillation: Train a smaller model to mimic a larger one's embeddings.
Fine-tuned embeddings can improve recall@10 by 10-30% on domain-specific data. The cost is a few hundred labeled pairs and modest compute.
Vector space geometry and nearest-neighbor complexity
Modern embedding spaces are high-dimensional (384-3072 dimensions). Intuitions from 2D/3D geometry fail. Key facts:
- The curse of dimensionality: In high-dimensional spaces, distances are more uniform — even random vectors are surprisingly similar. This makes nearest-neighbor problems harder.
- Approximate Nearest Neighbor (ANN) trade-offs: Exact retrieval is O(N) per query (slow for millions of vectors). ANN algorithms (HNSW, IVF, ScaNN) achieve O(log N) or better by sacrificing recall — you find approximately the closest neighbor, not always the absolute closest.
- Recall-latency trade-off: Tighter search boundaries (higher recall) mean examining more candidates, increasing query latency. You configure index parameters (HNSW ef, IVF nprobe) to tune this trade-off.
For production systems serving millions of vectors with millisecond latency targets, choosing the right index and tuning parameters is critical.
Embedding robustness and failure modes
Embeddings are not magic. They can fail in predictable ways:
Typos and misspellings: Embeddings encode spelling. "teh quick brown fox" embeds differently from "the quick brown fox." Systems need spell-checking or character-level robustness for user-facing applications.
Language shifts: Embeddings trained in 2024 may not handle Aug-2026 language (new slang, new technology names). Models drift.
Domain shift: Medical embeddings trained on journal articles perform poorly on patient forums using lay language.
Short queries: Single-word queries often retrieve poorly because a word alone has ambiguous meaning. "Bank" might retrieve finance articles or riverbank articles. Context matters.
Mitigation strategies: query expansion (add related terms), re-ranking retrieved results with a cross-encoder LM, user feedback loops (RLHF-style) to update retrieval over time.
Frequently Asked Questions
Q: What is the difference between an embedding and a token, and do I confuse them?
A: A token is a discrete symbol (like a subword or word). An embedding is a continuous vector representation. In transformer models, each token gets an input embedding (initialized as a learned vector from a lookup table) and an output embedding (the final hidden state after processing). When people say "embedding model," they usually mean a model that takes variable-length input (a sentence, document, or image) and outputs a single embedding vector. Token embeddings are single vectors for single symbols; embedding models output one vector per document.
Q: How long can I make the input to an embedding model, and what happens with long documents?
A: Most models accept 512-8,192 tokens. OpenAI text-embedding-3 and Jina AI both support 8,192 tokens. For documents longer than the model's context window, the standard approach is chunking: split the document into overlapping chunks (e.g., 1,000 tokens with 100-token overlap), embed each chunk separately, and store all chunks in the vector database. At retrieval time, the system retrieves the most similar chunks and can combine them before injecting into an LLM.
Q: Are embeddings deterministic or random, and can I rely on them over time?
A: For a given model version and input, embeddings are deterministic — you always get the same vector. However, updating a model (even minor version bumps) changes vectors completely. Never mix embeddings from different model versions in the same vector database; you cannot meaningfully compare similarity between vectors from Model A v1 and Model B v2. If you upgrade your embedding model, you must re-embed and re-index all stored documents.
Q: Should I use open-source or closed-source (proprietary API) embedding models?
A: Closed-source (OpenAI, Voyage, Cohere) offer slightly better accuracy and are easiest to use (just call an API). Open-source (BGE-M3, Nomic, e5-mistral, available on Hugging Face) can be self-hosted, fine-tuned, and have no per-call costs. For prototypes and when you care most about accuracy, closed-source is simpler. For production with high volume, cost sensitivity, or domain-specific data, open-source is often better. Many teams use both—closed-source models for high-accuracy applications, open-source for bulk indexing.
Q: Do I actually need a vector database, or can I just use NumPy for similarity search?
A: Under 50,000 vectors with no real-time concurrency requirements, NumPy brute-force is fine and fastest to develop. Compute similarity between query and all stored vectors in milliseconds. Above that scale (100K-100M vectors), you need a vector database with an ANN index (HNSW, IVF) to get sub-second latency. pgvector (Postgres extension) is a good middle ground if you already use Postgres; Qdrant and Pinecone are purpose-built and faster.
Q: What does a cosine similarity score of 0.8 or 0.5 mean practically, and how do I calibrate thresholds?
A: Cosine similarity is heuristic and model-dependent. For OpenAI text-embedding-3, scores >0.5 typically mean "relevantly related"; >0.7 means "very closely related." For some models, random pairs score >0.6. Calibrate on your own data: build a small ground-truth dataset (100-200 query-document pairs labeled as relevant/irrelevant), compute similarities, plot a histogram, and choose a threshold. Typical production thresholds range from 0.4-0.7 depending on domain, model, and tolerance for false positives.
Embedding Drift and Temporal Stability
A subtle but critical issue in production embedding systems: embedding drift. When an embedding model is updated (even minor version bumps), all vectors change completely. A vector from Model A v1.0 is not comparable to a vector from Model A v1.1, even for identical input.
The problem: If you built a vector database in January 2025 with 100 million embeddings, then upgrade your embedding model to a new version in August 2026, the old vectors become useless. Similarity comparisons across versions fail catastrophically. You cannot mix old and new embeddings.
Real-world impact: A company running a production RAG system must choose: (1) live with an outdated embedding model forever (degrading quality), or (2) rebuild the entire vector database whenever a better model is released (expensive and risky). This locks organizations into embedding model choices, creating switching costs.
Mitigation strategies:
- Matryoshka embeddings solve this partially: if a new model is Matryoshka-aware, you can sometimes migrate by truncating old embeddings to the new dimension space. Not always reliable.
- Versioning discipline: Embed a model version identifier in your database schema. Know which vectors came from which model. If you upgrade, re-embed incrementally on a schedule.
- Multi-model ensembles: Some systems keep embeddings from multiple models and average similarity scores. Slower but more robust to model drift.
- Relative stability: some models (Cohere's, OpenAI's) are more stable across versions than others. Consider stability when selecting a model, not just accuracy.
This is rarely discussed in marketing materials but is a major practical headache.
Multilingual, Multimodal, and Specialized Embeddings
The embedding ecosystem has expanded far beyond text:
Multilingual embeddings (Cohere embed-3-multilingual, BGE-M3, e5-multilingual) solve a different problem: placing text in many languages into a shared vector space so that semantically equivalent sentences in different languages are nearby. "How do I reset my password?" (English) and "¿Cómo restablezco mi contraseña?" (Spanish) should have similar vectors. This enables cross-lingual search and machine translation assistance. These are trained on parallel corpora and are generally effective; slight accuracy trade-offs compared to language-specific models but massive utility for global applications.
Multimodal embeddings (CLIP, SigLIP, CoCa) embed images and text into a shared space. A human can query an image database with text: "cat sleeping on a couch." The text and image embeddings are in the same space, so cosine similarity works across modalities. These revolutionized image search and enable applications like visual question answering. Limitations: these models are often trained on broad web data; specialized images (medical scans, scientific microscopy) often require domain-specific multimodal models.
Specialized embeddings exist for code (CodeBERT, github.com's CodeSearchNet), biomedical text (BioBERT, SciBERT), and legal documents. These are often distilled or fine-tuned from larger models on domain-specific corpora. For a legal document search system, domain-specific embeddings almost always beat generic ones.
Audio embeddings (CLAP — Contrastive Language-Audio Pre-training) embed audio clips and text descriptions into a shared space. Less mature than image-text embeddings but rapidly improving. Use cases include audio search and sound event detection.
Streaming and Real-Time Embedding Systems
Most production systems we've discussed assume batch processing: embed a corpus offline, build an index, query at search time. But some applications need to embed in real time as data arrives.
Stream ingestion: A company wants to index customer support tickets as they arrive. Each ticket should be immediately searchable. The embedding system must:
- Embed the ticket (latency: 10-100ms depending on model size and hardware).
- Index it in the vector database (latency: 1-10ms).
- Have it searchable immediately (no batch wait).
This requires low-latency embedding services (often running on GPUs or specialized hardware) and real-time vector database updates. Streaming databases like Kafka can buffer and replay traffic; embedding services can keep up. But if embedding latency exceeds ingestion rate, backlogs form.
Online learning: Some systems allow the embedding model itself to adapt over time based on user feedback (e.g., if a user marks a retrieved result as irrelevant, update the model weights to deprioritize that similarity in future). This is RLHF (Reinforcement Learning from Human Feedback) applied to embeddings. Few systems do this well; most use static models with post-hoc re-ranking.
Embedding Interpretability and Explainability
A significant limitation: embedding vectors are opaque. Each dimension is a learned feature — not human-interpretable. You cannot ask "what does dimension 42 mean?" and get a useful answer.
Why this matters: In high-stakes applications (medical diagnosis, hiring, lending), systems must explain decisions. If an embedding-based search returns a medical diagnosis, regulators may demand to know why. "The 1,024-dimensional vector was similar" is not a satisfactory explanation.
Approaches to interpretability:
- Dimensionality reduction for visualization (t-SNE, UMAP): Project 1,024-dimensional vectors to 2D for visualization. Reveals clusters but loses information; clusters in 2D may not reflect true high-dimensional structure.
- Attention weights: If the embedding model is a Transformer (most are), attention weights can show which input tokens contributed most to the embedding. Partially interpretable.
- Retrieval explainability: Show the top-K similar documents retrieved for a query. Users can inspect why each was chosen (though the algorithm itself remains opaque).
- Concept-based embeddings: Some research models embed text as a weighted combination of interpretable concepts. Bleeding-edge and rarely production-ready.
Most production systems accept this opacity in exchange for accuracy. Interpretability remains an open research problem.
Embedding Quality Metrics and Benchmarking
We mentioned MTEB leaderboard, but in practice, you evaluate embeddings on your own data. Standard metrics:
Recall@K: For a query, did the true relevant document appear in the top K retrieved? Recall@10 = 90% means 90% of queries had the relevant document in top 10.
MRR (Mean Reciprocal Rank): Average of 1/rank of the first relevant result. If the relevant doc is rank 1, MRR=1.0; rank 5, MRR=0.2. Penalizes models that get the right doc but rank it lower.
NDCG (Normalized Discounted Cumulative Gain): Assumes multiple relevant documents with varying relevance scores. Penalizes worse (lower rank) placements more heavily. Standard in ranking systems.
MAP (Mean Average Precision): Average of precision at each rank where a relevant doc appears. Common in information retrieval.
Domain-specific metrics: If you're recommending products, you might care about conversion rate (did the user buy the recommended product?) more than similarity scores. Connect embedding quality to business metrics.
The caveat: MTEB rankings are aggregated over dozens of tasks. A model top-ranked on MTEB might underperform on your specific domain. Micro-benchmarking (evaluating on your exact use case) always trumps macro-benchmarks.
Source: Muennighoff et al. (2022). "MTEB: Massive Text Embedding Benchmark"; Thakur et al. benchmarking papers on retrieval systems.
Production Deployment: From Lab to Scale
Taking embeddings into production involves infrastructure beyond the model:
Serving: Embedding inference must be fast and scalable. Options:
- Cloud APIs (OpenAI, Cohere): Simple, reliable, pay-per-call. Latency ~100ms, costs scale with volume.
- Self-hosted: Run the embedding model on your own servers. Lower per-call cost at high volume, but requires GPU infrastructure, monitoring, and failover.
- Hybrid: Expensive queries use the cloud API; routine queries use a self-hosted smaller model. Complex to manage.
Latency requirements: For search, users tolerate ~100-500ms end-to-end. If embedding takes 100ms and database query takes 50ms, that leaves only 350ms for the rest of the pipeline. Latency budgeting matters.
Monitoring: Track embedding quality over time. Drift detection: if your embedding model updates, re-embed a subset and measure similarity drift. Alert if drift exceeds thresholds.
Cost management: If you embed 1 billion documents at $0.02 per million tokens (typical), that's $20,000 upfront. Recurring embedding of new documents adds ongoing cost. Open-source models reduce per-call cost to near-zero but require infrastructure.
Most production systems use a mix: self-hosted open-source for bulk embedding (cost-efficient), cloud APIs for real-time queries (low-latency).
The takeaway
Embeddings—dense numerical vectors representing semantic meaning—are the foundation of modern AI applications from search to recommendation to retrieval-augmented generation. By converting documents and queries into a shared vector space where similarity correlates with meaning, embeddings enable systems to understand content semantically rather than just matching keywords. The embedding model you choose, the vector database architecture, how you fine-tune and calibrate thresholds on domain-specific data, and how you manage model drift and scale inference all matter far more than theoretical leaderboard rankings. Production embedding systems are complex: they require attention to temporal drift, multimodal capabilities, real-time streaming, and cost/latency trade-offs. As RAG systems become standard for giving LLMs access to fresh knowledge, embeddings will remain central to AI infrastructure, and mastering their practical deployment will remain a key differentiator.
Read next
Follow new articles
Add SciHub101 to your feed reader to see new science and technology stories.
About the Author
SciHub101 is an independent science and technology publication founded and edited by Buğra Sözeri, a software engineer and app developer based in Türkiye. AI tools assist with drafting and research. Source documentation in the existing archive is uneven, and articles are being reviewed and corrected. The Editorial Team byline identifies editorial responsibility; it does not certify specialist or independent human review. Where we compare products, anything built by our own team is labelled as such — see the editorial guidelines and About page for how corrections and disclosures work.
Related Articles

What Is llms.txt? AI-Friendly Site Files
llms.txt is a markdown file proposed by Jeremy Howard in 2024 that helps LLMs understand your site. Format, examples, and 2026 adoption status.

Context Window Explained — LLM Memory Limits
Context windows define how much text an LLM can process at once. Learn token limits, the lost-in-the-middle problem, and 2026 model comparisons.

What Is RAG? Retrieval-Augmented Generation Explained
RAG is the technique making AI tools actually useful for real work — letting models look things up instead of relying on memory. Here's how it works and why it matters.

What Is Cohere? Enterprise LLM Provider
Cohere is the enterprise-focused LLM provider founded by a transformer co-author. Learn its models, strengths, and how it compares to OpenAI and Anthropic.

LangChain vs LlamaIndex — LLM Framework Compared
LangChain and LlamaIndex both build LLM apps in Python. Differences, strengths, code examples, and when to pick each in 2026.

What Is a Vector Database? Beyond SQL
Vector databases store embeddings and run similarity search via ANN indexes. Pinecone, Weaviate, Qdrant, pgvector — compared for 2026.