AI Embeddings Explained: From Text to Vectors

Every modern AI application that deals with text relies on embeddings. Search engines use them to find relevant results. RAG systems use them to retrieve context. Recommendation systems use them to suggest similar content. Yet many developers treat embeddings as a black box. This guide opens the box and shows you how embeddings work, how to generate them, and how to use them effectively.

What Are Embeddings?

An embedding is a list of numbers (a vector) that represents the meaning of a piece of text. Similar texts produce similar vectors. Dissimilar texts produce different vectors. That is the entire concept.

from openai import OpenAI
import numpy as np

client = OpenAI()

def get_embedding(text: str, model: str = "text-embedding-3-small") -> list[float]:
    """Convert text to a vector embedding."""
    response = client.embeddings.create(
        model=model,
        input=text
    )
    return response.data[0].embedding

# Generate embeddings for two sentences
vec1 = get_embedding("The cat sat on the mat")
vec2 = get_embedding("A feline rested on the rug")
vec3 = get_embedding("The stock market crashed yesterday")

print(f"Vector dimension: {len(vec1)}")  # 1536 for text-embedding-3-small
print(f"First 5 values: {vec1[:5]}")

The vector for “The cat sat on the mat” and “A feline rested on the rug” will be close together in the 1536-dimensional space because they mean similar things. The vector for “The stock market crashed” will be far away because it is semantically unrelated.

Measuring Similarity

Cosine similarity is the standard metric for comparing embeddings. It measures the angle between two vectors, ignoring their magnitude. A score of 1.0 means identical direction (same meaning), 0.0 means orthogonal (unrelated), and -1.0 means opposite.

def cosine_similarity(vec_a: list[float], vec_b: list[float]) -> float:
    """Compute cosine similarity between two vectors."""
    a = np.array(vec_a)
    b = np.array(vec_b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

# Compare our three sentences
sim_cat_feline = cosine_similarity(vec1, vec2)
sim_cat_stock = cosine_similarity(vec1, vec3)

print(f"Cat vs Feline: {sim_cat_feline:.4f}")  # ~0.85 (high similarity)
print(f"Cat vs Stock:  {sim_cat_stock:.4f}")   # ~0.15 (low similarity)

You can also use dot product (when vectors are normalized) or Euclidean distance. Cosine similarity is preferred because it is scale-invariant, meaning the length of the text does not skew the comparison.

Batch Embedding Generation

For production workloads, always batch your embedding requests to reduce latency and cost:

def get_embeddings_batch(
    texts: list[str],
    model: str = "text-embedding-3-small",
    batch_size: int = 100
) -> list[list[float]]:
    """Generate embeddings for multiple texts in batches."""
    all_embeddings = []
    
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i + batch_size]
        response = client.embeddings.create(
            model=model,
            input=batch
        )
        # Sort by index to maintain order
        batch_embeddings = [item.embedding for item in sorted(response.data, key=lambda x: x.index)]
        all_embeddings.extend(batch_embeddings)
        print(f"Embedded {min(i + batch_size, len(texts))}/{len(texts)}")
    
    return all_embeddings

# Embed a dataset
documents = [
    "Python is a programming language",
    "JavaScript runs in the browser",
    "Machine learning uses statistical models",
    "Deep learning is a subset of machine learning",
    "React is a JavaScript framework",
    "PyTorch is used for deep learning",
]

embeddings = get_embeddings_batch(documents)
print(f"Generated {len(embeddings)} embeddings")

Application 1: Semantic Search

The most common use case for embeddings is search. Instead of matching keywords, you match meaning.

class SemanticSearch:
    """Simple semantic search using embeddings."""
    
    def __init__(self, documents: list[str]):
        self.documents = documents
        self.embeddings = get_embeddings_batch(documents)
    
    def search(self, query: str, top_k: int = 3) -> list[dict]:
        """Find the most similar documents to a query."""
        query_embedding = get_embedding(query)
        
        # Compute similarity with all documents
        scores = []
        for i, doc_embedding in enumerate(self.embeddings):
            score = cosine_similarity(query_embedding, doc_embedding)
            scores.append({"document": self.documents[i], "score": score, "index": i})
        
        # Sort by similarity (highest first)
        scores.sort(key=lambda x: x["score"], reverse=True)
        return scores[:top_k]

# Build search index
search = SemanticSearch(documents)

# Search by meaning, not keywords
results = search.search("neural networks for computer vision")
for r in results:
    print(f"  [{r['score']:.3f}] {r['document']}")
# Even though "neural networks" and "computer vision" aren't in any document,
# it will find "Deep learning is a subset of machine learning" and
# "PyTorch is used for deep learning" because the meanings are related.

Application 2: Text Clustering

Embeddings let you group similar documents automatically without predefined categories.

from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
import numpy as np

def cluster_documents(
    documents: list[str],
    n_clusters: int = 3
) -> dict[int, list[str]]:
    """Cluster documents by semantic similarity."""
    embeddings = get_embeddings_batch(documents)
    embedding_matrix = np.array(embeddings)
    
    # Cluster using K-means
    kmeans = KMeans(n_clusters=n_clusters, random_state=42, n_init=10)
    labels = kmeans.fit_predict(embedding_matrix)
    
    # Group documents by cluster
    clusters = {}
    for doc, label in zip(documents, labels):
        clusters.setdefault(int(label), []).append(doc)
    
    return clusters

# Example with mixed topics
mixed_docs = [
    "How to train a neural network",
    "Best practices for Python testing",
    "Understanding gradient descent",
    "pytest fixtures and parameterization",
    "Backpropagation algorithm explained",
    "Unit testing with mocks in Python",
    "The weather forecast shows rain tomorrow",
    "Climate change affects global temperatures",
]

clusters = cluster_documents(mixed_docs, n_clusters=3)
for cluster_id, docs in clusters.items():
    print(f"\nCluster {cluster_id}:")
    for doc in docs:
        print(f"  - {doc}")

Application 3: Classification

You can classify text without training a model by comparing embeddings to labeled examples.

def zero_shot_classify(
    text: str,
    categories: dict[str, list[str]]
) -> dict[str, float]:
    """Classify text by similarity to category examples."""
    text_embedding = get_embedding(text)
    
    category_scores = {}
    for category, examples in categories.items():
        example_embeddings = get_embeddings_batch(examples)
        
        # Average similarity to all examples in this category
        similarities = [
            cosine_similarity(text_embedding, ex_emb) 
            for ex_emb in example_embeddings
        ]
        category_scores[category] = sum(similarities) / len(similarities)
    
    return dict(sorted(category_scores.items(), key=lambda x: x[1], reverse=True))

# Define categories with example texts
categories = {
    "technical_support": [
        "My application keeps crashing",
        "I can't log into my account",
        "The API returns a 500 error",
    ],
    "billing": [
        "I was charged twice",
        "How do I update my payment method",
        "Cancel my subscription",
    ],
    "feature_request": [
        "Can you add dark mode",
        "It would be great to have export to PDF",
        "Please support multiple languages",
    ]
}

# Classify new messages
test_messages = [
    "The page won't load and shows a timeout error",
    "I need a refund for last month",
    "Would love to see a mobile app version",
]

for msg in test_messages:
    scores = zero_shot_classify(msg, categories)
    top_category = list(scores.keys())[0]
    print(f"\n'{msg}'")
    print(f"  -> {top_category} ({scores[top_category]:.3f})")

Application 4: Anomaly and Duplicate Detection

Embeddings make it straightforward to find outliers or near-duplicates in a dataset.

def find_duplicates(
    documents: list[str],
    threshold: float = 0.92
) -> list[tuple[int, int, float]]:
    """Find near-duplicate document pairs."""
    embeddings = get_embeddings_batch(documents)
    duplicates = []
    
    for i in range(len(documents)):
        for j in range(i + 1, len(documents)):
            sim = cosine_similarity(embeddings[i], embeddings[j])
            if sim >= threshold:
                duplicates.append((i, j, sim))
    
    return duplicates

def find_anomalies(
    documents: list[str],
    threshold: float = 0.3
) -> list[dict]:
    """Find documents that are outliers compared to the rest."""
    embeddings = get_embeddings_batch(documents)
    embedding_matrix = np.array(embeddings)
    
    # Compute mean embedding (centroid)
    centroid = embedding_matrix.mean(axis=0)
    
    anomalies = []
    for i, (doc, emb) in enumerate(zip(documents, embeddings)):
        sim_to_centroid = cosine_similarity(emb, centroid.tolist())
        if sim_to_centroid < threshold:
            anomalies.append({
                "index": i,
                "document": doc,
                "similarity_to_centroid": sim_to_centroid
            })
    
    return anomalies

Choosing an Embedding Model

Different models offer different trade-offs between quality, speed, cost, and dimensionality.

# OpenAI models
models_comparison = {
    "text-embedding-3-small": {
        "dimensions": 1536,
        "cost_per_million_tokens": 0.02,
        "quality": "Good for most use cases",
    },
    "text-embedding-3-large": {
        "dimensions": 3072,
        "cost_per_million_tokens": 0.13,
        "quality": "Best quality, highest cost",
    }
}

# You can reduce dimensions for text-embedding-3 models
response = client.embeddings.create(
    model="text-embedding-3-large",
    input="Hello world",
    dimensions=256  # Reduce from 3072 to 256
)
compact_embedding = response.data[0].embedding
print(f"Compact embedding dimension: {len(compact_embedding)}")  # 256

For open-source alternatives that run locally:

from sentence_transformers import SentenceTransformer

# Free, local, no API key needed
model = SentenceTransformer("all-MiniLM-L6-v2")  # 384 dimensions, very fast

texts = ["Hello world", "Greetings earth"]
embeddings = model.encode(texts)
print(f"Shape: {embeddings.shape}")  # (2, 384)

# Higher quality local model
model_large = SentenceTransformer("all-mpnet-base-v2")  # 768 dimensions
embeddings_large = model_large.encode(texts)
print(f"Shape: {embeddings_large.shape}")  # (2, 768)

Use OpenAI embeddings when quality matters most and you are already using their API. Use sentence-transformers when you need to run locally, process large volumes without API costs, or need data privacy.

Best Practices

A few lessons from production embedding systems:

Normalize your text. Strip extra whitespace, remove irrelevant formatting, and keep inputs clean. Embeddings are sensitive to noise.

import re

def clean_for_embedding(text: str) -> str:
    """Clean text before generating embeddings."""
    text = re.sub(r'\s+', ' ', text)  # Collapse whitespace
    text = re.sub(r'[^\w\s.,!?;:\-\'\"()]', '', text)  # Remove special chars
    text = text.strip()
    return text[:8000]  # Truncate to avoid token limits

Cache embeddings. Generating embeddings costs money and time. Store them alongside your documents and only regenerate when the text changes.

Match query and document style. If your documents are formal reports but your queries are casual questions, the similarity scores will be lower than expected. Consider prepending “query: ” or “search_document: ” prefixes as some models are trained with these.

Wrapping Up

Embeddings convert text into a mathematical space where meaning can be measured with arithmetic. Generate them with a single API call, compare them with cosine similarity, and use them for search, clustering, classification, and anomaly detection. Start with OpenAI’s text-embedding-3-small for prototyping. Switch to open-source models like sentence-transformers when you need local inference or cost control. The key insight is that once text becomes vectors, an entire toolkit of linear algebra, clustering algorithms, and nearest-neighbor search becomes available for solving problems that keyword matching never could.