RAG with Vector Databases: Embeddings, HNSW, and Filters to Stop Hallucinations

Have you ever asked an AI assistant a question about your company’s internal policies, only to get a confident but completely wrong answer? That moment of disappointment is what we call hallucination. It happens because Large Language Models (LLMs) are trained on static data from the past. They don’t know your latest product updates, your private customer records, or the new regulations that passed yesterday. Worse, they might just make things up to sound helpful.

This is where Retrieval-Augmented Generation, or RAG, comes in. Think of RAG as giving your AI a library it can actually read before answering. Instead of relying solely on its memory, the system first retrieves relevant facts from your own data sources-like databases, documents, or data lakes-and feeds them into the prompt. This grounds the AI in reality, drastically reducing hallucinations and ensuring your answers are accurate and up-to-date.

But how do you find the right fact among millions of documents in milliseconds? That’s the job of vector databases, embeddings, and smart indexing algorithms like HNSW. Let’s break down how this architecture works, why it matters for your business, and how to build it without getting lost in the technical weeds.

What Are Embeddings and Why Do They Matter?

To understand RAG, you first need to understand embeddings. An LLM doesn’t “read” text like you do. It sees numbers. An embedding is a mathematical representation of text, images, or other data converted into high-dimensional vectors-essentially long arrays of numbers. These numbers capture the semantic meaning of the content.

Imagine you have two sentences: "The dog chased the cat" and "A canine pursued a feline." To a human, these mean the same thing. To a basic keyword search engine, they share almost no words. But to an embedding model, these sentences will result in vectors that are very close together in mathematical space. This proximity signals similarity.

In a RAG system, the process starts by taking your domain-specific dataset and chunking it into smaller, semantic elements. You then use a specialized embedding model to compute a vector for each chunk. Popular choices include open-source models like all-MiniLM-L12-v2 from the SentenceTransformer package, which produces efficient 384-dimensional vectors, or cloud-native options like Amazon Titan Text Embedding v2 for production environments on AWS Bedrock.

When a user asks a question, their query is also turned into a vector using the exact same model. The system then searches for the vectors in your database that are closest to the query vector. This ensures consistency in the semantic space. If the embedding model changes between indexing and querying, the math breaks, and your retrieval accuracy plummets.

The Role of Vector Databases in RAG

You could store these vectors in a standard spreadsheet or a traditional relational database, but that would be painfully slow. Traditional databases are built for exact matching-finding a record where ID equals 123. They aren’t designed to calculate the distance between two points in a 384-dimensional space across millions of rows.

Vector databases are specifically architected for this workload. They serve as the long-term memory for your LLM, storing millions to billions of embeddings and retrieving the top-k nearest neighbors in milliseconds. Whether you use a purpose-built solution like Pinecone or an extension like Pgvector for PostgreSQL, the goal is the same: perform similarity searches efficiently.

Pgvector, for instance, adds a "vector" data type to PostgreSQL along with three query operators for similarity searching:

• Euclidean distance: Measures the straight-line distance between points.
• Cosine distance: Measures the angle between vectors, which is often better for text embeddings.
• Negative inner product: An alternative metric useful in specific contexts.

Most modern text embedding models are optimized for cosine similarity. It focuses on the orientation of the vectors rather than their magnitude, which aligns well with how semantic meaning is distributed in high-dimensional spaces.

HNSW Indexing: Speeding Up Search Without Sacrificing Accuracy

Here’s the problem with naive similarity search: if you have one million vectors, the database has to compare your query against all one million vectors to find the best match. This is called a brute-force search, and it takes seconds. In a real-time chat application, seconds feel like hours. Users bounce.

This is where Hierarchical Navigable Small World (HNSW) indexing saves the day. HNSW is a graph-based algorithm designed for approximate nearest neighbor (ANN) searches. Instead of checking every single vector, HNSW organizes your data into a multi-layered graph structure.

Think of it like navigating a city. The top layer of the graph has long-range connections that let you jump quickly across large distances to get near your destination. As you move down the layers, the connections become shorter and more detailed, allowing you to fine-tune your location until you land on the most similar vector.

The trade-off here is between speed and recall. Recall refers to how many of the truly relevant results you actually find. HNSW allows you to tune parameters to balance this. For most enterprise applications, the slight drop in theoretical perfection is worth the massive gain in speed.

Consider a real-world test case using Pgvector with one million rows of 384-dimensional embeddings. Building an HNSW index on this dataset took about 33 minutes. But once indexed, similarity search times dropped from several seconds per query to mere milliseconds. That’s a 100x+ performance improvement. This makes real-time RAG commercially viable.

Alternative Indexing Methods: IVFFlat vs. HNSW

HNSW isn’t the only game in town. Another common method is IVFFlat (Inverted File with stored vectors). IVFFlat uses geometric partitioning to break your vector space into small, queryable sub-indexes or clusters. When a query comes in, the system only searches the specific partitions that are likely to contain the answer, ignoring the rest.

So, which one should you choose? Here is a quick comparison:

If you need the absolute fastest response times and can afford the extra memory and build time, HNSW is usually the winner. If you are dealing with massive datasets where even approximate clustering helps, IVFFlat might be a lighter option. Some advanced systems even combine methods using techniques like Reciprocal Rank Fusion (RRF) to boost relevance further.

Filtering: Adding Business Logic to Semantic Search

Semantic similarity is powerful, but it’s not enough on its own. Imagine a multi-tenant SaaS platform. Company A shouldn’t see the documents belonging to Company B, even if those documents are semantically similar to the query. Or consider compliance: you might want to exclude sensitive HR documents from general employee queries.

This is where filtering mechanisms come in. Vector databases allow you to constrain similarity searches based on metadata. Before or during the vector search, you apply filters like `tenant_id = '123'` or `document_type != 'confidential'`. This ensures that the retrieved results are not only semantically relevant but also contextually appropriate.

The challenge with filtering is performance. If you filter too aggressively, you might end up searching an empty subset of your index. Modern vector databases handle this by integrating metadata indexes alongside vector indexes, allowing for efficient filtered searches across large datasets without scanning the entire space.

Building Your RAG Pipeline: Step-by-Step

Implementing RAG requires careful architectural decisions. Here is a practical workflow based on industry best practices:

1. Chunking: Break your source documents into manageable pieces. Too large, and you lose precision; too small, and you lose context. Aim for semantic completeness in each chunk.
2. Embedding: Pass each chunk through your chosen embedding model (e.g., all-MiniLM-L12-v2). Store the resulting vectors along with the original text and metadata (like author, date, tenant ID).
3. Indexing: Create an HNSW index on your vector column. Tune the `m` and `ef_construction` parameters if you’re using Pgvector to balance build speed and search quality.
4. Query Processing: When a user asks a question, embed the query using the same model.
5. Retrieval: Query the vector database for the top-k nearest neighbors, applying any necessary metadata filters.
6. Generation: Append the retrieved chunks to your LLM prompt. Ask the LLM to answer the question based only on the provided context.

A pro tip: Keep your embeddings close to your source data. If you’re using PostgreSQL with Aurora ML, you can generate embeddings directly within the database using stored procedures. This eliminates the need to export data for transformation and keeps your pipeline simple and secure.

Common Pitfalls to Avoid

Even with the right tools, RAG implementations can fail if you ignore these common traps:

• Inconsistent Embedding Models: Never mix and match embedding models for indexing and querying. The vector spaces won’t align.
• Ignoring Chunk Size: Poor chunking leads to fragmented context. Experiment with different chunk sizes and overlap strategies.
• Overlooking Latency: Without proper indexing (like HNSW), your app will be too slow for users. Always benchmark your search times.
• Forgetting Filters: Security and privacy rely on metadata filters. Test them rigorously to prevent data leakage between tenants or roles.

RAG with vector databases is no longer just a research concept. It’s the backbone of reliable, enterprise-grade generative AI. By mastering embeddings, leveraging HNSW for speed, and applying strict filters, you can build AI applications that are not only smart but also trustworthy.