# Vector Databases by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm) In [Embeddings](https://www.kirupa.com/ai_fundamentals/embeddings.htm), Marcus typed ***my tent zipper won't budge*** into Trailhead Help, and the assistant found ***Fixing a jammed slider*** because the question and the article landed close together on a map of meaning. We breezed right past one step in that story, though. We said we'd ask the index for the stored embeddings closest to the question, and we moved on as if that were the easy part. With three articles on a toy map, it is the easy part. Trailhead's real index is a different story. In [Reranking](https://www.kirupa.com/ai_fundamentals/reranking.htm), we'll see that it holds about 40,000 chunks, and every one of those chunks has an embedding of 1,536 numbers. Every time a customer asks something, our code has to find the handful of chunks whose embeddings sit closest to the question's, and it has to do that quickly, because the customer is sitting there waiting. The thing that does this job is a **vector database**. In the following sections, we'll look at what one stores, why the obvious way of searching gets slow, the clever indexes that make it fast (and what they give up), and how to pick one. Onwards! ## What Goes in a Vector Database A **vector** is the math name for a list of numbers, which is exactly what an embedding is. So a vector database is a database built to store those lists and to quickly find the ones closest to a new list. It stores more than just the vectors, though. Here is what one of Trailhead's chunks looks like once it's stored: ![One chunk stored as one row. The ID, warranty-tents-1, is how we find it again. The vector, a list of 1,536 numbers starting 0.021, -0.113, is what we search. The text, Tents are covered against manufacturing defects, is what we hand the model. The metadata, doc: Warranty PDF and status: current, is for filtering.](https://www.kirupa.com/ai_fundamentals/images/vector_databases_record.png) Starting at the top, the ID is how we find the chunk again later. The vector is the 1,536 numbers we search on. The text is what we'll actually hand the model, since the model reads words and has no use for our list of numbers. The metadata is a set of labels, like which document the chunk came from and whether it's still current, that we can filter on. ## Checking Every Vector The simplest way to search is to compare the question's vector with every single stored vector, using the cosine similarity from Embeddings, and then keep the top few. This is called **exact search** (aka brute force, or a flat index), and it's exactly as thorough as it sounds: ![Marcus's question is compared with every stored chunk, one by one, and each comparison produces a similarity score: 0.41, 0.77, 0.998, 0.12, 0.65, and so on for all 40,000 chunks. The best score, 0.998, belongs to Fixing a jammed slider.](https://www.kirupa.com/ai_fundamentals/images/vector_databases_brute_force.png) Marcus's question gets a score against every chunk: 0.41 here, 0.77 there, and 0.998 for Fixing a jammed slider. Exact search never misses, because it looks at everything. The catch is the amount of arithmetic. Each comparison multiplies 1,536 pairs of numbers, so one search through Trailhead's 40,000 chunks takes 61,440,000 multiplications, and that's for a single question. Computers are fast, and for 40,000 chunks, that's honestly fine. The trouble is that the work grows right along with the collection, and all of it happens again for every question: | Chunks in the index | Multiplications per question | |---|---| | 40,000 | 61,440,000 | | 1,000,000 | 1,536,000,000 | | 10,000,000 | 15,360,000,000 | A company with years of documents and support tickets can end up with millions of chunks, and checking all of them for every question on a busy Saturday gets slow and expensive. So most vector databases cheat a little. ## Close Enough Is Good Enough When Marcus asks his question, we don't actually need the mathematically perfect top five. If the search hands back four of the true top five plus the sixth-best chunk, Marcus is very unlikely to notice, especially once [Reranking](https://www.kirupa.com/ai_fundamentals/reranking.htm) reorders things anyway. Giving up that little bit of accuracy buys a huge amount of speed. Searches that make this trade are called **approximate nearest neighbor** searches (aka ANN), and the two most common ways to do them are called HNSW and IVF. ### HNSW: Hopping Across a Graph Imagine every chunk as a point on our map, and imagine that ahead of time, we link each point to a handful of its nearest neighbors. What we have now is a **graph**: points, plus links between the points that are close to each other. To search it, we start at some point and look at its neighbors. We hop to whichever neighbor is closest to the question. Then we look at that point's neighbors and do it again, until no neighbor is any closer than the point we're standing on: ![Chunks are points, and each point is linked to a few of its nearest neighbors. The search starts at one point and, at every hop, moves to whichever linked neighbor is closest to Marcus's question. After four hops, no neighbor is any closer, and the search stops at the point nearest the question, having checked only a handful of the points.](https://www.kirupa.com/ai_fundamentals/images/vector_databases_hops.png) Follow the green line from Start. Each numbered hop moves to a neighbor that's closer to Marcus's question. After hop 4, the search reaches the closest chunk and stops, because none of that chunk's neighbors are any closer. Along the way, the search only looked at the points on its path and their neighbors, which is a small slice of the whole graph. There's a problem with this, though. On a graph with millions of points and only short links, the starting point might be very far away from the question, and hopping one short link at a time would take forever. The fix is to add express layers on top, a little like taking the highway for most of a trip and local streets for the last bit. The top layer has only a few points, joined by long links. Each layer below it has more points and shorter links, and the bottom layer has every point: ![Three layers of the same graph. The top layer has only three points with long links, the middle layer has eight, and the bottom layer has every point. The search starts at the top, takes one long hop toward Marcus's question, drops down a layer, hops again, drops to the bottom layer, and takes one last short hop to the closest chunk.](https://www.kirupa.com/ai_fundamentals/images/vector_databases_hnsw.png) The search starts at the top layer, where one long hop gets it into the right part of the map. Then it drops to the middle layer and hops again. Finally, it drops to the bottom layer, where one short hop lands on the closest chunk. This layered graph is called a **Hierarchical Navigable Small World** graph (aka **HNSW**), from a [paper by Malkov and Yashunin](https://arxiv.org/abs/1603.09320) that was first posted in 2016. The ***small world*** part refers to networks where we can get from any point to any other point in just a few hops, like the six degrees of separation idea. HNSW is the workhorse of vector search at the time of writing, and pgvector, Qdrant, Weaviate, Elasticsearch, and OpenSearch all support it. As [pgvector's docs](https://github.com/pgvector/pgvector) point out, its downsides are that it's slow to build and it likes a lot of memory. ### IVF: Searching Only the Closest Groups The other popular approach skips the graph. Ahead of time, it sorts all of the vectors into groups of neighbors and gives each group a center. At question time, it compares the question with the centers first, and then it searches only inside the few groups whose centers are closest: ![Ahead of time, the chunks are sorted into four groups, each with a center. At question time, we compare Marcus's question with the four centers and search only the two closest groups. That's fast, but the chunk that is truly closest to the question sits just over the border in a group we skipped, so the search misses it and returns the best chunk from the searched groups instead.](https://www.kirupa.com/ai_fundamentals/images/vector_databases_ivf.png) The question is closest to the centers of the two Searched groups, so the index looks inside those two and skips the other two. That's a lot less work than checking everything. Now look at the red point. That chunk is truly the closest one to the question, but it sits just over the border in a group the index skipped. So the search never sees it, and it returns Best found instead. This approach is called an **inverted file index** (aka **IVF**). Meta's [Faiss library](https://github.com/facebookresearch/faiss/wiki/Faster-search), released in 2017, includes it, and pgvector offers it as an index type called IVFFlat. pgvector's docs note that HNSW gets a better balance of speed and accuracy, while IVFFlat builds faster, uses less memory, and needs some data in the table first, so it can work out its groups. ### The Speed and Recall Dial Both kinds of index come with a dial for how hard to look. In HNSW, the dial is how many candidates the search keeps track of as it hops, a setting often called `ef_search`. In IVF, the dial is how many groups to search, often called `probes` or `nprobe`: ![A slider for how many candidates the index checks: ef_search for HNSW, probes for IVF. Turned toward check fewer, searches are faster but may miss a few of the truly closest chunks. Turned toward check more, searches are slower but find nearly all of them.](https://www.kirupa.com/ai_fundamentals/images/vector_databases_dial.png) When we turn the dial toward Check fewer, searches get faster, but they may miss a few true neighbors. When we turn it toward Check more, searches get slower, but they find nearly all of them. pgvector's docs put it plainly for HNSW: ***"A higher value provides better recall at the cost of speed"***. At the time of writing, its defaults are 40 candidates for HNSW and 1 group for IVFFlat. Turn IVF's dial all the way up so it searches every group, and we're back to exact search. The word **recall** in that quote has a plain meaning: of the chunks that are truly closest, how many did the index actually return? If the index returns 9 of the 10 truly closest chunks, its recall is 90%. The same word comes up again for the whole retrieval pipeline, and it gets a full article of its own in [Recall vs. Precision](https://www.kirupa.com/ai_fundamentals/recall_vs_precision.htm). ## Measuring "Close" So far, we've measured closeness with cosine similarity, which looks at the angle between two vectors. Vector databases usually support a couple of other measures too, and it helps to know what each one pays attention to: ![The same two vectors, A and B, drawn from the origin three times. Cosine looks only at the angle between them. The dot product multiplies and adds their numbers, so it cares about the angle and how long the vectors are. Euclidean distance is the straight-line gap between their tips, and smaller means closer.](https://www.kirupa.com/ai_fundamentals/images/vector_databases_distances.png) Cosine looks only at the angle between A and B, so it ignores how long the lines are. The dot product multiplies the two lists of numbers pair by pair and adds everything up, so it cares about the angle and also rewards longer vectors. Euclidean distance is the straight-line gap between the tips of A and B, so for this one, a smaller number means closer. Here's what each measure says about Marcus's question and the three articles from our toy map in Embeddings: | Article | Cosine (higher is closer) | Dot product (higher is closer) | Euclidean (lower is closer) | |---|---|---|---| | Fixing a jammed slider | 0.998 | 1.35 | 0.224 | | Choosing a tent for two people | 0.821 | 0.79 | 0.608 | | Patching a sleeping pad | 0.738 | 0.71 | 0.728 | All three measures put the jammed slider article first, and here they even agree on the whole order. With real embeddings, they often agree too, because many embedding models return vectors that all have the same length. [OpenAI's embeddings](https://developers.openai.com/api/docs/guides/embeddings), for example, ***"are normalized to length 1"***, and OpenAI's docs point out that cosine similarity and Euclidean distance then produce identical rankings, and that a plain dot product gives the same answer as cosine, just slightly faster. They recommend cosine similarity and say the choice typically doesn't matter much. The practical rule is to use the measure that your embedding model's docs recommend, and to build the index with that same measure. In pgvector, for example, an HNSW index is built for one specific measure, like `vector_cosine_ops` for cosine distance. ## Filtering While Searching Remember the labels in each row? They're what let us say ***only search the current Canadian policies***, which matters a lot to Leah, a customer in Toronto we'll meet properly in [Metadata Filtering](https://www.kirupa.com/ai_fundamentals/metadata_filtering.htm). How a vector database applies that kind of filter makes a surprisingly big difference. pgvector's docs give a nice concrete example: with an HNSW index and its default of 40 candidates, if a filter matches 10% of the rows, only about 4 of those 40 candidates will match. Here is that problem for Leah, if 1 in 10 of Trailhead's chunks is tagged CA: ![Two panels for Leah in Canada, where 1 in 10 chunks is tagged CA. Filter after: the index finds its 40 best candidates first, and only 4 of them are CA, so 4 results are left. Filter during: the index only considers CA chunks, so all 10 results are CA.](https://www.kirupa.com/ai_fundamentals/images/vector_databases_filter.png) With Filter after, the index finds its 40 best candidates first, and only then does the filter throw away everything that isn't CA. Leah is left with 4 results, and the best Canadian chunk might not even be one of them. With Filter during, the search only looks at CA chunks in the first place, so Leah gets a full set of 10. Most vector databases can filter during the search in some form. [Qdrant](https://qdrant.tech/documentation/concepts/filtering/) filters on a chunk's labels (which it calls the payload) while it searches, and pgvector added iterative index scans in version 0.8.0, which keep scanning until enough rows match. Metadata Filtering digs into when to filter and where the filter values should come from. ## The Options at the Time of Writing There are a lot of vector databases, and the list changes quickly. Rather than ranking them, it helps to sort them by where they live: ![Four groups of options at the time of writing. Inside a database you already run: pgvector for Postgres, Elasticsearch, and OpenSearch. Inside your app, with no server: Chroma and LanceDB. Vector databases you can run yourself: Qdrant, Weaviate, and Milvus. Managed only: Pinecone.](https://www.kirupa.com/ai_fundamentals/images/vector_databases_options.png) Here's a little more about each one, going by their own docs at the time of writing: | Option | What it is | Where it runs | |---|---|---| | [pgvector](https://github.com/pgvector/pgvector) | An open-source extension that adds vector search to Postgres (aka PostgreSQL), with exact search, HNSW, and IVFFlat | Any Postgres we run, including managed ones like Amazon RDS, Google Cloud SQL, Azure Database for PostgreSQL, Supabase, and Neon | | [Elasticsearch](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dense-vector) | A search engine with a vector field type and approximate search that uses HNSW | Wherever we run Elasticsearch | | [OpenSearch](https://docs.opensearch.org/latest/vector-search/) | An open-source search engine, now under the Linux Foundation, with HNSW and IVF through its Faiss and Lucene engines | Wherever we run OpenSearch | | [Chroma](https://docs.trychroma.com/docs/overview/introduction) | An open-source database that can run inside a Python app, in memory or saved to disk, or as its own server | Our app or our server. Its hosted [Chroma Cloud is winding down](https://www.trychroma.com/cloud-wind-down), with reads ending on December 22, 2026, while the open-source project continues | | [LanceDB](https://github.com/lancedb/lancedb) | An open-source library that runs inside our process with no server, built on the Lance columnar format | Our app, with data on a local disk or in cloud storage, plus LanceDB Enterprise for a managed setup | | [Qdrant](https://qdrant.tech/documentation/) | An open-source vector database written in Rust that uses HNSW and filters on labels while searching | Our servers, or Qdrant Cloud | | [Weaviate](https://docs.weaviate.io/weaviate) | A mostly open-source vector database written in Go, with HNSW by default and built-in hybrid search | Our servers, or Weaviate Cloud | | [Milvus](https://milvus.io/docs/install-overview.md) | An open-source vector database and a graduated LF AI & Data Foundation project, with many index types, including HNSW, IVF, and GPU indexes | A Python library (Milvus Lite), one server, a whole cluster, or Zilliz Cloud | | [Pinecone](https://docs.pinecone.io/guides/get-started/overview) | A fully managed vector database that isn't open source | Pinecone's cloud, though Enterprise customers can [run the data part in their own cloud account](https://docs.pinecone.io/guides/production/bring-your-own-cloud) | Notice that three of these aren't separate databases at all. pgvector, Elasticsearch, and OpenSearch add vector search to something a lot of teams already run. ## How to Choose With that many options, here's the path I'd walk for Trailhead, and for most teams that are just getting started: ![A decision path. Already run Postgres? Then pgvector. Otherwise, already run Elasticsearch or OpenSearch? Then their vector search. Otherwise, just trying it out? Then Chroma or LanceDB. Otherwise, want someone else to run it? Then a managed service. If not, run Qdrant, Weaviate, or Milvus on your own servers.](https://www.kirupa.com/ai_fundamentals/images/vector_databases_choose.png) The first two questions are about what we already run. Let's say Trailhead's store already keeps its products and orders in Postgres. Adding pgvector puts the chunks right next to that data, so there's one database to back up, one set of permissions to manage, and plain SQL for filters like `region = 'CA'`. Here's roughly what that looks like: ```sql CREATE EXTENSION vector; CREATE TABLE chunks ( id bigserial PRIMARY KEY, doc text, region text, content text, embedding vector(1536) ); CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops); SELECT doc, content FROM chunks WHERE region = 'CA' ORDER BY embedding <=> $1 LIMIT 5; ``` The `<=>` operator is pgvector's cosine distance, and `$1` is where the question's embedding goes. The same thinking applies if Trailhead already runs Elasticsearch or OpenSearch for its site search. Their vector search sits right next to the keyword index, which comes in very handy in [Hybrid Search](https://www.kirupa.com/ai_fundamentals/hybrid_search.htm). If we're just trying out an idea, an embedded option like Chroma or LanceDB runs inside our app with no server at all. When the vectors become the main event, with a huge collection, lots of traffic, or a need for specialized features, a dedicated vector database like Qdrant, Weaviate, Milvus, or Pinecone starts to earn its keep. The last question is who runs it. A managed service handles the servers, upgrades, and backups for us, in exchange for money and for keeping our data on someone else's service. Running it ourselves gives us more control, but then we're the ones who get paged when it falls over. Services change, too, as Chroma's hosted service winding down shows, so it's worth keeping our original documents and chunking code around. Rebuilding an index somewhere else is then a normal job, the same kind we'll see in [Keeping the Index Fresh](https://www.kirupa.com/ai_fundamentals/keeping_the_index_fresh.htm). Check the limits before committing, too. At the time of writing, pgvector can index vectors of up to 2,000 numbers. That covers the 1,536 from OpenAI's `text-embedding-3-small`, but the 3,072 from `text-embedding-3-large` only fit if we store them at half precision, which pgvector can index at up to 4,000. ## Conclusion A vector database stores each chunk's vector next to its text and its labels, and its main job is answering one question very quickly: which stored vectors are closest to this one? Checking every vector always gets the right answer, but the work grows with every chunk we add. Approximate indexes check only a small slice instead. HNSW hops across a graph with express layers on top, and IVF searches only the closest groups. Both give up a little recall for their speed, and both have a dial for how much. Around that core, the database needs to measure closeness the same way our embedding model expects, and it needs to filter on labels during the search, so the filter doesn't throw away most of the results. When it comes to picking one, the best first move is usually the boring one: use the database you already run, and reach for a dedicated vector database once the vectors become the main event. So far, we've been storing chunks without ever asking how big each chunk should be. That turns out to matter a lot, and it's what we'll look at next in [Chunk Size and Chunk Overlap](https://www.kirupa.com/ai_fundamentals/chunk_size_and_overlap.htm). And Marcus's zipper question? It still finds Fixing a jammed slider, only now it gets there in a handful of hops instead of 40,000 comparisons.