Tiger Data released pg_vectorsearch, a new vector index for Postgres

Teams running vector search in Postgres have had one good index to pick. It is called HNSW, and it needs a lot of memory. Tiger Data, the company behind TimescaleDB, has released a new open source Postgres extension called pg_vectorsearch. Its index is called PRISM. PRISM runs 100 million vectors on a quarter of the memory HNSW needs.

The extension is in alpha. Tiger Data says it is for testing, not production. It needs PostgreSQL 18 or later. Each vector can have up to 1968 dimensions. Filtered search that mixes WHERE and ORDER BY in one query is not supported yet. Tiger Data plans to add it at the first stable release.

What PRISM does

PRISM stands for Partitioned Routing Index for Similarity Matching. Most Postgres teams today use HNSW through the pgvector extension. HNSW builds a graph that connects every vector to its near neighbours. A search walks the graph. That means lots of random reads, which only stay fast if most of the graph sits in memory.

PRISM uses a different method. It groups similar vectors into partitions and stores them as a list in a compressed format, packed into database pages. A search finds the right partitions first, then scans them in order. It also uses a method called RaBitQ quantization. RaBitQ gives the search error bounds for each partition. The search can then rule out vectors that cannot be the answer, without checking them in full precision.

The numbers Tiger Data reports

Every figure here comes from Tiger Data's own public benchmark tool, measured on the same PostgreSQL configuration and instance types. Recall is recall@10 against an exact brute-force search. HNSW ran with pgvector's 16-bit halfvec type.

At 100 million vectors, PRISM built the index in 1 hour and 40 minutes. It did so on a machine with 128 GB of memory. HNSW took 42 hours on a machine with 512 GB. PRISM then served queries at a p50 latency of 6 ms at 95 percent recall. HNSW served them at 8 ms on four times the memory.

At 1 million vectors and 99 percent recall, Tiger Data reports PRISM at 15 ms on a 2 GB machine. It reports HNSW at 392 ms on a 4 GB machine. That is about 23 times slower on twice the memory, by Tiger Data's figures. Tiger Data runs the benchmark on its own tool. An independent check would make the numbers firmer.

For a team already using pgvector, PRISM reads vector and halfvec columns directly. The two extensions can be installed next to each other. Tiger Data plans to move its older pgvectorscale extension into maintenance mode.

On a Hacker News thread about the release, practitioners said they were interested in a cheaper vector search inside Postgres. They also asked about independent benchmarks. They asked whether filtered search would work. They asked whether PRISM would replace pgvectorscale. One commenter, Hermon Hagos, wrote that fast semantic search without a separate vector store "is real money saved". Hagos also wrote that the extension is still alpha and should not go to clients yet.

A team building a retrieval feature on Postgres now has a reason to test PRISM against its current HNSW setup. Run the test on the team's own data. Do it before paying for more memory or adding a separate vector database. The memory saving is the figure that changes what the stack costs to run every month.

Source

• Announcing pg_vectorsearch: Scalable Open-Source Vector Search in PostgreSQL, Tiger Data
• Open Source pg_vectorsearch Brings More Scalable Vector Search to PostgreSQL, InfoQ
• pg_vectorsearch on GitHub, Tiger Data