HeadlinesBriefing favicon HeadlinesBriefing.com

DuckDB ACORN-1 Extension Fixes Filtered Vector Search

Hacker News •
×

A developer has created a DuckDB community extension that solves a critical limitation in filtered vector search using the ACORN-1 algorithm. The fork of duckdb-vss adds true prefiltered approximate nearest neighbors, addressing the problem where WHERE clauses were previously applied after HNSW index results returned, often yielding fewer results than requested.

Traditional vector similarity search with filtering suffered from a fundamental flaw: queries like SELECT ... WHERE category = 'X' ORDER BY distance LIMIT 10 would return incomplete results because filtering happened post-search. This extension pushes filter predicates into the HNSW graph traversal using ACORN-1's two-hop expansion through failed neighbors, ensuring filtered queries return the correct number of results with high recall. The implementation includes selectivity-based strategy switching - using post-filter for high selectivity (>60%), ACORN-1 for medium selectivity (1-60%), and brute-force exact scan for very low selectivity (<1%).

Benchmarks with 228k movies and 768-dim embeddings show dramatic improvements: Korean-only filtering jumped from 0/10 to 10/10 results, while English-only filtering maintained perfect recall. The extension requires no special syntax - the optimizer automatically detects WHERE + ORDER BY distance + LIMIT patterns. Configuration options include SET hnsw_acorn_threshold = 0.6 and SET hnsw_bruteforce_threshold = 0.01 for tuning performance.