HeadlinesBriefing favicon HeadlinesBriefing.com

Browser-Native Semantic Search With 4MB Model

Hacker News •
×

Static sites have long relied on keyword search like Lunr.js, which builds an inverted index at compile time but understands nothing about meaning. The author previously built a Python semantic engine using sentence-transformers, yet serving it required heavy server infrastructure — Transformers.js brings that to the browser but weighs 23.45 MB and takes two seconds to initialize, excessive for a 14-post blog.

The breakthrough is model2vec's potion-base-8M, a distilled embedding table rather than a neural network. Its forward pass is three lines: tokenize, look up one 256-dimensional vector per token, average them, normalize. The 29,528-token float32 table reaches 81% of MiniLM's retrieval quality at ~30 MB. Chunking posts into overlapping 600-character segments preserves rare terms like "pydub" that would otherwise drown in whole-post averaging.

Quantizing to int8 with per-row scales shrinks the model to 4 MB while maintaining 0.999958 cosine fidelity. The model's stopword behavior emerges naturally: common tokens like "the" have near-zero vector magnitude, while distinctive terms like "guantanamo" dominate. A custom WordPiece tokenizer in ~80 lines of JavaScript handles three gotchas — no [CLS]/[SEP] markers, unknown tokens deleted not embedded, and special-token config ignored — enabling fully client-side semantic search with no server, no API, and a payload smaller than a hero image.