HeadlinesBriefing favicon HeadlinesBriefing.com

Usenet Archive Toolkit: Create Searchable Archives

Hacker News •
×

The Usenet Archive Toolkit (UAT) project offers a suite of tools to process Usenet messages into a coherent, searchable archive, addressing the decay of online discussions.

UAT is designed for offline access, allowing users to browse and search archives without an internet connection, unlike cloud-based solutions like Google Groups. Archives are stored locally, ensuring data persistence against service shutdowns. The UAT archive format is optimized for fast access and efficient search, compressing individual messages while using a whole-archive dictionary for better compression. It can merge multiple message sources, fill gaps, and eliminate duplicate or extraneous messages often found in other archives. UAT also includes spam filtering and robust decoding of malformed message headers and encodings.

The toolkit supports various import formats including maildir, mbox, and Google Groups crawls, and can fetch messages from NNTP servers. Processing utilities handle message identification, metadata extraction, merging, UTF-8 conversion, and recalculating message connectivity. Data filtering tools remove duplicates and spam, while search utilities build word lexicons for efficient querying. Access to archives is provided through a simple libuat interface, with memory mapping for efficient data retrieval and low memory overhead, demonstrated by the tbrowser utility requiring only 25 MB for 2.5 million messages.