HeadlinesBriefing favicon HeadlinesBriefing.com

LLM-Powered Web Extraction Library Released

Hacker News •
×

Lightfeed Extractor, a new TypeScript library, offers a robust solution for web data extraction using LLMs. Traditional scrapers constantly break when sites change layouts, while raw HTML fed to LLMs creates problems with noise, malformed JSON, and URL handling. The library handles the entire pipeline from raw HTML to validated structured data, solving persistent pain points in web scraping.

Lightfeed Extractor converts HTML to LLM-ready markdown by extracting main content while stripping navigation and footer elements. It works with any LangChain-compatible LLM and uses Zod schemas for type-safe extraction. The library includes built-in JSON recovery to handle malformed outputs, ensuring partial data extraction even when some elements fail.

The tool integrates browser automation via Playwright with anti-bot patches and can pair with the company's browser agent for AI-driven page navigation. Developers can install it via npm with Apache 2.0 licensing. The library represents a standardized approach to the repetitive boilerplate code teams typically write for web extraction with AI enrichment.