Logfire is delivering up to 1,000x faster queries for semi-structured data through a technique called dynamic shredding. Queries that previously timed out after 30 seconds now complete in under a second, with customers reporting "basically instant" response times. The company is rolling out this feature to all customers in February 2026.
The optimization addresses how Logfire stores OpenTelemetry data, which contains fixed-schema fields (timestamps, span names) and semi-structured attributes (HTTP status codes, user IDs, regions). Initially, all attributes were stored as JSON blobs in a single column, parsed efficiently using Jiter. This approach made fundamental query limitations when searching across semi-structured data.
Dynamic shredding transforms the JSON attributes into a columnar format optimized for querying, enabling better zone map pruning and Parquet file scanning. The technique automatically identifies and extracts frequently queried attributes into separate columns while keeping less common ones in structured JSON, dramatically improving analytical query performance.
This breakthrough allows Logfire to handle complex telemetry workloads at scale, making observability data more actionable and responsive for development teams monitoring distributed systems.
Source: Hacker News · Summarized by HeadlinesBriefing