ByteDance introduces USDB, a unified sequence storage system designed to handle the growing demands of next-generation recommender models. These models consume increasingly long, attribute-rich user histories, creating a storage asymmetry between small continuous updates and repeated large reads. USDB addresses this by using cascade LSM time-based compaction to materialize application-specific histories, while differentiated replicas share persistent files.
Partial dumps reconstruct inference inputs from recent state and versioned history, and a hybrid columnar format accelerates projected reads. These designs reuse maintenance results across ingestion, inference, and training. Production migration records show 85.8% lower mean write latency, while normalized deployment accounting shows 42.1% lower infrastructure cost.
Controlled comparisons with an Optimized RocksDB sequence baseline demonstrate faster long-history reads across various projections and client loads. Additionally, aggregation pushdown sustains 8.5x the throughput of ordinary reads under a constrained receive budget. Production reconstruction and direct-file training results highlight how shared sequence maintenance supports fresh serving and repeated historical access.
Source: Hacker News · Summarized by HeadlinesBriefing