Enterprise data integration pipelines face a critical challenge: scaling throughput without compromising data correctness. This account details scaling from 500 to 8,000 events per second while maintaining two non-negotiable guarantees.

The first guarantee prevents later entity states from being overwritten by earlier versions. Each entity carries a source-owned version number, and writes reject stale data through a last-write-wins mechanism where 'last' means highest version number, not arrival time. This allows aggressive parallelism without ordering concerns.

The second guarantee ensures accurate duplicate detection. Every accepted record writes its dedup-log entry and business data in the same database transaction, making them commit together or not at all. Moving from application-level checks to database primary-key constraints eliminated duplicate slips during high concurrency.

Throughput scaling employs strategic partitioning. Events for the same entity route to the same partition via entity ID hashing, maintaining natural order. However, when one large account generated 100 times normal traffic, sub-partitioning spread hot entities across partitions using entity ID plus event type as the key. A background job continuously monitors entity rates, promoting and demoting entities between regular and fine-grained partitioning. The version check system absorbs any out-of-order processing that sub-partitioning introduces.

Micro-batching delivers actual speed improvements by eliminating per-event network round-trips and reducing separate database transaction overhead.