Databricks is open sourcing Dicer, its internal auto-sharding system for building low-latency, scalable services. The system powers critical components like Unity Catalog and SQL orchestration, managing dynamic shard assignments to maintain responsiveness during restarts, failures, and shifting workloads. By colocating state with application logic, Dicer avoids the high costs and latency of stateless architectures that repeatedly fetch data from databases or remote caches.

Traditional static sharding creates major headaches: downtime during scaling, split-brain scenarios after crashes, and performance bottlenecks from hot keys. Dicer introduces an intelligent control plane that continuously updates assignments based on application health, load, and termination signals. This approach keeps cache hit rates high and eliminates availability dips, addressing the core fragility of consistent hashing schemes in production environments.

Dicer operates on key ranges called Slices, assigning them to Resources (pods) across the application keyspace. This model scales to billions of keys while enabling soft leader election and efficient GPU utilization for AI workloads. Databricks invites collaboration to advance distributed systems design, offering Dicer as a solution for teams balancing performance demands with operational reliability and cloud efficiency.