X's engineering team has open-sourced the recommendation algorithm powering the platform's 'For You' feed, revealing a system built on Grok-based transformer models that replaced nearly all hand-crafted rules with machine learning. The codebase, published on GitHub under Apache-2.0 license, shows how X delivers personalized content in real-time without latency issues that could frustrate users.

When you open the 'For You' tab, the algorithm pulls from two sources: in-network content from accounts you follow, and out-of-network content discovered through similarity search. The system uses a two-tower model where your engagement history and metadata are encoded into embeddings, then matched against post embeddings to find relevant content. This approach means you see posts from unknown authors if the ML predicts you'll find them interesting.

The architecture consists of four core components: Home Mixer orchestrates the pipeline, Thunder provides real-time in-memory post storage, Phoenix handles both retrieval and ranking using the Grok transformer, and the Candidate Pipeline provides a modular framework for processing. The entire system is written in Rust (62.9%) and Python (37.1%), demonstrating how modern social media platforms balance performance with ML sophistication.