Moondream Engineering unveiled Photon, its inference engine that achieves near-realtime vision-language model inference at approximately 33ms on NVIDIA B200 hardware. The system delivers up to 35% higher decode throughput by addressing a fundamental bottleneck in how GPUs process AI workloads.

The issue stems from GPU bubbles—idle periods where the graphics processor waits for CPU housekeeping between token generations. In autoregressive models, each token depends on the previous one, requiring CPU-GPU synchronization for request scheduling, metadata setup, and token selection. This creates a sequential bottleneck where the GPU sits unused while the CPU completes fixed-cost operations.

Photon solves this through pipelined decoding, overlapping CPU and GPU work streams. The engine launches the next forward pass while the CPU finishes committing previous results. Two decode slots operate in a ping-pong fashion, preventing buffer collisions while allowing simultaneous processing. Each slot contains input staging, output buffers, and KV cache bookkeeping.

The second mechanism, forward-now-sample-later, decouples the next forward pass from sampling constraints. Photon runs the t+1 forward immediately, then builds the token mask after committing step t's results. This allows structured output generation for spatial tasks like point coordinates and object detection. The approach eliminates idle GPU time by treating token copies as background transfers rather than blocking operations.