HeadlinesBriefing favicon HeadlinesBriefing.com

How OpenAI Built GPT-Live

ByteByteGo •
×

How do you optimize AI when performance matters? And how can you apply AI to make performance better (and easier) than ever before? That’s what 30K engineers will explore at P99 CONF. Here’s a taste of the talks you can expect: Sandboxmaxxing at Lovable: Every Prompt Gets a Sandbox in < 1s; The Autonomous Performance Agent: A Netflix Production Story; Lessons Learned from Building Crazy Fast, Open Source Infrastructure for AI Agents; Give the Agent a Cluster: Effective AI for Performance Engineering at Scale; Managing 500 Billion+ Files for AI Workloads; How to Improve Your Cache Algorithm Using AI. GET YOUR FREE TICKET. Bonus: Registrants get immediate 30-day access to the complete O’Reilly library, and attendees can enter to win 1 of 500 free swag packs.

If you have used voice assistants before, you have probably experienced unexpected interruptions. You talk with the system, and the moment you pause briefly to think of the right term, it starts talking. Then you interrupt it so you can continue. This is a common frustration, because most voice models can either listen or speak, but not both at the same time.

Newer voice models, like OpenAI’s GPT-Live-1, change that by listening and speaking at the same time. The model constantly decides whether it should stay quiet, interrupt, or start talking. This makes the conversation feel more natural with fewer unintentional interruptions. Under the hood, these systems combine a new generation of voice model architecture with a serving system optimized for low latency. To understand how it all works end to end, we met with engineers on the GPT Voice team, Zahan Malkani and Justin Uberti (who created WebRTC). We thank both of them for sharing the details with us.

In this article, you’ll learn: the three generations of voice systems, including cascaded pipelines, turn-based end-to-end models, and full-duplex models; delegating thinking from talking, the core idea behind GPT-Live; the engineering behind the serving system, including the live and async paths; how evaluation is different in full-duplex voice systems; and engineering lessons for building realtime systems and what is next for voice.