Hey everyone. I’m Dan. I’m an engineer living in Los Angeles.

I’ve been in engineering for like 25 years and I’ve been lucky. One way I’ve been lucky lately is that I’ve gotten the chance to flail at making agents run reliably in production. I mean specifically “agents” that consumers are meant to use, to perform tasks on their behalf.

I’d consider those distinct from chatbots that users talk to with largely subjective outputs and outcomes. The goal of making a minimally-embarrassing agentic experience that I’m actually proud of imposes some serious challenges. Clearly not everyone is motivated by their inner sense of shame, as I am.

Some people are more than satisfied to give you a subjective advice machine, and let you wander into the wilderness to be eaten by bears. But not me. I’m here for you.

I want to say at the outset here that this is the most fun I’ve had building stuff in my whole career! It’s magical and addictive. I’m a dog in a ballpit. When I was 22 getting paid to write visual basic felt thrilling.

Working at a cool startup in Brooklyn 2007 made me feel like a golden god. The last decade+ has been a slog. I didn’t think I had it in me anymore.

But i’m feeling joy in programming again! I mean this sincerely, despite how deeply weird this talk is going to get. It’s going to get weird because I feel like everyone engaged in this line of work is potentially an at-risk person in some dimension or another. I’m breaking my brain using agents to run agents to build evaluation for other agents every day, and it’s so fun.

But I would say that, wouldn’t I. The veil between awesome engineering and complete psychological collapse has never been thinner. And in our field, that is really saying something.

I don’t feel like I definitely know what I’m doing. But I also don’t feel like I’ve read much by people that obviously know what they’re doing. And I’ve certainly read things from people who obviously don’t know what they’re doing.

It seemed like a reasonable time to compare notes. One thing I have noticed is that although LLM’s are generally speaking impressive, their demons still escape containment if you are monitoring what they’re up to with any amount of scale. We all academically understand that LLM’s cannot reliably follow instructions, tell the truth, or perform tasks.

But day-to-day they can trick us into thinking they’re pretty reliable. This perception falls apart immediately if you are trying to operate an agent that real people are using. They fail in subtle ways for sure, but they also fail in simple ways.

Like any good programmer I attempt to interact with my LLM with structured output. It’s nice, you can map Python code to a prompt automatically, and most of the time your schema is respected. Most of the time.

You can try to instruct the model to return a title that’s 80 characters or less. And it’ll work most of the time. But then sometimes it’ll completely botch it and flood your field with nonsense until it explodes.

It’s usually a tiny fraction of requests, but the smartest models still fail at this. And the fraction can be smaller or bigger depending on the exact nature of what you give the model, so you have to watch it like a hawk. What’s going on in there? Usually it’s a novel-length series of repeating notes to self about JSON, mostly.

When this happened to me most recently, it turned out that a fix was to rename the field from “title” to “heading.” That is currently working, but since the fix is fully deranged I expect it’ll be disturbed again at some point. The same sorts of issues exist with calling tools, or most other behaviors. A fraction of requests will be haunted, and spin out uncontrollably.

But despite this, the tech is tantalizing and magical. The problem shifts to one of constraining the behavior, but never fully taming the beast. To constrain the behavior you have to measure it—one way is to just run tests a ton of times.

The industry term of art for this is pass^k (“pass power...