One Capital Letter Was Silently Breaking My AI Support Bot, and It Wasn't in the New Model

Picture a support inbox for a bank where every message must be sorted into categories like lost card, refund request, or wrong charge. When this routing job is handed to an AI model instead of a person, the model returns a short note in a fixed JSON format. The program reading the answer does not understand English; it looks for exact labels like intent and priority, spelled precisely every single time. If a label is spelled slightly differently than expected, the program silently stops working for that message while everything still looks fine on the surface.

AI companies release new model versions constantly. Deciding which Large Language Model to use usually comes down to one number: how often it picks the right category. That single score can rise while something else, the exact shape of the reply, quietly gets worse. With plain prompting, writing "Return your answer as JSON" may still result in the model adding extra sentences or leaving out required fields. Structured Outputs is a stricter feature that makes the model follow a predefined structure, but the application still needs to verify values are correct.

I ran a real LLM regression test on a support triage assistant using 47 real customer messages from a public banking dataset. I tested three versions of an OpenAI model: an older version, the current production model, and a newer candidate. The newer model followed the format every time. The production model made a formatting mistake quietly, on every refund question in the sample. It spelled one label Request_refund with a capital R instead of the lowercase request_refund the system expects. A human reading the reply would call it correct, but a program matching labels exactly would silently drop every one of those tickets. This highlights the danger: a model can sound correct to a person and still be wrong for the software that uses its answer.

Picture a support inbox for a bank where every message must be sorted into categories like lost card, refund request, or wrong charge. When this routing job is handed to an AI model instead of a person, the model returns a short note in a fixed JSON format. The program reading the answer does not understand English; it looks for exact labels like intent and priority, spelled precisely every single time. If a label is spelled slightly differently than expected, the program silently stops working for that message while everything still looks fine on the surface.

AI companies release new model versions constantly. Deciding which Large Language Model to use usually comes down to one number: how often it picks the right category. That single score can rise while something else, the exact shape of the reply, quietly gets worse. With plain prompting, writing "Return your answer as JSON" may still result in the model adding extra sentences or leaving out required fields. Structured Outputs is a stricter feature that makes the model follow a predefined structure, but the application still needs to verify values are correct.

I ran a real LLM regression test on a support triage assistant using 47 real customer messages from a public banking dataset. I tested three versions of an OpenAI model: an older version, the current production model, and a newer candidate. The newer model followed the format every time. The production model made a formatting mistake quietly, on every refund question in the sample. It spelled one label Request_refund with a capital R instead of the lowercase request_refund the system expects. A human reading the reply would call it correct, but a program matching labels exactly would silently drop every one of those tickets. This highlights the danger: a model can sound correct to a person and still be wrong for the software that uses its answer.

Picture a support inbox for a bank where every message must be sorted into categories like lost card, refund request, or wrong charge. When this routing job is handed to an AI model instead of a person, the model returns a short note in a fixed JSON format. The program reading the answer does not understand English; it looks for exact labels like intent and priority, spelled precisely every single time. If a label is spelled slightly differently than expected, the program silently stops working for that message while everything still looks fine on the surface.

AI companies release new model versions constantly. Deciding which Large Language Model to use usually comes down to one number: how often it picks the right category. That single score can rise while something else, the exact shape of the reply, quietly gets worse. With plain prompting, writing "Return your answer as JSON" may still result in the model adding extra sentences or leaving out required fields. Structured Outputs is a stricter feature that makes the model follow a predefined structure, but the application still needs to verify values are correct.

I ran a real LLM regression test on a support triage assistant using 47 real customer messages from a public banking dataset. I tested three versions of an OpenAI model: an older version, the current production model, and a newer candidate. The newer model followed the format every time. The production model made a formatting mistake quietly, on every refund question in the sample. It spelled one label Request_refund with a capital R instead of the lowercase request_refund the system expects. A human reading the reply would call it correct, but a program matching labels exactly would silently drop every one of those tickets. This highlights the danger: a model can sound correct to a person and still be wrong for the software that uses its answer.

Picture a support inbox for a bank where every message must be sorted into categories like lost card, refund request, or wrong charge. When this routing job is handed to an AI model instead of a person, the model returns a short note in a fixed JSON format. The program reading the answer does not understand English; it looks for exact labels like intent and priority, spelled precisely every single time. If a label is spelled slightly differently than expected, the program silently stops working for that message while everything still looks fine on the surface.

AI companies release new model versions constantly. Deciding which Large Language Model to use usually comes down to one number: how often it picks the right category. That single score can rise while something else, the exact shape of the reply, quietly gets worse. With plain prompting, writing "Return your answer as JSON" may still result in the model adding extra sentences or leaving out required fields. Structured Outputs is a stricter feature that makes the model follow a predefined structure, but the application still needs to verify values are correct.

I ran a real LLM regression test on a support triage assistant using 47 real customer messages from a public banking dataset. I tested three versions of an OpenAI model: an older version, the current production model, and a newer candidate. The newer model followed the format every time. The production model made a formatting mistake quietly, on every refund question in the sample. It spelled one label Request_refund with a capital R instead of the lowercase request_refund the system expects. A human reading the reply would call it correct, but a program matching labels exactly would silently drop every one of those tickets. This highlights the danger: a model can sound correct to a person and still be wrong for the software that uses its answer.

Picture a support inbox for a bank where every message must be sorted into categories like lost card, refund request, or wrong charge. When this routing job is handed to an AI model instead of a person, the model returns a short note in a fixed JSON format. The program reading the answer does not understand English; it looks for exact labels like intent and priority, spelled precisely every single time. If a label is spelled slightly differently than expected, the program silently stops working for that message while everything still looks fine on the surface.

AI companies release new model versions constantly. Deciding which Large Language Model to use usually comes down to one number: how often it picks the right category. That single score can rise while something else, the exact shape of the reply, quietly gets worse. With plain prompting, writing "Return your answer as JSON" may still result in the model adding extra sentences or leaving out required fields. Structured Outputs is a stricter feature that makes the model follow a predefined structure, but the application still needs to verify values are correct.

I ran a real LLM regression test on a support triage assistant using 47 real customer messages from a public banking dataset. I tested three versions of an OpenAI model: an older version, the current production model, and a newer candidate. The newer model followed the format every time. The production model made a formatting mistake quietly, on every refund question in the sample. It spelled one label Request_refund with a capital R instead of the lowercase request_refund the system expects. A human reading the reply would call it correct, but a program matching labels exactly would silently drop every one of those tickets. This highlights the danger: a model can sound correct to a person and still be wrong for the software that uses its answer.

Picture a support inbox for a bank where every message must be sorted into categories like lost card, refund request, or wrong charge. When this routing job is handed to an AI model instead of a person, the model returns a short note in a fixed JSON format. The program reading the answer does not understand English; it looks for exact labels like intent and priority, spelled precisely every single time. If a label is spelled slightly differently than expected, the program silently stops working for that message while everything still looks fine on the surface.

AI companies release new model versions constantly. Deciding which Large Language Model to use usually comes down to one number: how often it picks the right category. That single score can rise while something else, the exact shape of the reply, quietly gets worse. With plain prompting, writing "Return your answer as JSON" may still result in the model adding extra sentences or leaving out required fields. Structured Outputs is a stricter feature that makes the model follow a predefined structure, but the application still needs to verify values are correct.

I ran a real LLM regression test on a support triage assistant using 47 real customer messages from a public banking dataset. I tested three versions of an OpenAI model: an older version, the current production model, and a newer candidate. The newer model followed the format every time. The production model made a formatting mistake quietly, on every refund question in the sample. It spelled one label Request_refund with a capital R instead of the lowercase request_refund the system expects. A human reading the reply would call it correct, but a program matching labels exactly would silently drop every one of those tickets. This highlights the danger: a model can sound correct to a person and still be wrong for the software that uses its answer.

Picture a support inbox for a bank where every message must be sorted into categories like lost card, refund request, or wrong charge. When this routing job is handed to an AI model instead of a person, the model returns a short note in a fixed JSON format. The program reading the answer does not understand English; it looks for exact labels like intent and priority, spelled precisely every single time. If a label is spelled slightly differently than expected, the program silently stops working for that message while everything still looks fine on the surface.

AI companies release new model versions constantly. Deciding which Large Language Model to use usually comes down to one number: how often it picks the right category. That single score can rise while something else, the exact shape of the reply, quietly gets worse. With plain prompting, writing "Return your answer as JSON" may still result in the model adding extra sentences or leaving out required fields. Structured Outputs is a stricter feature that makes the model follow a predefined structure, but the application still needs to verify values are correct.

I ran a real LLM regression test on a support triage assistant using 47 real customer messages from a public banking dataset. I tested three versions of an OpenAI model: an older version, the current production model, and a newer candidate. The newer model followed the format every time. The production model made a formatting mistake quietly, on every refund question in the sample. It spelled one label Request_refund with a capital R instead of the lowercase request_refund the system expects. A human reading the reply would call it correct, but a program matching labels exactly would silently drop every one of those tickets. This highlights the danger: a model can sound correct to a person and still be wrong for the software that uses its answer.

Picture a support inbox for a bank where every message must be sorted into categories like lost card, refund request, or wrong charge. When this routing job is handed to an AI model instead of a person, the model returns a short note in a fixed JSON format. The program reading the answer does not understand English; it looks for exact labels like intent and priority, spelled precisely every single time. If a label is spelled slightly differently than expected, the program silently stops working for that message while everything still looks fine on the surface.

AI companies release new model versions constantly. Deciding which Large Language Model to use usually comes down to one number: how often it picks the right category. That single score can rise while something else, the exact shape of the reply, quietly gets worse. With plain prompting, writing "Return your answer as JSON" may still result in the model adding extra sentences or leaving out required fields. Structured Outputs is a stricter feature that makes the model follow a predefined structure, but the application still needs to verify values are correct.

I ran a real LLM regression test on a support triage assistant using 47 real customer messages from a public banking dataset. I tested three versions of an OpenAI model: an older version, the current production model, and a newer candidate. The newer model followed the format every time. The production model made a formatting mistake quietly, on every refund question in the sample. It spelled one label Request_refund with a capital R instead of the lowercase request_refund the system expects. A human reading the reply would call it correct, but a program matching labels exactly would silently drop every one of those tickets. This highlights the danger: a model can sound correct to a person and still be wrong for the software that uses its answer.

Picture a support inbox for a bank where every message must be sorted into categories like lost card, refund request, or wrong charge. When this routing job is handed to an AI model instead of a person, the model returns a short note in a fixed JSON format. The program reading the answer does not understand English; it looks for exact labels like intent and priority, spelled precisely every single time. If a label is spelled slightly differently than expected, the program silently stops working for that message while everything still looks fine on the surface.

AI companies release new model versions constantly. Deciding which Large Language Model to use usually comes down to one number: how often it picks the right category. That single score can rise while something else, the exact shape of the reply, quietly gets worse. With plain prompting, writing "Return your answer as JSON" may still result in the model adding extra sentences or leaving out required fields. Structured Outputs is a stricter feature that makes the model follow a predefined structure, but the application still needs to verify values are correct.

I ran a real LLM regression test on a support triage assistant using 47 real customer messages from a public banking dataset. I tested three versions of an OpenAI model: an older version, the current production model, and a newer candidate. The newer model followed the format every time. The production model made a formatting mistake quietly, on every refund question in the sample. It spelled one label Request_refund with a capital R instead of the lowercase request_refund the system expects. A human reading the reply would call it correct, but a program matching labels exactly would silently drop every one of those tickets. This highlights the danger: a model can sound correct to a person and still be wrong for the software that uses its answer.

Picture a support inbox for a bank where every message must be sorted into categories like lost card, refund request, or wrong charge. When this routing job is handed to an AI model instead of a person, the model returns a short note in a fixed JSON format. The program reading the answer does not understand English; it looks for exact labels like intent and priority, spelled precisely every single time. If a label is spelled slightly differently than expected, the program silently stops working for that message while everything still looks fine on the surface.

AI companies release new model versions constantly. Deciding which Large Language Model to use usually comes down to one number: how often it picks the right category. That single score can rise while something else, the exact shape of the reply, quietly gets worse. With plain prompting, writing "Return your answer as JSON" may still result in the model adding extra sentences or leaving out required fields. Structured Outputs is a stricter feature that makes the model follow a predefined structure, but the application still needs to verify values are correct.

I ran a real LLM regression test on a support triage assistant using 47 real customer messages from a public banking dataset. I tested three versions of an OpenAI model: an older version, the current production model, and a newer candidate. The newer model followed the format every time. The production model made a formatting mistake quietly, on every refund question in the sample. It spelled one label Request_refund with a capital R instead of the lowercase request_refund the system expects. A human reading the reply would call it correct, but a program matching labels exactly would silently drop every one of those tickets. This highlights the danger: a model can sound correct to a person and still be wrong for the software that uses its answer.

Picture a support inbox for a bank where every message must be sorted into categories like lost card, refund request, or wrong charge. When this routing job is handed to an AI model instead of a person, the model returns a short note in a fixed JSON format. The program reading the answer does not understand English; it looks for exact labels like intent and priority, spelled precisely every single time. If a label is spelled slightly differently than expected, the program silently stops working for that message while everything still looks fine on the surface.

AI companies release new model versions constantly. Deciding which Large Language Model to use usually comes down to one number: how often it picks the right category. That single score can rise while something else, the exact shape of the reply, quietly gets worse. With plain prompting, writing "Return your answer as JSON" may still result in the model adding extra sentences or leaving out required fields. Structured Outputs is a stricter feature that makes the model follow a predefined structure, but the application still needs to verify values are correct.

I ran a real LLM regression test on a support triage assistant using 47 real customer messages from a public banking dataset. I tested three versions of an OpenAI model: an older version, the current production model, and a newer candidate. The newer model followed the format every time. The production model made a formatting mistake quietly, on every refund question in the sample. It spelled one label Request_refund with a capital R instead of the lowercase request_refund the system expects. A human reading the reply would call it correct, but a program matching labels exactly would silently drop every one of those tickets. This highlights the danger: a model can sound correct to a person and still be wrong for the software that uses its answer.

Picture a support inbox for a bank where every message must be sorted into categories like lost card, refund request, or wrong charge. When this routing job is handed to an AI model instead of a person, the model returns a short note in a fixed JSON format. The program reading the answer does not understand English; it looks for exact labels like intent and priority, spelled precisely every single time. If a label is spelled slightly differently than expected, the program silently stops working for that message while everything still looks fine on the surface.

AI companies release new model versions constantly. Deciding which Large Language Model to use usually comes down to one number: how often it picks the right category. That single score can rise while something else, the exact shape of the reply, quietly gets worse. With plain prompting, writing "Return your answer as JSON" may still result in the model adding extra sentences or leaving out required fields. Structured Outputs is a stricter feature that makes the model follow a predefined structure, but the application still needs to verify values are correct.

I ran a real LLM regression test on a support triage assistant using 47 real customer messages from a public banking dataset. I tested three versions of an OpenAI model: an older version, the current production model, and a newer candidate. The newer model followed the format every time. The production model made a formatting mistake quietly, on every refund question in the sample. It spelled one label Request_refund with a capital R instead of the lowercase request_refund the system expects. A human reading the reply would call it correct, but a program matching labels exactly would silently drop every one of those tickets. This highlights the danger: a model can sound correct to a person and still be wrong for the software that uses its answer.