A developer built a symptom-checking chatbot for a telehealth startup, aiming for a cautious, thorough system. Instead, it terrified users. The AI listed rare conditions like brain tumors and leukemia for common symptoms, sending people to the ER for stress or allergies. The app generated panic, costly emergency visits, and legal threats, proving technical correctness can be practically harmful in healthcare.

The root cause was a prompt demanding comprehensive information that included serious, rare conditions for every symptom. This created a 'medical student syndrome' effect at scale, where users fixated on worst-case scenarios. The AI treated all possibilities with equal weight, alarming people unnecessarily and undermining its purpose as a calming guide.

The fix involved rebuilding the system around likelihood and context. Common conditions were prioritized, and rare ones were only mentioned when symptom patterns justified them. Responses shifted from alarming lists to clear guidance, explaining when a doctor visit was truly needed. After the change, user anxiety dropped, ER visits became appropriate, and reviews turned positive.