As more Americans turn to Large Language Models for medical questions—a KFF poll found one in three adults have used AI for health info—health systems are rushing to deploy their own branded chatbots. Executives frame these tools, like PatientGPT from K Health and Hartford HealthCare, as a convenience and a safer alternative to unmonitored commercial versions patients already use to navigate access issues.

This push enters a U.S. healthcare system already struggling with poor outcomes and high costs, where over 100 million people lack a primary care provider. While the intent is to meet user demand, experts caution that the evidence base for improved patient outcomes from integrated chatbots remains nonexistent. Concerns over liability and monitoring also cloud these debuts.

Early studies show LLMs struggle with real-world user prompts, correctly identifying conditions only about a third of the time when users don't know the right input. Despite this, Hartford HealthCare is expanding its beta test to tens of thousands, reducing human review of interactions to just 20 conversations daily amid the wider rollout.

Meanwhile, Epic, the EHR giant, is also seeing adoption of its assistant, Emmie, at systems like Sutter Health. While testing suggests failure rates in high-risk scenarios can drop via red teaming, the real-world impact of the remaining failure rate is still an open question for providers.