AI Symptom Checkers’ Actual Triage Accuracy: What a Decade of Research Actually Shows

📖 The Foundational Study: Semigran et al., 2015

The modern research into this category effectively starts with a 2015 audit study published in The BMJ (Semigran, Linder, Gidengil, and Mehrotra), which tested 23 publicly available symptom checkers using 45 standardized clinical vignettes covering a range of common and serious conditions. The results, which still get cited in nearly every study since: the checkers listed the correct diagnosis first in about one-third of cases, and included the correct diagnosis somewhere in the top 20 possible results in just over half of cases. Triage advice — telling a user whether to seek emergency care, see a doctor soon, or handle things with self-care — was, in the study’s own words, highly variable, with appropriate triage advice ranging dramatically depending on the specific tool and condition.

This single study established both the standard test format still used in research today, and the uncomfortable baseline finding this category has been working to improve ever since: symptom checkers, as a category, have never been reliably accurate.

🏥 The Most Rigorous Recent Study: Real Emergency Department Patients

Most symptom-checker research uses hypothetical clinical vignettes rather than real patients, which is a reasonable research method but has real limitations. A 2023 study published in JMIR mHealth and uHealth by researchers at Brown University (Fraser, Crossland, Bacher, Ranney, Madsen, and Hilliard) took a more clinically direct approach: it used actual, previously collected data from real patients who had used the Ada Health symptom checker before being seen in an emergency department, then compared those tools’ outputs against the patients’ actual final ED diagnoses and independent physician review.

The results, with exact figures: for top-1 diagnostic accuracy (getting the single most likely diagnosis right), Ada scored 30%, ChatGPT 3.5 scored 40%, ChatGPT 4.0 scored 33%, and WebMD scored 40% — compared to a mean accuracy of 47% for the physicians reviewing the same cases. For top-3 diagnostic accuracy (the correct diagnosis appearing anywhere in the top three suggestions), Ada and ChatGPT 3.5 both scored 63%, ChatGPT 4.0 scored 50%, and WebMD scored 57%, against a 69% mean for physicians. Across every measure, human physicians outperformed every AI tool tested — a genuinely important baseline finding on its own.

The study’s own conclusion is worth quoting directly: ChatGPT 3.5 had high diagnostic accuracy but a high unsafe triage rate, while ChatGPT 4.0 had the poorest diagnostic accuracy of the AI tools but a lower unsafe rate and the highest triage agreement with physicians. In plain terms: getting the diagnosis right and giving safe triage advice are not the same skill, and a tool can genuinely excel at one while struggling with the other.

📊 The Big-Picture Systematic Review: How Wide Is the Range, Really?

A 2025 systematic review published in npj Digital Medicine (Kopka, von Kalckreuth, and Feufel) synthesized findings across 19 included studies (screened from 1,549 candidates) to answer a bigger question: how accurate are these tools overall, and how do they compare not just to each other but to ordinary people making the same decision without any tool at all?

The findings are genuinely striking in their range. Symptom-assessment apps (like NHS 111 online and similar tools) showed self-triage accuracy that was, in the review’s own words, “moderate but highly variable” — spanning from just 11.5% to as high as 90.0% depending on the specific application and study. Large language models showed a narrower, more consistent range of 57.8% to 76.0%. And — this is the part worth sitting with — ordinary laypeople, given the same triage decisions to make without any digital tool at all, scored 47.3% to 62.4%, also with relatively low variability.

🌍 Independent Cross-Validation From a Different Country

It’s worth knowing this pattern isn’t specific to one research team or one healthcare system. A separate 2024 study published in Scientific Reports, conducted in a Turkish emergency department, independently evaluated Ada, WebMD, ChatGPT 3.5, and ChatGPT 4.0 against expert physician judgment using a similar three-tier triage framework. Its results were broadly consistent with the Brown University findings despite being run independently, in a different country’s healthcare system: diagnostic accuracy ranked Ada and ChatGPT 3.5 highest (63% each), with WebMD (57%) and ChatGPT 4.0 (50%) behind; triage accuracy ranked ChatGPT 4.0 highest (76%), followed by WebMD (70%), Ada (62%), and ChatGPT 3.5 lowest (59%).

That two independently conducted studies, in different healthcare systems, converged on such similar numbers is a meaningful sign that these findings reflect something real about the tools themselves, not a quirk of one study’s specific patient sample.

🤖 What Happens as the Models Get Newer?

A more recent analysis, published in 2026 in Communications Medicine (a Nature-family journal), evaluated 22 different ChatGPT model versions specifically on care-seeking advice, using the same standardized 45-vignette approach pioneered by the original Semigran study, with each vignette tested ten times per model to check consistency. The best-performing model in this analysis correctly identified the appropriate care-seeking category in 74% of cases — consistent with the general LLM accuracy range found in the broader systematic review above. But the study also identified a consistent pattern across model versions: all of the models tested tended to overtriage (recommend a higher level of care than necessary) and specifically struggled to correctly identify cases where self-care was actually the appropriate response. This is a meaningfully different error pattern than ChatGPT 3.5’s alarming 41% unsafe rate in the Brown University ED study — it suggests that, in a general vignette-testing context, newer models tend to err toward excessive caution rather than dangerous under-triage, which is a safer failure mode, even though it may also mean unnecessary healthcare visits and system strain if relied on at scale.

🚨 Why “Unsafe Triage” Is the Number That Actually Matters Most

It’s worth being explicit about why this article has spent more space on triage safety than on raw diagnostic accuracy. Getting a diagnosis wrong in the top 3 possibilities is a real limitation, but a symptom checker’s core promise to users isn’t “tell me exactly what I have” — it’s “tell me whether I need to worry.” A tool that’s diagnostically imperfect but consistently and safely tells you to seek care when you should is providing real value. A tool that confidently and specifically diagnoses your condition but then tells you self-care is fine when you’re actually experiencing an emergency — the exact pattern found in ChatGPT 3.5’s 41% unsafe rate — is providing the single most dangerous kind of wrong answer this category can give.

⏰ Why This Matters More Than Ever Right Now

This research isn’t a niche academic concern — it describes tools being used at genuinely enormous scale. Recent industry data indicates a substantial share of U.S. adults have used general AI chatbots for health information in the past year, with a meaningful portion specifically seeking advice about physical symptoms rather than just general information. With major AI companies now building dedicated health-focused products that connect directly to users’ medical records and wearable data, the gap between “general chatbot occasionally asked about symptoms” and “purpose-built health AI system used for real triage decisions” is narrowing quickly — which makes the accuracy and safety research covered in this article more relevant, not less, than when the original 2015 BMJ study was published. It’s also worth noting a distinction the research itself draws out: a general-purpose LLM like ChatGPT was never specifically designed or validated as a medical triage tool the way a purpose-built symptom checker was — yet the Brown University study found people are already using it that way, and its performance in that specific role is exactly what the study measured.

📋 Quick Comparison

✅ Questions to Ask About Any Symptom Checker or Health Chatbot

• Is there a specific, independently published study testing this exact tool’s triage accuracy — not just its general diagnostic knowledge, but specifically whether its “should I seek care” recommendations match physician judgment?
• Does the available research distinguish between “unsafe” (dangerously under-cautious) and “too cautious” errors? As shown above, these are very different problems, and a tool’s overall “accuracy” number can hide a genuinely dangerous unsafe-triage rate.
• Which specific model or version is being used? The research above shows meaningful differences even between ChatGPT 3.5 and 4.0 tested in the same study — “ChatGPT” alone isn’t a specific enough claim to evaluate.
• For anything involving potentially serious or emergency symptoms, would you make the same decision without the tool? Given that laypeople alone already achieve 47–62% self-triage accuracy in published research, a specific tool needs real evidence behind it to be trusted over your own judgment for anything serious.
• Is the tool being used to decide whether to seek care, or just to understand medical information after a diagnosis is already established? These are different, differently risky use cases.

🎯 Who This Is Actually Relevant To

If you’re using a symptom checker or AI chatbot for general health information, understanding lab results, or preparing questions for a doctor’s visit, the evidence base here is much less concerning — these are lower-stakes uses than acute triage decisions, and the research in this article specifically concerns triage and diagnostic accuracy, not general information quality.

If you’re using one of these tools to decide whether a symptom warrants emergency care, urgent care, or can wait for a routine appointment, the research is directly relevant and worth taking seriously: overall accuracy varies enormously by tool, unsafe triage recommendations occur at real, measured, non-trivial rates even in the best-studied tools, and ordinary human judgment without any tool at all already performs in a similar range to many of these products. For anything involving chest pain, difficulty breathing, sudden severe symptoms, or anything that feels like a genuine emergency, calling emergency services or going to an emergency department directly — rather than waiting on any app’s or chatbot’s triage recommendation — remains the evidence-supported response, especially given that a real, statistically significant unsafe-triage rate has been directly measured in at least one widely used tool.

❓ Frequently Asked Questions

How accurate are AI symptom checkers?

It varies enormously by tool. A 2025 systematic review found symptom-checker app accuracy ranging from 11.5% to 90.0%, while AI chatbots (LLMs) clustered more consistently between 57.8% and 76.0%. There is no single accurate answer to “how accurate are they” without specifying which tool.

Is ChatGPT safe to use for deciding whether to go to the ER?

A 2023 Brown University study found ChatGPT 3.5 gave “unsafe” triage recommendations (recommending less care than physicians judged necessary) in 41% of tested emergency department cases — a statistically significantly higher rate than the Ada Health symptom checker tested in the same study (14%). ChatGPT 4.0 performed better on this specific measure but still had a 22% unsafe rate.

Are AI symptom checkers more accurate than just guessing myself?

Research suggests it depends on the tool. A 2025 systematic review found ordinary laypeople achieve 47.3% to 62.4% self-triage accuracy without any digital tool, which overlaps with or exceeds the low end of the symptom-checker app range (11.5% to 90.0%) — meaning some symptom-checker apps may perform worse than simply using your own judgment, while better-performing tools do outperform unaided laypeople.

Do newer AI models make safer recommendations than older ones?

Research is mixed and specific to what’s being measured. In one study, ChatGPT 4.0 had a lower “unsafe” triage rate and higher physician agreement than ChatGPT 3.5, but also had lower diagnostic accuracy. A broader 2026 analysis of 22 ChatGPT versions found a consistent tendency across models to overtriage rather than under-triage, which is a safer error pattern but still not a fully solved problem.

📚 Sources & References

• Semigran, H.L., Linder, J.A., Gidengil, C., Mehrotra, A., “Evaluation of symptom checkers for self diagnosis and triage: audit study,” The BMJ, 2015;351:h3480
• Fraser, H., Crossland, D., Bacher, I., Ranney, M., Madsen, T., Hilliard, R., “Comparison of Diagnostic and Triage Accuracy of Ada Health and WebMD Symptom Checkers, ChatGPT, and Physicians for Patients in an Emergency Department: Clinical Data Analysis Study,” JMIR mHealth and uHealth, 2023;11:e49995
• Kopka, M., von Kalckreuth, N., Feufel, M.A., “Accuracy of online symptom assessment applications, large language models, and laypeople for self-triage decisions,” npj Digital Medicine, 2025;8:178
• “Emergency department triaging using ChatGPT based on emergency severity index principles: a cross-sectional study,” Scientific Reports, 2024
• Kopka, M., et al., “Evaluating the accuracy of ChatGPT model versions for giving care-seeking advice,” Communications Medicine, 2026
• Kopka, M., Napierala, H., Privoznik, M., Sapunova, D., Zhang, S., Feufel, M., “The RepVig framework for designing use-case specific representative vignettes and evaluating triage accuracy of laypeople and symptom assessment applications,” Scientific Reports, 2024

Verification notes: The Fraser et al. 2023 study’s exact figures (diagnostic and triage percentages, unsafe rates, and the p=.009 significance value) were pulled directly from the published study’s results section. The Turkish emergency department study (Scientific Reports, 2024) is presented as independent cross-validation, having been conducted by a separate research team in a different healthcare system, and its consistency with the Fraser et al. findings is noted as a meaningful corroboration rather than a coincidence. The Kopka et al. 2025 systematic review’s wide accuracy range (11.5–90.0% for apps) reflects genuine variability across the 19 studies it synthesized and is not a typo or an outlier being cherry-picked — this range is the review’s own reported finding.

💡 The Bottom Line

A full decade of published research, from a 2015 landmark BMJ study through 2026 large language model comparisons, converges on a consistent, honest picture: AI symptom checkers and health chatbots are neither a solved problem nor a scam — they’re a genuinely wide-ranging category where accuracy depends heavily on the specific tool, the specific version, and what you’re actually measuring. The single most important, most actionable finding is this: at least one major, widely used AI chatbot has been shown in a real clinical study to give unsafe triage advice at a statistically significantly higher rate than a purpose-built symptom checker — a concrete, measured safety difference that should inform which tool you trust with an actual “should I go to the ER” decision, rather than assuming all AI health tools carry the same risk.