{"id":5658,"date":"2026-06-26T13:01:13","date_gmt":"2026-06-26T13:01:13","guid":{"rendered":"https:\/\/houstonmovingjournal.com\/?p=5658"},"modified":"2026-06-26T13:01:13","modified_gmt":"2026-06-26T13:01:13","slug":"heres-how-often-ai-doctors-get-things-wrong-according-to-human-experts","status":"publish","type":"post","link":"https:\/\/houstonmovingjournal.com\/?p=5658","title":{"rendered":"Here’s how often AI \u2018doctors\u2019 get things wrong, according to human experts"},"content":{"rendered":"

Artificial intelligence-powered chatbots such as ChatGPT can accurately answer health-related questions about three-quarters of the time, according to new research from Penn State University, a finding that highlights both the promise and limitations of using AI for medical guidance.<\/p>

Read more 12-year-old girl finds mother dead in apparent Tomball murder-suicide<\/a><\/p>

A team, led by Penn State associate professor and researcher Amulya Yadav, recruited participants to submit more than 200 health-related symptom descriptions and medical questions to a variety of AI systems like ChatGPT. Nine board-certified physicians then evaluated the responses for accuracy and reliability.<\/p>

AI \u2018doctors\u2019 have 76% accuracy rate<\/h2>
<\/div>

By the numbers:<\/p>

The study found that large language models (LLMs) achieved an average accuracy rate of approximately 76% when responding to health questions submitted by users. ChatGPT outperformed other large language models included in the analysis.\u00a0<\/p>

Overall, AI systems performed significantly better than traditional search engines such as Google and Bing when answering health-related questions.<\/p>

\n
\"\"<\/figure>\n

ChatGPT app displayed on a phone screen is seen in illustration. (Credit: Jakub Porzycki\/NurPhoto via Getty Images)<\/span> <\/p> <\/div><\/div>

“I was surprised they [LLMs] performed as well as they did. But certainly I think that, you know, we should all be very wary of trusting anything that comes out of a language model, because it is basically a random token generator,” Yadav told The Associated Press.<\/p>

The findings will be presented this week at the 2026 Association for Computing Machinery Conference on Fairness, Accountability and Transparency (FAccT) in Montreal.<\/p>

Where AI performed best, struggled most<\/h2>

Dig deeper:<\/p>

The study also identified several areas where AI struggled. Questions related to dermatology, mental health and internal medicine generally received lower accuracy scores.<\/p>

Dermatology questions often require image analysis, an area where AI systems remain less capable than they are with text-based information. Mental health questions also pose challenges because they require nuanced judgment.<\/p>

By contrast, AI performed best on more routine health inquiries, including common illnesses and general medical questions.<\/p>

AI becomes resource to people without doctors<\/h2>

According to the\u00a0World Health Organization<\/u>, more than 50% of the world population does not have access to proper health care and full coverage.<\/p>

Read more Doctors warn rapid weight loss from popular medications can cause back pain<\/a><\/p>

Yadav says while the average person with access to a doctor shouldn\u2019t give it up for Dr. AI, it can be an important resource to people across the globe who don\u2019t have access to a doctor.<\/p>

RELATED: <\/strong>AI shaping our health care decisions, even whether we go to the doctor<\/strong><\/p>

“That seems to be a wonderful thing, right? Especially to that half of the population where they don’t have access to a doctor. So that’s one way to spin it,” Yadav said, but warned that “Large language models at this point of time are not as accurate as a human physician, and therefore we should not, or we should be very cautious. That’s our message to the general public that we should be very cautious in using large language models to self-diagnose.”\u00a0<\/p>

More people continue to turn to AI<\/h2>

Big picture view:<\/p>

The findings come as more people turn to AI tools for information traditionally sought from healthcare providers or online searches.\u00a0<\/p>

In fact, for nearly half of Americans, AI plays some sort of role in their health care decisions, a recent Gallup poll found<\/u>.\u00a0<\/p>

In most cases, that role supplements other medical advice or serves as a way to get quick answers. However, in some cases, AI is being used as a substitute for a trip to the doctor\u2019s office altogether.<\/p>