Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models

Summary: The researchers evaluated the performance of ChatGPT, a large language-based AI, on the United States Medical Licensing Exam (USMLE). The USMLE is a set of three standardized tests required for medical licensure in the US. The results showed that ChatGPT performed at or near the passing threshold (60%) on all three exams without specialized training or reinforcement. The AI also demonstrated a high level of concordance and insight in its explanations, suggesting that it may have potential to assist with medical education and potentially, clinical decision-making in the future. This marks a notable milestone in AI maturation and increased confidence in its trust and explainability.

(By ChatGPT, reviewed and edited)

Article: Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models – PLOS Digital Health

News Release: ChatGPT can (almost) pass the US Medical Licensing Exam – PLOS

Commentaries:

Expert reaction to study on ChatGPT almost passing the US Medical Licensing Exam – Science Media Centre

New and surprising evidence that ChatGPT can perform several intricate tasks relevant to handling complex medical and clinical information – News Medical

Commentary from one of the authors on Twitter (thread – click for more)

our work showing that ChatGPT passed the US Medical Licensing Exam (USMLE) is published @PLOSDigiHealth!

there's tons of chatter about LLMs in healthcare.

AI is not going to replace doctors anytime soon, but here are the key takeaways from this work:

? https://t.co/OQs2Tlfwjx pic.twitter.com/v3UWjScLth

— Morgan Cheatham (@morgancheatham) February 13, 2023