Preeti
Researchers at the Icahn School of Medicine at Mount Sinai conducted a study that revealed the limitations of large language models (LLMs) in medical coding. The study, published in NEJM AI, found that LLMs from OpenAI, Google, and Meta had limited accuracy in reproducing medical codes, with GPT-4 showing the best performance but still falling below 50 percent accuracy. The study emphasized the need for refinement and validation of these technologies before clinical implementation.
The researchers extracted over 27,000 unique diagnosis and procedure codes from routine care at Mount Sinai Health System and compared the codes generated by the LLMs with the original codes. They found that while GPT-4 showed the highest exact match rates, there were still a significant number of errors. GPT-3.5, the next best-performing model, tended to be more vague in its code generation.
The study authors highlighted the importance of rigorous evaluation and refinement of AI technologies before deploying them in sensitive areas like medical coding. They suggested that integrating LLMs with expert knowledge could automate medical code extraction, potentially improving billing accuracy and reducing administrative costs in healthcare.
The researchers plan to develop tailored LLM tools for accurate medical data extraction and billing code assignment in the future. Overall, the study underscores the need for caution and ongoing development in utilizing AI in healthcare to ensure reliability and efficacy.
Source link
Source link: https://scitechdaily.com/ai-falls-short-large-language-models-struggle-with-medical-coding-study-shows/
GIPHY App Key not set. Please check settings