MIT Researchers Develop New Method to Improve Radiologists’ Diagnostic Accuracy
A multidisciplinary team of researchers at MIT, in collaboration with Harvard Medical School-affiliated hospitals, has developed a novel framework to assess and improve the reliability of radiologists’ diagnostic reports. This innovative approach addresses the inherent ambiguity in medical images, such as X-rays, where radiologists often use terms like “may” or “likely” to describe the presence of a pathology.
The study reveals that radiologists tend to be overconfident when using phrases like “very likely” and underconfident when using less assertive terms like “possibly.” To tackle this, the team created a system that quantifies the reliability of certainty phrases used by radiologists, leveraging clinical data to provide suggestions for more accurate clinical reporting.
Peiqi Wang, an MIT graduate student and lead author of the paper, emphasizes the importance of precise language in radiology reports, stating, “The words radiologists use are important. They affect how doctors intervene, in terms of their decision making for the patient. If these practitioners can be more reliable in their reporting, patients will be the ultimate beneficiaries.”
The research team, led by senior author Polina Golland, a professor at MIT CSAIL, includes Barbara D. Lam, Yingcheng Liu, Ameneh Asgari-Targhi, Rameswar Panda, William M. Wells, and Tina Kapur. Their work focuses on decoding uncertainty in the language used by radiologists. For instance, a radiologist might describe a chest X-ray as showing a “possible” pneumonia, leading to a follow-up CT scan. Alternatively, a “likely” pneumonia diagnosis might prompt immediate treatment.
The challenge lies in measuring the calibration, or reliability, of ambiguous natural language terms. Unlike AI models with confidence scores, human language doesn’t easily translate to precise probabilities. The researchers’ approach treats certainty phrases as probability distributions, capturing more nuances of word meaning.
The team leveraged prior surveys of radiologists to obtain probability distributions for diagnostic certainty phrases. By comparing predicted probability scores with actual results, they formulated an optimization problem to adjust phrase usage, aligning confidence with reality. This calibration map suggests specific terms a radiologist should use for more accurate reporting.
Findings from clinical reports revealed that radiologists were often underconfident with common conditions like atelectasis but overconfident with ambiguous conditions like infection. The framework also evaluated the reliability of language models, offering a more nuanced confidence representation.
Future plans include expanding the study to abdominal CT scans and assessing radiologists’ receptiveness to calibration-improving suggestions.
Atul B. Shinagare, associate professor of radiology at Harvard Medical School, notes, “This study takes a novel approach to analyzing and calibrating how radiologists express diagnostic certainty in chest X-ray reports, offering feedback on term usage and associated outcomes. This approach has the potential to improve radiologists’ accuracy and communication, which will help improve patient care.”
The research was funded by a Takeda Fellowship, the MIT-IBM Watson AI Lab, the MIT CSAIL Wistron Research Collaboration, and the MIT Jameel Clinic.
