MIT Researchers Develop New Method to Improve Radiologists’ Diagnostic Accuracy Using AI
A multidisciplinary team of researchers at MIT, in collaboration with Harvard Medical School-affiliated hospitals, has developed a novel framework to assess and improve the reliability of radiologists’ diagnostic reports. This innovative method addresses the inherent ambiguity in medical images by quantifying the certainty phrases radiologists use, such as “may” or “likely,” when describing pathologies like pneumonia.
The study reveals that radiologists tend to be overconfident when using phrases like “very likely” and underconfident with terms like “possibly.” To address this, the researchers created a framework that utilizes clinical data to provide suggestions for radiologists to choose more accurate certainty phrases, enhancing the reliability of their clinical reporting.
Peiqi Wang, an MIT graduate student and lead author of the research paper, emphasizes the importance of precise language in radiology reports: “The words radiologists use are important. They affect how doctors intervene, in terms of their decision-making for the patient. If these practitioners can be more reliable in their reporting, patients will be the ultimate beneficiaries.”
The team’s approach treats certainty phrases as probability distributions rather than single numerical values, capturing more nuances in their meaning. By leveraging prior surveys of radiologists, the researchers obtained probability distributions corresponding to various diagnostic certainty phrases. This allows them to compare a model’s predicted probability scores with actual results, improving the alignment of confidence with reality.
The framework also formulates an optimization problem that adjusts the frequency of certain phrases to better align confidence with the accuracy of diagnoses. This results in a calibration map suggesting which certainty terms a radiologist should use for more accurate reports. For instance, the study found that radiologists were often underconfident when diagnosing common conditions like atelectasis but overconfident with ambiguous conditions like infection.
The researchers also applied their method to evaluate the reliability of language models, offering a more nuanced representation of confidence compared to classical methods that rely on confidence scores. They noted that language models often use phrases like “certainly,” which may discourage verification of the statements’ correctness.
Senior author Polina Golland, a professor at MIT, along with other researchers including Barbara D. Lam, Yingcheng Liu, Ameneh Asgari-Targhi, Rameswar Panda, William M. Wells, and Tina Kapur, presented the research at the International Conference on Learning Representations. The team plans to expand their study to include data from abdominal CT scans and explore how receptive radiologists are to calibration-improving suggestions.
Atul B. Shinagare, associate professor of radiology at Harvard Medical School, who was not involved in the study, comments, “This study takes a novel approach to analyzing and calibrating how radiologists express diagnostic certainty in chest X-ray reports, offering feedback on term usage and associated outcomes. This approach has the potential to improve radiologists’ accuracy and communication, which will help improve patient care.”
