Making AI models more trustworthy for high-stakes settings

In high-stakes environments such as medical diagnosis, the ambiguity inherent in complex data presents significant challenges for clinicians. For instance, distinguishing between pleural effusion and pulmonary infiltrates on a chest X-ray can be incredibly difficult, yet crucial for patient care. While artificial intelligence models promise to augment human expertise by identifying subtle details and boosting diagnostic efficiency, their predictions often lack the nuanced reliability required for critical decisions.

Current AI approaches, like conformal classification, aim to provide a set of plausible diagnoses rather than a single prediction, offering a crucial safety net for clinicians. However, these sets can often be impractically large, overwhelming users with too many possibilities and hindering the very efficiency they aim to improve. This dilemma has been a persistent hurdle in the broader adoption of AI in sensitive fields.

Breaking new ground, researchers at MIT have unveiled a simple yet profoundly effective improvement that promises to revolutionize the trustworthiness of AI models. Their innovative method significantly reduces the size of these prediction sets by up to 30 percent, all while enhancing the reliability of the AI’s output. This breakthrough could dramatically streamline the diagnostic process, allowing clinicians to focus more efficiently on the most probable conditions, ultimately leading to improved and expedited treatment for patients.

This advancement holds vast potential beyond medical imaging, applicable across a diverse range of classification tasks, from identifying animal species in wildlife photography to complex industrial inspections. “With fewer classes to consider, the sets of predictions are naturally more informative in that you are choosing between fewer options. In a sense, you are not really sacrificing anything in terms of accuracy for something that is more informative,” explains Divya Shanmugam PhD ’24, a postdoc at Cornell Tech who led this research during her time as an MIT graduate student.

Shanmugam collaborated with Helen Lu ’24, Swami Sankaranarayanan (a former MIT postdoc and current research scientist at Lilia Biosciences), and senior author John Guttag, the Dugald C. Jackson Professor of Computer Science and Electrical Engineering at MIT and a distinguished member of the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL). Their groundbreaking work is set to be formally presented at the prestigious Conference on Computer Vision and Pattern Recognition in June.

Traditionally, AI assistants for critical tasks often provide a probability score alongside each prediction, indicating the model’s confidence. Yet, research has frequently shown these probabilities to be inaccurate, undermining user trust. Conformal classification attempts to rectify this by offering a set of probable diagnoses, guaranteed to contain the correct one. The inherent uncertainty in AI, however, frequently results in sets too vast to be practically useful—for instance, an animal classification model might output 200 predictions from 10,000 potential species just to offer a strong guarantee.

“That is quite a few classes for someone to sift through to figure out what the right class is,” Shanmugam notes. Furthermore, these sets can be unreliable, with minor input changes (like image rotation) yielding entirely different predictions.

To address this, the MIT team ingeniously integrated Test-Time Augmentation (TTA), a technique previously used to boost computer vision model accuracy. TTA generates multiple augmented versions of a single image (e.g., cropped, flipped, zoomed) and aggregates the model’s predictions from each version. As Shanmugam elucidates, “In this way, you get multiple predictions from a single example. Aggregating predictions in this way improves predictions in terms of accuracy and robustness.”

The researchers applied TTA by holding out a portion of labeled image data for the conformal classification process. On these held-out data, they learned to aggregate augmentations in a way that maximizes the underlying model’s prediction accuracy. Subsequently, conformal classification was run on these new, TTA-transformed predictions. The result? A significantly smaller set of probable predictions, crucially maintaining the original confidence guarantee. This innovative combination of TTA with conformal prediction is “simple to implement, effective in practice, and requires no model retraining,” according to Shanmugam.

Across various standard image classification benchmarks, their TTA-augmented method consistently reduced prediction set sizes by 10 to 30 percent compared to prior work in conformal prediction. Remarkably, this reduction was achieved while preserving the essential probability guarantee. The team also discovered that the accuracy boost from TTA outweighed the cost of sacrificing some labeled data normally used for conformal classification, prompting intriguing questions about future data allocation strategies in post-training steps. Supported in part by the Wistron Corporation, this research opens doors to more reliable and efficient AI systems for the most critical applications, ensuring a future where AI truly assists rather than overwhelms. Future work will explore applying this approach to text classification and optimizing TTA for computational efficiency.