3 Questions: Helping Students Spot Bias in AI Datasets | Proaitools
In an era where artificial intelligence models are increasingly deployed in critical sectors like healthcare, a significant gap exists in the training of future AI practitioners. Many courses focus on the deployment of AI models, such as those assisting doctors in diagnosing diseases and determining treatments, often overlook a vital aspect: the detection of flaws within the training data itself. This oversight can lead to biased AI, impacting the efficacy and fairness of these tools.
Leo Anthony Celi, a senior research scientist at MIT’s Institute for Medical Engineering and Science, a physician at Beth Israel Deaconess Medical Center, and an associate professor at Harvard Medical School, highlights these concerns in a new paper. Celi advocates for integrating more thorough data evaluation into AI education. He points out that AI models predominantly trained on clinical data from specific demographics, such as white males, often underperform when applied to broader populations. The implications of such biases are far-reaching and necessitate a shift in how AI is taught.
In a recent interview, Celi addressed key questions about bias in AI datasets and offered insights into how educators can better equip students to tackle these challenges:
Q: How does bias infiltrate datasets, and what measures can be taken to mitigate these shortcomings?
A: “Any inherent problems within the data will inevitably be reflected in the AI model. Instances of instruments and devices that do not function uniformly across different individuals underscore this issue. For example, pulse oximeters have been shown to overestimate oxygen levels in people of color due to inadequate representation in clinical trials. Students need to understand that medical devices are often optimized for healthy young males, neglecting the diverse patient population they serve. The FDA’s current requirements do not mandate device efficacy across diverse demographics, which further exacerbates the problem.”
Celi also cautions against relying on electronic health record systems as the primary building blocks for AI, noting that these records were not designed as learning systems. He suggests exploring transformer models of numeric electronic health record data to mitigate the effect of missing data due to social determinants of health and provider implicit biases.
Q: Why is it crucial for AI courses to address the sources of potential bias? What did your analysis of course content reveal?
A: “Our course at MIT, initiated in 2016, recognized the danger of encouraging students to prioritize model performance metrics without adequately considering the underlying data’s flaws. This prompted us to investigate how widespread this issue is. Alarmingly, many online courses and syllabi neglect to emphasize the importance of scrutinizing data. Our review of 11 courses indicated that only five included sections on bias, and only two provided significant discussions on the topic.”
Despite this, Celi acknowledges the value of these courses but stresses the need for them to evolve. “As more individuals engage with AI, it’s imperative that they possess the skills to work with AI responsibly. We hope our paper will highlight the existing gaps in AI education.”
Q: What specific content should course developers integrate into their curricula?
A: “Firstly, provide students with a checklist of critical questions to ask upfront: Where did the data originate? Who were the observers and data collectors? Understanding the institutional landscape is crucial. For ICU databases, it’s important to consider who gets admitted and who doesn’t, as this introduces sampling selection bias. If minority patients face barriers to ICU admission, the resulting models will be ineffective for them. Ideally, 50 percent or more of the course content should focus on understanding the data, as modeling becomes more straightforward with a solid grasp of the data’s context.”
Celi also emphasizes the importance of critical thinking, advocating for diverse datathons that bring together individuals from various backgrounds to analyze health and disease within local contexts. He notes that critical thinking emerges naturally when people from different generations and fields collaborate.
“We urge our students not to commence model building until they thoroughly understand the data’s origin, the patients included, the devices used for measurement, and the consistency of device accuracy across different individuals,” Celi advises.
By acknowledging the imperfections in data and fostering critical thinking, educators can empower students to harness AI’s potential while minimizing the risk of harm. Celi’s work aims to inspire a shift in AI education, emphasizing data understanding as a cornerstone of responsible AI development.
</n
