AI Education Must Address Data Bias: MIT Expert Urges Curriculum Reform
In an era where artificial intelligence is increasingly integrated into critical fields like medicine, a significant gap exists in how students are trained to deploy these models. Leo Anthony Celi, a senior research scientist at MIT’s Institute for Medical Engineering and Science, is advocating for a crucial addition to AI education: training students to identify and address biases within the datasets used to develop AI models.
Celi’s concerns, highlighted in a new paper, stem from the observation that many AI courses focus predominantly on model building, often overlooking the critical evaluation of the data underpinning these models. He emphasizes that models trained primarily on data from specific demographics, such as white males, frequently exhibit poor performance when applied to other populations.
The Problem of Bias in AI Datasets
Bias seeps into datasets in various ways. Celi points to the example of pulse oximeters, which were found to overestimate oxygen levels in people of color due to insufficient representation in clinical trials. He also notes that medical devices are typically optimized for healthy, young males, neglecting the diverse needs of a broader patient population. The FDA’s requirement for device approval based on performance in healthy subjects further exacerbates this issue.
Electronic health record (EHR) systems, often used as building blocks for AI, also present challenges. These records were not designed for AI learning, necessitating careful consideration when used for algorithm development. Celi suggests exploring transformer models of numeric EHR data to mitigate the impact of missing data and provider biases.
The Need for Curriculum Reform
Celi’s concerns led him to analyze AI courses, revealing that many fail to adequately address potential biases in datasets. His research found that of 11 courses reviewed, only five included sections on bias, with only two containing significant discussions on the topic.
He stresses the importance of equipping students with the agency to work critically with AI. His hope is that the study will highlight the importance of data literacy within AI education.
Incorporating Critical Thinking
Celi suggests incorporating a checklist of questions to guide students in evaluating data sources. These questions should address the origin of the data, the observers involved in data collection, and the characteristics of the institutions from which the data was obtained. He highlights the importance of understanding sampling selection bias, such as who gets admitted to the ICU.
The MIT Critical Data consortium organizes datathons, bringing together diverse groups of healthcare professionals and data scientists to analyze health and disease in local contexts. Celi emphasizes that critical thinking thrives in diverse environments, fostering a deeper understanding of data and its limitations.
Celi urges students to resist building models until they thoroughly understand the data’s origins, the patients included, and the accuracy of measurement devices across different individuals. He also encourages the use of local datasets to ensure relevance and uncover potential issues.
Acknowledging that data collection is an iterative process, Celi encourages students to embrace imperfections and learn from mistakes. The Medical Information Mart for Intensive Care (MIMIC) database, he notes, took a decade to develop a decent schema, thanks to ongoing feedback and improvements.
Ultimately, Celi aims to instill a sense of awareness regarding the potential problems in data. He emphasizes that fostering critical thinking is key to realizing the immense potential of AI while mitigating the risk of harm.
