AI Education Must Address Dataset Bias: MIT Expert

In an era where artificial intelligence is increasingly integrated into critical fields like medical diagnostics, a significant gap exists in how students are trained to develop AI models. Many courses overlook the crucial step of evaluating potential biases within the datasets used to train these models. Leo Anthony Celi, a senior research scientist at MIT, a physician at Beth Israel Deaconess Medical Center, and an associate professor at Harvard Medical School, highlights this issue in a new paper, urging educators to emphasize thorough data assessment.

Celi points out that AI models trained primarily on data from specific demographics, such as white males, often exhibit poor performance when applied to other populations. This bias can lead to inaccurate diagnoses and inappropriate treatments for underrepresented groups. In an interview, Celi addresses how these biases infiltrate datasets and what educators can do to rectify this critical oversight.

Q: How does bias get into these datasets, and how can these shortcomings be addressed?

A: Celi explains that any flaws present in the data will inherently influence the modeling outcomes. He cites the example of pulse oximeters, which were found to overestimate oxygen levels in people of color due to insufficient representation in clinical trials. “We remind our students that medical devices and equipment are optimized on healthy young males. They were never optimized for an 80-year-old woman with heart failure, and yet we use them for those purposes,” Celi notes. He also cautions against relying heavily on electronic health record systems, as they were not designed as learning systems and may contain inherent biases.

To mitigate these issues, Celi suggests exploring innovative solutions like transformer models of numeric electronic health record data, which can model relationships between laboratory tests, vital signs, and treatments to offset missing data caused by social determinants of health and provider biases. A paper on transformer model can be found here.

Q: Why is it important for courses in AI to cover the sources of potential bias? What did you find when you analyzed such courses’ content?

A: Recognizing the problem in their own MIT course, Celi and his colleagues realized they were inadvertently encouraging students to prioritize model performance over data quality. “Our suspicion was that if you looked at the courses where the syllabus is available online, or the online courses, that none of them even bothers to tell the students that they should be paranoid about the data,” Celi states. An analysis of 11 courses revealed that only five included sections on bias, and only two discussed it in depth.

Despite acknowledging the value of these courses, Celi stresses the need for a greater emphasis on teaching the right skill sets. “It’s important for people to really equip themselves with the agency to be able to work with AI. We’re hoping that this paper will shine a spotlight on this huge gap in the way we teach AI now to our students.”

Q: What kind of content should course developers be incorporating?

A: Celi recommends starting with a checklist of questions to evaluate data sources, observers, and the landscape of institutions where data is collected. Understanding who makes it to the ICU, for example, can reveal sampling selection biases. Celi argues that understanding the data should constitute at least 50% of the course content, as modeling becomes straightforward once the data is properly understood.

The MIT Critical Data consortium organizes datathons worldwide, bringing together diverse professionals to analyze health and disease in local contexts. Celi emphasizes that critical thinking flourishes when people from different backgrounds and generations collaborate. He advises students to thoroughly understand the data’s origin, patient selection, and device accuracy before building any models. He encourages to look for data sets that are local, so that they are relevant.

Celi concludes by advocating for continuous improvement and acknowledging that initial data collection may be flawed. He expresses enthusiasm for the potential of AI but underscores the immense risk of harm if data biases are not addressed. Those attended a datathon said that their world has changed and they realize the immense potential. Celi’s work aims to inspire a shift in AI education, fostering critical thinking and responsible AI development.