Bias in Artificial Intelligence

Imagine an algorithm that selects nursing candidates for a multi-specialty practice—but it only selects white females. Consider a revolutionary test for skin cancer that does not work on African Americans. What about a model that directs poorer patients to a skilled nursing facility rather than their home as it does for wealthier patients? These are ways in which ungoverned artificial intelligence (AI) might perpetuate bias.

With the current hyperbole around AI approaching an all-time high, it takes little imagination to see how the algorithms applied in other industries can be used in health care. Google algorithms for automated image classification can be modified to read CT scans of a patient with cancer [1] or predict treatable blinding retinal diseases [2]. The AI methods used to predict the risk of loan default can be tweaked to predict the risk of sepsis [3, 4] or pneumonia [5].

In health care, clinical decision support has long been integrated into our electronic health records to guide safe medication use, use of clinical best practices, and prioritization of high-risk patients. We could, of course, choose to ignore any algorithm's suggestions much like we might snub Amazon's book recommendations. However, when AI systems go beyond recommendations and act autonomously, we must pause and consider the implications. At best, we streamline processes, reduce variation in care, and remove human biases from decision-making [6-8]. At worst, we erode trust, perpetuate gender, ethnicity, and income disparities, and distance ourselves from patient care decisions.

There is ample evidence of bias in AI [9]. Also known as algorithmic bias, it is what we experience when a machine-learning model produces a systematically wrong result. Just as this article is a reflection of the bias of its author, algorithms have authors and are assembled according to instructions made by people. Bias is a reflection of the data algorithm authors choose to use, as well as their data blending methods, model construction practices, and how results are applied and interpreted. That is to say, these processes are driven by human judgments.

Health care is one of the most challenging industries when it comes to data, primarily due to the fact that the industry's operational systems were not designed for modern analytics and are often not fully integrated with internal or external data systems. We are still learning about the full spectrum of factors that determine health outcomes [10-12]. Sadly, most health care organizations still grapple with issues like data quality, data governance, and effective use of Health IT to improve outcomes. That is to say, we may use the data that we have as opposed to the data that is “right”.

We are plagued with data that cannot be integrated across people, time, or place; information collected for billing purposes which does not fully reflect the underlying diagnosis and treatment [13, 14]; and data collection practices that are highly biased toward those who can afford health care services. We often see this manifested in patients' access to care where the data in the EHR can be shallower for some segments of the population [15] or in curated health care data that is resold by brokers where bias exists toward those who can afford devices, applications, and technology [16].

As we evaluate sources of bias in our models, it is essential that we establish principles to guide our work. We must adopt four primary tenets: transparency, trust, fairness, and privacy.

Transparency stresses the responsibility of AI authors to explain not only what went into an algorithm and its results, but what decisions they made and why. The goal is to understand the process by which an algorithmic system makes decisions, and we must ensure the model can be explained. Often called the “black-box problem,” this challenge often poses issues for physicians who seek insight into what the AI is doing.

Trust begins with transparency, verification, and accountability. As Dr. Wyatt Decker, the Mayo Clinic's chief medical information officer, points out: “… clinician involvement is important no matter how smart the machines get. There is a strong need for the engagement of medical experts to validate and oversee AI algorithms in healthcare” [17].

“Fairness” is a social construct, and in the context of bias in AI we are referring to being responsible for social mores. Algorithms are discriminatory in that they seek tiny patterns of influence in the data. Anthropomorphically speaking, we want a model that is socially responsible—one that does not discriminate against people based on traits that we would generally consider protected (eg, age, gender, sexual orientation, race, or ethnicity).

Privacy reflects on the nature of our relationship with our patients. While there are certainly cases of people using geospatially derived variables, purchase history, and social media data to augment the medical records, we must ensure the protection of individual privacy at all times.

There is a growing body of work in the legal [18], regulatory [19], and ethical oversight of AI models [20]. These sources ask that we look beyond the technical processes for data selection, model building, and validation and adopt formal AI governance strategies. In this context, AI governance is in the process of assigning and assuring organizational accountability, decision rights, risks, policies, and investment decisions for applying artificial intelligence. Newly proposed federal legislation, The Algorithmic Accountability Act of 2019, would require businesses to conduct an impact assessment that covers the risk associated with algorithms' accuracy, fairness, bias, dis-crimination, privacy, and security [21]. Surely health care will fall under the umbrella of such legislation. Research and consulting firm Gartner, Inc. predicts that by 2022 the first US medical malpractice case involving a medical decision made by an advanced AI algorithm will have been heard [22]. It will not be because an algorithm produced an incorrect diagnosis. “It will be due to the failure to use an algorithm that was proven to be more accurate and reliable than the human alone [22].”

It is our professional and moral obligation to do what we can to ensure that AI is safe for our patients and care teams. Given the tenets of trust, fairness, transparency, and privacy, we should focus on solutions that may help care teams automate the activities that take them away from their patients, not on replacing them. Continued advancement of AI in health care will require stakeholder education and the management of expectations so that we can eradicate unintentional bias and engender trust in transparent, clinically validated models. After all, as Manu Tandon, chief information officer of Beth Israel Deaconess Medical Center in Boston, suggests, “We are not looking for robots to do work for us, we are looking to make better decisions by benefiting from machine learning and AI” [17].

• ©2019 by the North Carolina Institute of Medicine and The Duke Endowment. All rights reserved.