New York City announced a recent effort to end race adjustments in clinical algorithms used by the city’s hospitals that experts say perpetuate racist assumptions in healthcare and can lead to substandard care for minority populations.
Clinical algorithms are used by healthcare facilities to assist in how medical care is doled out to patients and many hospitals use a process called race norming – also known as race adjustment – that has been proven to negatively impact the kind of care Black and Latino residents of the city receive.
In a statement, the city’s experts used multiple examples of real-world situations that have affected the city’s residents, including an “adjustment” factor for Black patients, in reference to kidney function, that calculates Black kidney function levels to be healthier than white patients for the same measured result. This has led to delays in care needed by Black patients.
Race norming in maternal care has led to Black and Latino mothers being more likely to have unnecessary caesarean sections despite having the same age, health status and past birthing history as White women. These algorithms more generally have led to less favorable outcomes for Black and Latino people who give birth and exacerbated existing maternal and health inequities, according to the city.
As a way to address this, the New York City Health Department announced the formation of the Coalition to End Racism in Clinical Algorithms (CERCA) to end the inclusion of race adjustment in clinical algorithms.
“The Coalition to End Racism in Clinical Algorithms will help lead to a healthier and more equitable city” said Health Commissioner Dr. Dave Chokshi.
Deputy Commissioner and Chief Medical Officer, Dr. Michelle Morse, said raising awareness about the problematic concept and practice of race adjustment in clinical algorithms has the potential to create more equitable healthcare outcomes and experiences for those who identify as Black, Indigenous, and People of Color.
NYC Health + Hospitals President and CEO Mitchell Katz added that the city’s public health system is looking forward to collaborating with the health department and other health systems.
“This coalition has the power to further accelerate the work and impact of our system’s existing ‘Medical Eracism’ initiative that also aims to challenge the status quos of race-based algorithms present in diagnosis and treatment that have been widely accepted in our field for decades,” Katz said.
CERCA has 12 members – NYC Health + Hospitals, Maimonides Medical Center, MountSinai Health System, New-York Presbyterian, Northwell Health, NYU Langone Hospitals, One Brooklyn Health, SBH Health System, SUNY Downstate, Wyckoff Heights Medical Center, and Cortelyou Medical Associates.
Each has said they will end race adjustment in at least one clinical algorithm and create plans for the evaluation of racial inequities and patient engagement. They also plan to meet bi-monthly for two years and will produce an annual report, the first of which will come in June 2022.
Black medical experts have long taken issue with race adjustments used in the algorithms that hospitals deploy for treatment. Dr. Dorothy Roberts, professor of Africana studies, Law and Sociology at the University of Pennsylvania, said race adjustments in medicine are “not only based on false, racist beliefs about human biology, but also have been shown to harm Black patients.”
Caitlin Seeley George, campaign director at technology advocacy group Fight for the Future, said race adjustments are not only an “extremely deplorable medical practice based on racist beliefs,” but are also a prime example of how artificial intelligence can exacerbate these racist beliefs.
“The concept of biological differences based on race existed long before AI, but this technology has been weaponized to exacerbate these racist practices. There are so many algorithms in place that mostly work in a black box with little-to-no transparency into how they make their decisions. In order to tear down racist structures across our society we need to open up these algorithms to find what other racist beliefs are a part of their calculations,” Seeley George said.
“We need all of our health systems to do this work, we need legislation that stops harmful algorithms from being adopted, and in New York City it must go beyond one clinical algorithm (which is their current scope). And these efforts must also include the real people who have been harmed by these algorithms as well as human rights organizations and experts who can give critical insight into the harms of these algorithms on real people.”
Dr. Danya Glabau, assistant professor at the NYU Tandon School of Engineering and director of the Science and Technology Studies Department of Technology, Culture and Society at the university, said “algorithms” in the context of medicine have a much longer history than the computerized systems that the council seems to be preparing to address.
Glabau has written extensively about how mundane technologies can have a significant effect on health outcomes.
Medical algorithms, according to Glabau, are essentially any kind of decision tree doctors use to make treatment decisions, and using non-computerized medical algorithms has been a cornerstone of evidence-based medicine for decades.
Digital, automated algorithms, however, take the physician’s judgment out of treatment decisions to a greater or lesser extent because a computer makes the decisions, she explained, adding that when the digital algorithms were rolled out, it was thought that removing humans would remove human racism.
“However, since automated algorithms’ decisions are based on data from past human decisions, human biases like racism and classism still factor into these tools. So they don’t really solve the problem of racism on their own because the history of medicine is racist,” Glabau said.
“It’s hard to say exactly how widespread digital algorithms are and how many of them look at race in particular. But the chances are that most providers use several on a daily basis, and may or may not be aware of it. We know that digital algorithms are built into a wide variety of tools in medicine today, from scheduling systems to medical imaging tools to diagnostics technology to treatment delivery devices. Part of the problem with pinning down how many there are and how many take race into account is that these algorithms tend to be proprietary and how they take different data or factors into account isn’t always clearly revealed.”
Glabau explained that another issue is that organizations that adopt these algorithms may not have people with the programming or ethical expertise involved in the review process to appropriately vet the possibility that algorithms may discriminate. Even algorithms that do not take race into account, she added, explicitly may pick up on it indirectly via things like zip code.
The council needs to investigate how many digital algorithms are shaping care, how many take race into account, and how many take other factors into account that might correlate with, or act as a proxy for, race, Glabau told ZDNet.
Researchers like Amy Moran-Thomas have shown that even simple devices like pulse oximeters can have racist outcomes.
“In this case, designers simply did not consider how skin color would affect the readings given by an optical sensor. We also know that tools like scheduling software can have racist outcomes even though scheduling doesn’t seem to have anything to do with race. But Black patients in particular were double or triple booked because many had difficulties making it to appointments on time due to factors outside of their control,” Glabau said.
“These examples show how tricky it can be to anticipate how algorithmic and other digital systems will have racist outcomes. In a city like New York, where COVID has hit BIPOC communities hard and where zip code is correlated with income and racial segregation, such seemingly mundane technologies can have significant consequences for health.”
But the only way for the council to succeed is for it to be given full access to hospital system operations, software and technical documentation from the companies that produce the algorithms, Glabau added.
They also need to be given the space to make binding guidelines that can be implemented across the city.
“If this council isn’t given teeth, it may find shocking information or make well-intended recommendations, but it will not change anything for patients or accomplish its anti-racist mandate,” Glabau explained.
“On the tech company side, greater transparency and regulation of medical algorithms is absolutely necessary. The FDA has revised and tightened their review procedures for medical algorithms since 2017, which I view as a positive first step. Race and racism should be an explicit measure of review for approval. On the hospital and medical provider side, greater in-house expertise in ethics and social science is necessary to understand the possible implications of using new algorithmic systems. Experts in computer and data science should also be involved in the procurement process so that health care organizations fully understand how the algorithm is designed, how it uses past and new patient data, and how it makes recommendations.”
