Amazon’s role in research on fairness in AI draws mixed reaction
SEATTLE — Amazon has partnered with the taxpayer-funded National Science Foundation on a three-year, $20 million program to fund basic research into fairness in artificial intelligence systems, which are under increasing scrutiny as they spread in society and sometimes amplify existing biases.
Some researchers welcomed the move, suggesting it signals a growing awareness of the importance of this area of inquiry. Others raised concerns about Amazon’s participation, pointing to potential conflicts of interest for researchers who would scrutinize artificial intelligence technologies at both the company and its competitors.
One researcher likened the arrangement to the fox guarding the hen house.
A grant solicitation released last week is one of the first from the foundation — the foremost federal funder of computer-science research at universities — specifically focused on fairness in AI, though the agency has long supported research in areas such as big-data analysis, computer vision and machine learning that underpin the systems lumped under the catchall term AI.
AI research: The goal of the Amazon-National Science Foundation joint funding is to support research into “transparency, explainability, accountability, potential adverse biases and effects, mitigation strategies, validation of fairness, and considerations of inclusivity” in AI systems, enabling “broadened acceptance” of the systems. The program would parcel out up to $7.6 million — half from Amazon — over three years to as many as nine selected projects.
Even that relatively small sum is a meaningful infusion into a crucial, complex aspect of AI research that has been understudied, as evidenced by a growing catalog of instances in which AI systems have produced racially biased recidivism predictions, misidentified people of color, generated gender-biased language and downgraded resumes from female job applicants for technical roles.
“It’s really quite huge,” said Bill Howe, an associate professor in University of Washington’s Information School, referring both to the funding and Amazon’s participation. He added: “This is the first that’s expressly focused on fairness for AI. So it’s something that’s been a long time coming and it’s good to see it out there.”
An Amazon executive said in a blog post that the company and the National Science Foundation have each committed to $10 million in funding through 2021, and plan additional calls for proposals.
Conflict of interest? Other researchers say Amazon’s participation — as a corporation that fields and profits from the AI technologies in question — is problematic, raising potential conflicts of interest and giving the company insight into the work of a research community that might be scrutinizing its products, and those of its competitors, on the very topics that the grant would fund.
“It puts researchers in an odd predicament,” said Nicholas Weber, an assistant professor, also at the UW’s Information School, who studies research funding and transparency, as well as national research policy. “These corporate co-sponsors are more or less piggybacking on the thorough and unique system of peer review that NSF organizes.”
The commingling of funding from Amazon and the foundation could cause researchers who don’t want to accept money from the company to avoid a conflict of interest not to apply to a federal funding program that is otherwise uniquely tailored to their area of focus, he said.
Sarah T. Roberts, an assistant professor in the UCLA department of information studies, said on Twitter last week: “I’d be the first to love a cut of this funding to support research and researchers but … ever feel like there are a lot of foxes guarding the hen house?”
Tension: The varied reactions to Amazon’s co-sponsorship of this federal research program point to a tension between the technology giants that develop and deploy AI, and their erstwhile collaborators in academia.
There is a long history of industry-academic collaboration, particularly in the technology sector, with universities providing basic research and trained professionals, and their industrial partners providing guidance, funding and employment opportunities. The funding from the National Science Foundation is an increasingly important conduit.
But there are examples of that relationship becoming adversarial as academic researchers identify and publicize the shortcomings and negative repercussions of AI systems.
A study by Deborah Raji of the University of Toronto and Joy Buolamwini of MIT Media Lab found Amazon’s facial-recognition technology, Rekognition, was worse at identifying gender when analyzing female and darker-skinned faces than competing technologies from IBM and Microsoft. Amazon pushed back on the researchers’ methodology and findings and said a more recent version of the technology was not tested.
“Their claims of being bias free are based on internal evaluations,” Buolamwini wrote in a response earlier this year. “This is why we did an external evaluation to provide an outside perspective.”
No competitive advantage: The Amazon-NSF grant program, like other corporate partnerships at the National Science Foundation, is structured to maintain research independence and was evaluated to ensure it gives Amazon no competitive advantage, said spokesman Robert Margetta.
“There’s a mutual understanding between NSF and Amazon, as with all public private partnerships that we enter in to, that the purpose of this is to advance fundamental research,” he said.
Corporate research partnerships are not new at the foundation. Margetta said the computer-science directorate within the agency has run corporate partnerships for at least a decade, with an increased emphasis in recent years.
Meanwhile, the foundation’s budget is shrinking. The agency’s budget — mostly allocated to funding basic research — is nearly $8.1 billion in the current federal fiscal year. Its proposed 2020 budget cuts funding by more than a $1 billion. (National Science Foundation-funded projects at the UW amounted to $116 million last fiscal year.)
Looking at bias: Miriam Vogel, executive director of EqualAI, one of a growing number of nonprofits and research groups focused on issues of racial and gender bias in AI, said it’s encouraging to see Amazon and the foundation take this step. Vogel previously worked at the Department of Justice, leading a program to train federal law enforcement agents to recognize implicit or unconscious bias.
Many of the biased outcomes produced by AI systems today reflect biases in the data sets that train them. Those data sets, such as arrest records or hiring patterns, may contain racial or gender biases.
Vogel said researchers and the industry need to develop tools that can identify those latent biases before they are perpetuated and amplified by the AI systems.
“This is not an insignificant problem,” she said. “This is something that they need to be aware of.”