Today, as artificial intelligence (AI) permeates everyday life, questions are emerging about its fairness. A new study from Penn State University and Oregon State University examined whether ordinary people can recognize racial bias in the data used to train AI systems for facial and emotion recognition. The results are troubling: most users did not notice the bias at all, even when the data was right in front of them.
Experiments and human inattention
The researchers conducted three experiments with a total of 769 participants. In the first experiment, they showed people data in which happy faces predominantly belonged to white people, while sad faces were mostly Black. The AI thus learned to associate race with emotions, which led to errors—for example, it misclassified emotions in Black individuals while working correctly for white individuals. Despite this, most participants did not detect the bias. Only Black participants were more sensitive to it, especially when they saw sad images of their own group.
In the second experiment, the data lacked sufficient representation of racial groups. For example, all the happy and sad faces were white, which taught the AI to ignore other races. Once again, people overlooked the bias until they saw the system perform incorrectly. The third experiment compared various scenarios, including variants in which the sad faces were white and the happy faces were Black, or in which there was no racial bias. The results confirmed that Black participants identified the problem more often when it involved negative emotions associated with their group.
The study’s authors, including S. Shyam Sundar of Penn State, emphasized that people often trust AI as a neutral tool even when it is not. Sundar noted that participants failed to see how race was intertwined with emotions, even though it was apparent. Cheng "Chris" Chen of Oregon State University, who led the study, explained that bias in the data leads to errors in which the system favors the dominant group—in this case, white individuals.
Other studies confirm the problem with detecting bias
Other research has reached similar conclusions. For example, a study in Scientific Reports showed that people can "inherit" bias from AI. In experiments involving medical diagnoses, participants who worked with biased AI continued to repeat its errors even after the AI was removed. This means that exposure to bias changes human decision-making over the long term.
Research from the University of South Carolina identified four types of cognitive bias in interactions with AI: priming (prior influence), anchoring (fixating on the first piece of information), framing, and availability. These mechanisms explain why people lose objectivity when evaluating AI outputs.
The National Institute of Standards and Technology (NIST) warns that bias in AI is not only about data but also about the social context. Reva Schwartz of NIST said that AI systems influence decisions that affect people’s lives and that bias manifests throughout the entire process—from data collection to deployment. Common types include selection bias, confirmation bias, measurement bias, stereotyping bias, and out-group homogeneity bias.
Bias frequently appears in real-world applications. For example, résumé screening tools show preferences based on race or gender, as found in research from the University of Washington. AI text detectors, meanwhile, discriminate against non-native English-speaking authors, according to a Stanford study. Further research from USC revealed that as many as 38.6% of the "facts" used by AI contain bias.
Source: psu.edu
