Most users cannot identify AI bias, even in training data

Summary

Images used in the studies were of Black and white individuals. The first experiment showed participants biased representation of race in certain classification categories, such as happy or sad images that were unevenly distributed across racial groups. Happy faces were mostly white. Sad faces were mostly Black. The second showed bias pertaining to the lack of adequate representation of certain racial groups in the training data. For example, participants would see only white subject images in both happy and sad categories. In the third experiment, the researchers presented the stimuli from the first two experiments alongside their counterexamples, resulting in five conditions: happy Black/sad white; happy white/sad Black; all white; all Black; and no racial confound, meaning there was no potential mixing of emotion and race. For each experiment, the researchers asked participants if they perceived the AI system treated every racial group equally. The researchers found that over the three scenarios, most participants indicated that they did not notice any bias. In the final experiment, black participants were more likely to identify the racial bias, compared to their white counterparts and often only when it involved unhappy images of Black people. “We were surprised that people failed to recognize that race and emotion were confounded, that one race was more likely than others to represent a given emotion in the training data—even when it was staring them in the face,” Sundar said. “For me, that's the most important discovery of the study.” Sundar added that the research was more about human psychology than technology. He said people often “trust AI to be neutral, even when it isn’t.” Chen said people’s inability to detect the racial confound in the training data leads to reliance on AI performance for evaluation. “Bias in performance is very, very persuasive,” Chen said. “When people see racially biased performance by an AI system, they ignore the training data ch...