Positional preferences, order effects, prompt sensitivity undermine AI judgments

Summary

Beyond their everyday chat capabilities, Large Language Models are increasingly being used to make decisions in sensitive domains like hiring, health, law, and civic engagement. The exact mechanics of how we use these models in such scenarios is vital. There are many ways to have LLMs make decisions, including A/B decision-making, ranking, classification, "panels" of judges, etc. but every single method is individually fragile and subject to measurement biases that are rarely discussed.Engineers composing prompts often rely on anecdotes and untested folklore. We call it 'prompt-engineering': the practice of composing prompts to coax precisely the outputs we desire. However, it might be described better as 'playing' than 'engineering'. There are popular templates and tropes, but few are well proven. You'll often see high level instructions like "you are an [adjective] [role]", e.g. "you are an impartial judge". Throw in a superlative here and there, maybe some ALL-CAPS and a few examples, run it a couple of times, observe it working, and then stamp 'shipped'. In doing all of this, there is an implicit buy-in into a premise that LLMs are sufficiently predictable when asked 'just right' for a limited set of outputs, a premise that ongoing research continues to challenge [3].But even if you give an LLM just two options of equal merit and ask it for the best, it will tilt one way or the other. Or give it an essay to judge according to a few criteria, and you will see its score subtly shift by how we pose the question. Filtering and classification tasks too, like detecting toxicity or cyber-bullying, are highly fragile. Ranking CVs from job applicants is a famously hard problem fraught with social biases. We know this, but rarely do we consider the very ‘posing of the problem’ to an AI to be a massive lever of bias as well.Such unpredictability is rare in computation, but it's very similar to human cognition. LLMs consistently exhibit vulnerabilities and cognitive biases ...