Model Responses Alter According to User's Speech Style
In a groundbreaking study, researchers from Oxford University have discovered that two leading open-source language models, Llama3 and Qwen3, adjust their responses based on the user's presumed identity.
The research, titled "Language Models Change Facts Based on the Way You Talk", was conducted by three researchers at the university. The study found that these models exhibit different strengths of sensitivity in various applications, reflecting implicit sociolinguistic biases encoded in their training.
The findings indicate that both Llama3 and Qwen3 are highly sensitive to a user's ethnicity and gender in all applications. In the legal domain, Llama3 was more likely to give legally helpful answers to non-binary and female users than to male users, while Qwen3 provided less favorable answers to mixed ethnicity users and more favorable ones to black users compared to white users.
Similarly, in the salary recommendation application, both models recommended lower starting salaries to non-White and Mixed ethnicity users compared to White users. In the medical domain, Black users were given different answers nearly half the time, and were more likely than White users to be advised to seek care.
The researchers used a generalized linear mixed model to compare how the language models responded to users with different inferred identities across five applications. They found that Llama3 is more sensitive than Qwen3 to user identity in medical advice applications, while Qwen3 is much more sensitive than Llama3 to user identity in politicized factual information and government benefit eligibility applications.
Both models show particular sensitivity to user's age, religion, region of birth, and region of residence, with alterations in responses occurring for 50% or more of questions asked involving these identity categories in some applications.
The study also examined systematic bias, i.e., consistent patterns where the models vary properties of responses based on identity across all questions in an application. For example, whether the model consistently recommends one identity group seek medical assistance less than another throughout medical advice questions.
The authors note that these biases do not emerge from 'obvious' signals such as the user stating their race or gender overtly in conversations, but from subtle patterns in their writing, which are inferred and, apparently, exploited by the LLMs to condition the quality of response.
The researchers suggest that the tests conducted on these two models should be extended to a wider range of potential models, including API-only LLMs such as ChatGPT. They urge organizations deploying these models for specific applications to build on these tools and to develop their own sociolinguistic bias benchmarks before deployment to understand and mitigate the potential harms that users of different identities may experience.
[1] For more detailed information on the statistical analysis methods used, please refer to the original research paper.
In the realm of science and technology, the discovery of implicit sociolinguistic biases in leading open-source language models like Llama3 and Qwen3, as detailed in the paper titled "Language Models Change Facts Based on the Way You Talk", raises concerns about their application in health-and-wellness domains, such as medical advice and health information. The findings suggest that these models exhibit different biases based on a user's ethnicity, gender, age, religion, region of birth, and region of residence, potentially causing disparities in the quality of responses for users of different identities. This highlights the importance of artificial-intelligence developers conducting tests and establishing sociolinguistic bias benchmarks to ensure fairness and avoid exacerbating existing disparities.
