Bias

Bias is complicated and nuanced. Research has identified a wide range of sociotechnical harms from bias across LLMs. We separate the issues associated with bias from other LLM issues due to the complex and charged nature of the problem.

• Sept 2024: Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination The study finds that models default to “standard” varieties of English; it also finds that model responses to non-“standard” varieties consistently exhibit a range of issues: stereotyping (19% worse than for “standard” varieties), demeaning content (25% worse), lack of comprehension (9% worse), and condescending responses (15% worse).

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

• Examining Gender and Race Bias in Sentiment Analysis Systems

• BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation

• REDDITBIAS: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models

• GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction

• A Survey on Bias and Fairness in Machine Learning

• The Political Preferences of LLMs

• Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Models

• The Woman Worked as a Babysitter: On Biases in Language Generation

• Bias and Fairness in Large Language Models: A Survey