Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework

Abstract

The evaluation of large language models faces significant challenges. Technical benchmarks often lack real-world relevance, while existing human preference evaluations suffer from unrepresentative sampling, superficial assessmentdepth, and single-metric reductionism. To address these issues, we introduce HUMAINE, a framework for multidimensional, demographically aware measurementof human-AI interaction. We collected multi-turn, naturalistic conversations from23,404 participants that were stratified across 22 demographic groups, both in theUS and UK, to evaluate 28 state-of-the-art models across five human-centric dimensions. We use a hierarchical Bayesian Bradley-Terry-Davidson (BTD) model,with post-stratification to census data, and our analysis reveals three key insights.(1) We establish a clear performance hierarchy where google/gemini-2.5-proranks first overall, with a 95.6% posterior probability of being the top-ranked model. (2) We uncover significant preference heterogeneity, with user age emerging as the primary demographic axis of disagreement; a model’s perceived rankcan shift substantially across age groups, exposing failures in generalisation thatunrepresentative samples typically mask. (3) We quantify the vast difference indiscriminative power across evaluation dimensions, with ambiguous qualities likeTrust, Ethics & Safety showing a 65% tie rate, in stark contrast to the decisive10% tie rate for Overall Winner. Our work emphasises the need for a more multidimensional, demographically aware perspective in LLM evaluation. We releaseour complete dataset, interactive leaderboard, and open-source framework.