Take caution in using LLMs as human surrogates: Scylla ex machina
Authors: Y Gao, D Lee, G Burtch, S Fazelpour
Published: 2024
Publication: arXiv preprint arXiv:2410.19599, 2024 - arxiv.org
LLMs fail to accurately replicate human behavior in the 11-20 money request game, cautioning against their use as surrogates for human cognition in social science research.
Methods: The study evaluates the reasoning depth of various advanced LLMs through their performance on the 11-20 money request game, analyzing failure points related to input language, roles, and safeguarding.
Key Findings: The ability of LLMs to replicate human-like behavior and reasoning distribution in the context of social science simulations.
Limitations: LLMs rely on probabilistic patterns and lack the embodied experiences or survival objectives integral to human cognition; failures are diverse and unpredictable across models.
Institution: Boston University, Northeastern University
Research Area: LLMs as Human Surrogates, Social Science Research Methods, Human Behavior Simulation
Discipline: Economics, Artificial Intelligence, Social Science
Citations: 25