Generative AI meets open-ended survey responses: Research participant use of ai and homogenization
Authors: S Zhang, J Xu, AJ Alvero
Published: 2025
Publication: Sociological Methods & Research, 2025 - journals.sagepub.com
The study finds that 34% of research participants use generative AI tools like large language models (LLMs) to assist with open-ended survey responses, leading to more homogeneity and positivity in their answers, which could impact data validity by masking social variations.
Methods: The study conducted an original survey on a popular online platform and simulated comparisons between human-written responses from pre-ChatGPT studies and LLM-generated responses.
Key Findings: Use of LLMs by survey participants, differences in text homogeneity, positivity, and masking of social variation in open-ended survey responses.
Limitations: The study does not specify the full scope of how widespread these behaviors may be across different survey platforms or the extent of LLM impacts beyond the selected samples.
Institution: University of Maryland, Indiana University, University of Minnesota Duluth
Research Area: Sociological Methods, Generative AI, Survey Methodology
Discipline: Sociology,Social Science
Citations: 26