Underreporting of AI use: The role of social desirability bias
Abstract
Rapid integration of artificial intelligence (AI) into work and educational settings challenges organizations to gauge and respond to adoption rates. However, most measures of AI adoption come from self-reported surveys, producing estimates of AI use that disagree by up to 40 percentage points within the same setting. We investigate whether social desirability bias-the tendency to answer surveys in ways that would be viewed favorably by an outside party-can explain this discrepancy. Surveying 338 university students, we assess potential social desirability bias using a method from psychology, indirect questioning: students report both their own AI use and that of their peers. We find a significant gap, with approximately 60% of students reporting that they use AI compared to 90% of their peers. Through qualitative analysis of student explanations for this gap, we conclude that social desirability bias is a key driver of mis-measurement, causing underestimates of AI adoption in educational settings.
Study specs
- Institution
- University of Chicago
- Discipline
- Behavioral Economics
- Year
- 2024
- Human Data Platform
- Prolific
- Source
- View Source Google Scholar
Peer Review & Critical Discussion
Potential Selection Bias in 2023 Cohort
The participant pool shows a concerning overrepresentation of users from high-income demographics. Looking at Table 3, we can see that 78% of respondents had annual incomes above $75k, which significantly limits the generalizability of these findings to broader populations.
Non-naive Participants Issue
I've noticed a methodological concern regarding participant naivety. Given that Prolific users often complete multiple studies, there's a real risk that participants had prior exposure to similar experimental paradigms, which could confound the results.
RLHF Applicability to This Study Design
The implications for RLHF training pipelines are understated. If we accept the authors' conclusions about preference stability, this has direct consequences for how we should structure reward model training. The temporal decay effect described in Section 4.2 is particularly relevant.
Verify your expertise to join discussion
Create an account and verify your credentials to participate in peer discussions.