Why you shouldn’t trust data collected on MTurk

Authors: CS Kay

Published: 2025

Publication: Behavior Research Methods, 2025•Springer

Data collected on Amazon's Mechanical Turk (MTurk) shows substantial quality issues, with semantic antonym pairs being positively correlated instead of negatively, even after implementing data screening and using high-reputation participants.

Methods: 27 semantic antonym pairs were administered to participants from Connect (N=100), Prolific (N=100), and MTurk (N=400, N=600) to examine response quality and correlation patterns.

Key Findings: The correlation of responses to semantic antonym pairs as an indicator of data quality across different survey platforms.

Limitations: Does not provide detailed exploration of why MTurk data quality issues occur; limited to semantic antonym pair tests, which may not generalize to all survey formats.

Institution: Stanford University

Research Area: Behavior Research Methods

Discipline: Behavorial Science, Behavorial Research

Sample Size: 1200 participants

Citations: 1