Evaluating LLM-contaminated Crowdsourcing Data Without Ground Truth

Authors: Y Zhang, J Pang, Z Zhu, Y Liu

Published: 2025

Publication: arXiv preprint arXiv:2506.06991, 2025 - arxiv.org

The paper proposes a training-free scoring mechanism using peer prediction to detect and mitigate LLM-assisted cheating in crowdsourced annotation tasks, with theoretical guarantees and empirical validation.

Methods: A peer prediction-based mechanism quantifies correlations between worker answers while conditioning on LLM-generated labels, without requiring ground truth or high-dimensional training data.

Key Findings: Detection of LLM-assisted low-effort cheating in crowdsourced annotation tasks, focusing on theoretical effectiveness and empirical robustness.

Limitations: The approach assumes a crowdsourcing model accounting for LLM collusion and may require specific conditions for optimal performance.

Institution: Rutgers University, University of California Santa Cruz

Research Area: Artificial Intelligence, Computational Social Science

Discipline: Computational Social Science

Citations: 1

DOI: https://doi.org/10.48550/arXiv.2506.06991