Browse 29 peer-reviewed papers in Data Quality. Discover studies powered by high-quality human data from Prolific.
This page lists 29 peer-reviewed papers tagged with Data Quality in the Prolific Citations Library, a curated collection of research powered by high-quality human data from Prolific.
-
Authors: S Zhang, J Xu, AJ Alvero
Year: 2025
Published in: Sociological Methods & Research, 2025 - journals.sagepub.com
Institution: University of Maryland, Indiana University, University of Minnesota Duluth
Research Area: Sociological Methods, Generative AI, Survey Methodology
Discipline: Sociology, Social Science
The study finds that 34% of research participants use generative AI tools like large language models (LLMs) to assist with open-ended survey responses, leading to more homogeneity and positivity in their answers, which could impact data validity by masking social variations.
Methods: The study conducted an original survey on a popular online platform and simulated comparisons between human-written responses from pre-ChatGPT studies and LLM-generated responses.
Key Findings: Use of LLMs by survey participants, differences in text homogeneity, positivity, and masking of social variation in open-ended survey responses.
Citations: 26
-
Authors: A Söderström, A Shatte
Year: 2025
Published in: Behavior Research ..., 2021 - Springer
Institution: University of Helsinki
Research Area: Intelligent Agents, Health Research Methodology, Behavioral Research Methods
Discipline: Research Methodology, Behavioral Science
The study found that chatbot-assisted surveys modestly improve data quality, with most users finding the chatbots helpful and widely using them.
Methods: Randomized participants into chatbot-supported and unassisted survey conditions; assessed chatbot use, user satisfaction, and data quality via validated and deliberately confusing challenge items.
Key Findings: Effects of chatbot assistance on data quality, user satisfaction, and usage patterns in online questionnaires.
DOI: https://doi.org/10.3758/s13428-021-01574-w
Citations: 7
Sample Size: 300
-
Authors: N Byrd
Year: 2025
Published in: Byrd, N. (2025). Reflection-Philosophy Order Effects and Correlations Across Samples. Analysis. DOI: 10.1093/analys/anaf015. https://osf.io/preprints/psyarxiv/y8sdm
Institution: Stevens Institute of Technology
Research Area: Behavioral Research Methods, Experimental Psychology, Crowdsourcing Platforms
Discipline: Psychology
Reflective reasoning correlates with certain philosophical decisions, and the study suggests bidirectional causal paths between reflection and philosophy, with test order effects influencing reflection test outcomes but not philosophical decisions.
Methods: Participants from four sources (Amazon Mechanical Turk, CloudResearch, Prolific, and a university) were tested on reflective reasoning and their decisions on 10 philosophical thought experiments.
Key Findings: Impact of reflective reasoning on philosophical decisions and the effect of test order on reflection and philosophy outcomes.
Citations: 4
-
Authors: B Aksoy, S Nevo
Year: 2025
Published in: Participant Behavior and Motivations (March 21 ..., 2025 - papers.ssrn.com
Institution: Rensselaer Polytechnic Institute
Research Area: Crowdsourcing, Participant Behavior
Discipline: Computational Social Science
Research on Prolific reveals that participant compensation significantly impacts sample selection, potentially introducing biases, and offers insights into participant motivations and behavior to improve study reliability and design.
Methods: A carefully designed experiment was performed to analyze correlations between participants' reservation wages, socioeconomic attributes, and study compensations; sensitivity analyses were conducted for further guidance.
Key Findings: Participant reservation wages, socioeconomic attributes, perceptions of general behavior and motivations, and implications of study design decisions.
Citations: 3
-
Authors: Y Ba, MV Mancenido, EK Chiou, R Pan
Year: 2025
Published in: Behavior Research Methods, 2025 - Springer
Institution: University of Delaware, National Taiwan University, University of British Columbia, Monash University
Research Area: Crowdsourcing, Data Quality, Spamming Behavior Detection, LLM Applications in Behavioral Research
Discipline: Computer Science, Artificial Intelligence, Large Language Models
The paper introduces a systematic method to evaluate crowdsourced data quality and detect spam behaviors through variance decomposition, proposing a spammer index and credibility metrics to improve consistency and reliability in labeling tasks.
Methods: Variance decomposition, Markov chain models, and generalized random effects models were used to assess annotator consistency and credibility; metrics were applied to both simulated and real-world data from two crowdsourcing platforms.
Key Findings: Quality of crowdsourced data, spammer behaviors, annotators’ consistency, and credibility.
Citations: 2
-
Authors: DT Esch, N Mylonopoulos, V Theoharakis
Year: 2025
Published in: Behavior Research Methods, 2025 - Springer
Institution: University of Cologne, University of Piraeus, Aristotle University of Thessaloniki
Research Area: Crowdsourcing Behavioral Research, Mobile Data Collection
Discipline: Behavioral Research Methods
Mobile-based responses via platforms like Pollfish are comparable in quality to computer-based ones from MTurk and Prolific, though attentiveness varies significantly across platforms and is influenced by factors like incentives, distractions, and system 1 thinking.
Methods: Conducted two studies distributing the same survey across MTurk, Prolific, Pollfish, and Qualtrics panels to compare data quality and analyze attentiveness scores.
Key Findings: Attentiveness, device usage (mobile vs. computer), and factors influencing data quality such as incentives, respondent activity, distractions, and survey familiarity.
Citations: 1
-
Authors: S Hatgis-Kessell, WB Knox, S Booth, S Niekum
Year: 2025
Published in: arXiv preprint arXiv ..., 2025 - arxiv.org
Institution: Stanford University, UMass Amherst, Carnegie Mellon University
Research Area: Reinforcement Learning from Human Feedback (RLHF)
Discipline: Artificial Intelligence, Human-Computer Interaction
The paper investigates whether human preferences can be influenced to align more closely with assumed preference models in RLHF algorithms through interventions such as showing model-derived quantities, training on preference models, and modifying elicitation questions.
Methods: Three human studies were conducted where interventions were tested, including revealing model-derived quantities, training participants on a preference model, and altering how preference questions were framed.
Key Findings: Evaluated the impact of interventions on humans' expression of preferences to align better with the assumed preference models of RLHF algorithms.
DOI: https://doi.org/10.48550/arXiv.2501.06416
Citations: 1
-
Authors: CS Kay
Year: 2025
Published in: Behavior Research Methods, 2025•Springer
Institution: Stanford University
Research Area: Behavioral Research Methods
Discipline: Behavioral Science, Behavioral Research Methods
Data collected on Amazon's Mechanical Turk (MTurk) shows substantial quality issues, with semantic antonym pairs being positively correlated instead of negatively, even after implementing data screening and using high-reputation participants.
Methods: 27 semantic antonym pairs were administered to participants from Connect (N=100), Prolific (N=100), and MTurk (N=400, N=600) to examine response quality and correlation patterns.
Key Findings: The correlation of responses to semantic antonym pairs as an indicator of data quality across different survey platforms.
Citations: 1
Sample Size: 1200
-
Authors: J Beck
Year: 2025
Published in: 2025 - edoc.ub.uni-muenchen.de
Institution: Ludwig-Maximilians-Universität München, University of Bayreuth
Research Area: Annotation Quality, Human-AI Collaboration, Behavioral Science, Human-Computer Interaction
Discipline: Human-Computer Interaction
The study empirically evaluates annotation bias, proposes strategies to reduce its impact, and explores the use of large language models in automated and hybrid annotation workflows.
Methods: Empirical assessments and experimental evaluations involving annotation workflows and large language models.
Key Findings: Annotation bias, annotation quality, and the effectiveness of hybrid workflows integrating human input and AI models.
-
Authors: B Lebrun, S Temtsin, A Vonasch
Year: 2024
Published in: Frontiers in Robotics and ..., 2024 - frontiersin.org
Institution: University of Lausanne, University of California Berkeley, University of Massachusetts Amherst, Arizona State University
Research Area: AI in Social Science Research, Survey Methodology, Data Quality
Discipline: Artificial Intelligence
The study examines the integrity of online questionnaire responses and concludes that humans can identify AI-generated text with 76% accuracy, but current AI detection systems are ineffective, raising concerns about data quality in online surveys.
Methods: Human participants and automatic AI detection systems were tested on their ability to differentiate AI-generated text from human-generated text in the context of online questionnaires.
Key Findings: The study measured the ability of humans and AI detection tools to correctly identify whether text was generated by a human or an AI system for online questionnaire responses.
DOI: https://doi.org/10.3389/frobt.2023.1277635
Citations: 26
-
Authors: S Du, MT Babalola, P D'cruz, E Dóci
Year: 2024
Published in: Journal of Business ..., 2024 - Springer
Institution: Nottingham University Business School, University of Reading, Oxford Brookes University, University of Portsmouth
Research Area: Crowdsourcing Ethics, Social Science, Organizational Behavior
Discipline: Social Science
The paper explores the ethical, societal, and global implications of using crowdsourcing platforms for research, emphasizing the need for fair compensation, transparency, and consideration of global disparities between the Global North and South.
Methods: The paper provides a conceptual analysis and critique of crowdsourcing research practices, focusing on ethical and societal considerations.
Key Findings: Ethical, societal, and global implications of crowdsourcing research practices, including data quality, reporting transparency, fair remuneration, and the role of global disparities.
Citations: 24
-
Authors: AC Krendl, K Hugenberg, DP Kennedy
Year: 2024
Published in: Behavior research methods, 2024 - Springer
Institution: Indiana University
Research Area: Psychological Research Methods, Data Quality in Online Experiments, Theory of Mind Assessment
Discipline: Research Methodology, Cognitive Psychology
This study found that online samples can reliably complete dynamic, complex theory of mind tasks, though familiarity with task content can influence performance.
Methods: Compared in-lab and online participants' performance on two dynamic theory of mind tasks, using one familiar and one relatively novel TV-based paradigm and counterbalancing task order.
Key Findings: Performance on theory of mind tasks, including inferring beliefs, understanding motivations, detecting deception, identifying faux pas, and understanding emotions.
Citations: 13
Sample Size: 668
-
Authors: Eyal Peer
Year: 2024
Published in: CAMBRIDGE
Institution: Hebrew University, University of Cambridge
Research Area: Crowdsourcing, Research Methodology in Behavioral and Social Sciences
Discipline: Social, Behavioral Science
Citations: 7
-
Authors: E Becks, V Matkovic, T Weis
Year: 2024
Published in: 2025 IEEE International Conference on ..., 2025 - computer.org
Institution: University of Stuttgart, University of Applied Sciences Offenburg, University of Hohenheim
Research Area: Crowdsourced Online Studies, Human-Computer Interaction (HCI) in AI Systems, Behavioral Research Methodology
Discipline: Human-Computer Interaction
-
Authors: BD Douglas, PJ Ewell, M Brauer
Year: 2023
Published in: Plos one, 2023 - journals.plos.org
Institution: University of Alabama, University of Wisconsin-Madison, Florida Atlantic University
Research Area: Social Science Research Methods, Behavioral Research Methods, Data Quality in Crowdsourcing
Discipline: Social Science Research Methods
Citations: 1598
-
Authors: K Uittenhove, S Jeanneret, E Vergauwe
Year: 2023
Published in: Journal of Cognition, 2023 - pmc.ncbi.nlm.nih.gov
Institution: University of Lausanne, University of Geneva, EPFL, University of Neuchâtel, NiH
Research Area: Cognitive Psychology, Research Methodology, Behavioral Research Methods, Web-based Behavioral Research
Discipline: Cognitive Research, Psychology
Citations: 83
-
Authors: E Peer, D Rothschild, A Gordon, E Damer
Year: 2022
Published in: Behavior Research Methods, 2022 - Springer
Institution: The Hebrew University of Jerusalem, Microsoft Research, Prolific
Research Area: Online Behavioral Research, Data Quality, Research Methodology
Discipline: Computational Social Science, Behavioral Research Methods
Citations: 2112
-
Authors: J Tang, E Birrell, A Lerner
Year: 2022
Published in: ... symposium on usable privacy and security ..., 2022 - usenix.org
Institution: University of California Berkeley, George Washington University, Stanford University
Research Area: Online privacy and security surveys, External validity, Replication studies, Human-Computer Interaction (HCI) in security
Discipline: Computer Science, Behavioral Science
Citations: 164
-
Authors: S Mathôt, J March
Year: 2022
Published in: Language Learning, 2022 - Wiley Online Library
Institution: Ghent University, KU Leuven
Research Area: Online experimental methodology, Psycholinguistics, Behavioral research tools
Discipline: Psycholinguistics
Citations: 76
-
Authors: NR Greene, M Naveh-Benjamin
Year: 2022
Published in: Psychology and Aging, 2022 - psycnet.apa.org
Institution: University of Kansas, University of Arkansas, University of California Riverside
Research Area: Cognitive Aging Research, Online Experimentation, Sampling Methods in Psychology
Discipline: Psychology, Cognitive Aging Research
Citations: 56
