Annotation: Research Area — Prolific Citations Library

Discover 8 peer-reviewed studies in Annotation (2021–2025). Explore research findings powered by Prolific's diverse participant panel.

This page lists 8 peer-reviewed papers in the research area of Annotation in the Prolific Citations Library, a curated collection of research powered by high-quality human data from Prolific.

Papers (8 of 8)

Transforming data annotation with ai agents: A review of architectures, reasoning, applications, and impact

Authors: MM Karim, S Khan, DH Van, X Liu, C Wang, Q Qu

Year: 2025

Published in: Future Internet, 2025 - mdpi.com

Institution: Chinese Academy of Sciences, Zhejiang University, South-Central Minzu University

Research Area: Artificial Intelligence, Data Annotation, Multi-Agent Systems

Discipline: Artificial Intelligence

The paper reviews the role of AI agents powered by large language models in addressing challenges in data annotation, focusing on architectures, workflows, real-world applications, and future research directions for improving efficiency, scalability, transparency, and bias mitigation.

Methods: Comprehensive review and analysis of AI agent architectures, workflows, applications, and evaluation methods in data annotation across multiple industries.

Key Findings: Capabilities of LLM-driven agents in reasoning, adaptive learning, collaborative annotation, and their impact on quality assurance, cost, scalability, and bias mitigation.

Citations: 10
Improving annotation quality: empirical insights into bias, human-AI collaboration, and workflow design

Authors: J Beck

Year: 2025

Published in: 2025 - edoc.ub.uni-muenchen.de

Institution: Ludwig-Maximilians-Universität München, University of Bayreuth

Research Area: Annotation Quality, Human-AI Collaboration, Behavioral Science, Human-Computer Interaction (HCI)

Discipline: Human-Computer Interaction (HCI)

The study empirically evaluates annotation bias, proposes strategies to reduce its impact, and explores the use of large language models in automated and hybrid annotation workflows.

Methods: Empirical assessments and experimental evaluations involving annotation workflows and large language models.

Key Findings: Annotation bias, annotation quality, and the effectiveness of hybrid workflows integrating human input and AI models.
The origin and value of disagreement among data labelers: A case study of individual differences in hate speech annotation

Authors: Y Sang, J Stanton

Year: 2024

Published in: International Conference on Information, 2022 - Springer

Institution: Syracuse University

Research Area: Hate Speech Annotation, Individual Differences in Data Labeling

Discipline: Computational Social Science

This study explores disagreements among hate speech annotators and proposes a multidimensional scale to analyze individual differences, which could improve the value of minority-vote labels.

Methods: Mixed-method approach including expert interviews, concept mapping exercises, self-reporting questionnaires, and the development/testing of a multidimensional scale.

Key Findings: Individual differences (e.g., age, personality) and their relationship to annotators' label decisions in hate speech tasks.

DOI: https://doi.org/10.1007/978-3-030-96957-8_36

Citations: 46

Sample Size: 170
Augmented behavioral annotation tools, with application to multimodal datasets and models: A systematic review

Authors: E Watson, T Viana, S Zhang

Year: 2024

Published in: AI, 2023 - mdpi.com

Research Area: Behavioral Annotation Tools and Multimodal Data

Discipline: Computer Science

The paper systematically reviews augmented behavioral annotation tools, focusing on their evolution, current state, and application to multimodal datasets and models, highlighting best practices and emerging challenges in safe and ethical annotation for large-scale multimodal systems.

Methods: Systematic literature review analyzing crowd and machine learning-augmented behavioral annotation methods, with cross-disciplinary comparisons and structured synthesis of practices.

Key Findings: Evolution of behavioral annotation tools, their integration with machine learning, emerging trends (e.g., prompt engineering), challenges in large multimodal datasets, and ethical and engineering best practices.

DOI: https://doi.org/10.3390/ai4010007

Citations: 17
Annotator in the Loop: A Case Study of In-Depth Rater Engagement to Create a Prosocial Benchmark Dataset

Authors: S Schmer-Galunder, R Wheelock, Z Jalan

Year: 2024

Published in: Proceedings of the ..., 2024 - ojs.aaai.org

Institution: Google DeepMind, Google, Accenture, Amazon

Research Area: AI Ethics and Prosocial Data Annotation

Discipline: Artificial Intelligence, Ethics, Behavioral Science

DOI: https://doi.org/10.1609/aies.v7i1.31726

Citations: 3
Order Effects in Annotation Tasks: Further Evidence of Annotation Sensitivity

Authors: Jacob Beck, Stephanie Eckman, Bolei Ma, Rob Chew, Frauke Kreuter

Year: 2024

Published in: ACL Anthology

Institution: University of Maryland

Research Area: Annotation Sensitivity, Order Effects, Natural Language Processing, Social Science in AI

Discipline: Natural Language Processing (NLP), Computational Social Science
Temporal and second language influence on intra-annotator agreement and stability in hate speech labelling

Authors: G Abercrombie, D Hovy

Year: 2023

Published in: 17th Linguistic ..., 2023 - researchportal.hw.ac.uk

Institution: Heriot Watt University

Research Area: Hate Speech Annotation, Computational Linguistics, Natural language processing (NLP), Annotation

Discipline: Computational Linguistics

DOI: https://doi.org/10.18653/v1/2023.law-1.10

Citations: 23
Exploring cross-cultural differences in English hate speech annotations: From dataset construction to analysis

Authors: N Lee, C Jung, J Myung, J Jin

Year: 2021

Published in: Proceedings of the ..., 2024 - aclanthology.org

Institution: KAIST, Cardiff University

Research Area: Hate Speech Annotation, Cross-Cultural Bias, NLP Ethics

Discipline: Natural Language Processing (NLP), Computational Social Science

Citations: 44