Crowdsourcing: a modern tool for robust research sampling
Abstract
Crowdsourcing is the practice of engaging a group of people or “crowd” to work towards a common goal (Howe, 2006; Landers et al., 2019). It leverages the collective intelligence of a large group, typically via the internet, to complete tasks more quickly, efficiently, and innovatively than individuals or smaller teams. Crowdsourcing has been effectively applied to a broad range of real-world issues. A well-known example is Wikipedia, where volunteers globally write, edit, and update encyclopedia entries (Kaplan & Haenlein, 2014; Kittur et al., 2007). Individuals and organizations also use crowdsourcing to fund projects on platforms like GoFundMe or Kickstarter (Allon & Babich, 2020; Mollick, 2014). Even blockchain networks such as Bitcoin and Ethereum are maintained by a decentralized crowd of miners or validators. For example, blockchain validation and consensus mechanisms function through crowdsourcing, where the crowd processes transactions and adds them to the blockchain through mechanisms like proof of work (eg, Bitcoin) or proof of stake (eg, Ethereum). In essence, decentralized finance and blockchain technologies rely on a network of individual contributors to run and validate transactions (Kapengut & Mizrach, 2023; Nakamoto, 2008). The premise of crowdsourcing has played an essential role in Industrial-Organizational (IO) Psychology, both in practice and research. The concept underlying crowdsourcing was common in IO practice before Howe (2006) introduced the term. A classic IO example is gathering job analysis data involving information about tasks, responsibilities, and skills required for a job. By collecting data from various sources (eg, interviews, questionnaires from employees and supervisors, performance data, organizational documents, expert panels, and focus groups), practitioners can develop a comprehensive
Study specs
The paper discusses the theoretical and practical applications of crowdsourcing in various domains, referencing prior work and examples such as Wikipedia, crowdfunding platforms, and blockchain networks.
- Institution
- Auburn University
- Discipline
- Social Science
- Study Type
- Literature Review
- Year
- 2025
- Human Data Platform
- Prolific
- Source
- View Source Google Scholar
Measured Outcomes
The applications and impact of crowdsourcing in different fields, particularly its role in Industrial-Organizational Psychology for data collection and analysis.
Peer Review & Critical Discussion
Potential Selection Bias in 2023 Cohort
The participant pool shows a concerning overrepresentation of users from high-income demographics. Looking at Table 3, we can see that 78% of respondents had annual incomes above $75k, which significantly limits the generalizability of these findings to broader populations.
Non-naive Participants Issue
I've noticed a methodological concern regarding participant naivety. Given that Prolific users often complete multiple studies, there's a real risk that participants had prior exposure to similar experimental paradigms, which could confound the results.
RLHF Applicability to This Study Design
The implications for RLHF training pipelines are understated. If we accept the authors' conclusions about preference stability, this has direct consequences for how we should structure reward model training. The temporal decay effect described in Section 4.2 is particularly relevant.
Verify your expertise to join discussion
Create an account and verify your credentials to participate in peer discussions.