Online experimentation and sampling in cognitive aging research.
Abstract
Online data collection methods have become increasingly popular in many domains of psychology, but their use in cognitive aging studies remains relatively limited. Is it time for cognitive aging researchers to embrace these methods? Here, we weigh potential advantages and disadvantages of conducting online studies with young and older adults, relative to lab-based studies, with a particular focus on the study of human memory and aging. With online studies, it may be possible to assess whether age-related effects on cognition obtained in the laboratory generalize to other situations with different environmental or subject characteristics. However, there are many open questions about the representativeness of older adults on online data collection platforms, and issues surrounding data quality, selection effects, and other biasing characteristics, which must be carefully handled in cognitive aging studies which recruit young and older adult participants online. We consider the benefit of conducting experimentation both in the lab and online in providing converging evidence on a research question, and we offer an example of an experiment on adult age differences in associative recognition that was conducted in the laboratory and online. We also provide practical recommendations for ways to maximize the potential for online studies to contribute to our understanding of cognitive aging. (PsycInfo Database Record (c) 2025 APA, all rights reserved)
Study specs
- Authors
- NR Greene,M Naveh-Benjamin
- Discipline
- Psychology,Cognitive Aging Research
- Year
- 2022
- Human Data Platform
- Prolific
- Source
- View Source Google Scholar
Peer Review & Critical Discussion
Potential Selection Bias in 2023 Cohort
The participant pool shows a concerning overrepresentation of users from high-income demographics. Looking at Table 3, we can see that 78% of respondents had annual incomes above $75k, which significantly limits the generalizability of these findings to broader populations.
Non-naive Participants Issue
I've noticed a methodological concern regarding participant naivety. Given that Prolific users often complete multiple studies, there's a real risk that participants had prior exposure to similar experimental paradigms, which could confound the results.
RLHF Applicability to This Study Design
The implications for RLHF training pipelines are understated. If we accept the authors' conclusions about preference stability, this has direct consequences for how we should structure reward model training. The temporal decay effect described in Section 4.2 is particularly relevant.
Verify your expertise to join discussion
Create an account and verify your credentials to participate in peer discussions.