Survey curious? start-up guide and best practices for running surveys and experiments online
Abstract
Many researchers in finance and economics are interested in running online surveys or experiment, but do not know the particulars of how to implement them. Our goal is to provide the basic information to create a useful survey and post it online so you can collect data as quickly and easily as possible. We provide overviews and click-by-click instructions for popular survey platforms and participant recruitment platforms. Further, we highlight portions of the survey design literature that are likely to be most relevant to economists and outline a set of best practices to follow when creating your survey.
Study specs
- Authors
- A Bergman,A Chinco,SM Hartzmark
- Institution
- Yale School of Management,University of Miami School of Business,New York University,Columbia University
- Discipline
- Social Science Research Methods
- Year
- 2020
- Human Data Platform
- Prolific
- Source
- View Source Google Scholar
Peer Review & Critical Discussion
Potential Selection Bias in 2023 Cohort
The participant pool shows a concerning overrepresentation of users from high-income demographics. Looking at Table 3, we can see that 78% of respondents had annual incomes above $75k, which significantly limits the generalizability of these findings to broader populations.
Non-naive Participants Issue
I've noticed a methodological concern regarding participant naivety. Given that Prolific users often complete multiple studies, there's a real risk that participants had prior exposure to similar experimental paradigms, which could confound the results.
RLHF Applicability to This Study Design
The implications for RLHF training pipelines are understated. If we accept the authors' conclusions about preference stability, this has direct consequences for how we should structure reward model training. The temporal decay effect described in Section 4.2 is particularly relevant.
Verify your expertise to join discussion
Create an account and verify your credentials to participate in peer discussions.