Credtwi: Investigating Social Media Credibility with a Browser Plugin
Abstract
People now look for information online and on social media for everyday problems. Organizations and malevolent actors have taken the opportunity to spread misinformation/disinformation. It is increasingly important to understand the credibility of online information. We designed and implemented a research browser plugin, Credtwi. It injects credibility questionnaires directly into the user's Twitter feed, enabling crowdsourced data collection. We carried out a week-long field study where participants assessed the credibility of tweets on various topics. We provide insights into information credibility in the Twitter ecosystem by analyzing the assessments and study questionnaires. The participants' perception of Twitter as a credible information source decreased after using Credtwi. Our results suggest that the author's verification status and bio are the most important factors for their perceived credibility. Finally, we discovered significant differences between the assessments of the different genders. Our results contribute to the research on online social media content credibility.
Study specs
A browser plugin was used for crowdsourced credibility assessment through participant questionnaires during a week-long field study.
- Institution
- University of Oulu
- Discipline
- Human-Computer Interaction
- Sample Size
- N=150
- Study Type
- Experimental Study
- Year
- 2025
- Human Data Platform
- Prolific
- Source
- View Source DOI Google Scholar
Measured Outcomes
Perceptions of online tweet credibility, factors affecting tweet credibility (e.g., verification status, bio), variations in credibility assessments across genders.
Peer Review & Critical Discussion
Potential Selection Bias in 2023 Cohort
The participant pool shows a concerning overrepresentation of users from high-income demographics. Looking at Table 3, we can see that 78% of respondents had annual incomes above $75k, which significantly limits the generalizability of these findings to broader populations.
Non-naive Participants Issue
I've noticed a methodological concern regarding participant naivety. Given that Prolific users often complete multiple studies, there's a real risk that participants had prior exposure to similar experimental paradigms, which could confound the results.
RLHF Applicability to This Study Design
The implications for RLHF training pipelines are understated. If we accept the authors' conclusions about preference stability, this has direct consequences for how we should structure reward model training. The temporal decay effect described in Section 4.2 is particularly relevant.
Verify your expertise to join discussion
Create an account and verify your credentials to participate in peer discussions.