Beyond Face Value: Visual and Auditory Signals in Human and Machine Trust Judgments
Abstract
As conversational agents become increasingly multimodal, they invite human-like evaluations—especially in trust-sensitive contexts. Building on the human tendency to form rapid judgments from subtle visual and auditory cues, we explore how trust is constructed from faces and voices. In a behavioral experiment, 150 participants rated bimodal stimuli across four trust congruence conditions. We then trained a multimodal model using HuBERT and ResNet-50 with late fusion to predict trust scores. To examine alignment between human and model judgments, we applied Permutation Feature Importance (PFI) to compare the most influential features. Our results highlight the dominance of auditory cues in both human and model trust evaluations, while revealing subtle but meaningful differences in feature weighting across modalities and conditions.
Study specs
Behavioral experiment with trust ratings of bimodal stimuli across four trust congruence conditions, combined with a multimodal model trained using HuBERT and ResNet-50 with late fusion, analyzed using Permutation Feature Importance (PFI).
- Authors
- N Tyulina,Y Yu,TA Emmanouil,SI Levitan
- Discipline
- Applied Linguistics
- Sample Size
- N=150
- Study Type
- Experimental Study
- Year
- 2025
- Human Data Platform
- Prolific
- Source
- View Source Google Scholar
Measured Outcomes
The construction of trust from visual and auditory signals in both humans and multimodal models, focusing on modality dominance and feature weighting.
Peer Review & Critical Discussion
Potential Selection Bias in 2023 Cohort
The participant pool shows a concerning overrepresentation of users from high-income demographics. Looking at Table 3, we can see that 78% of respondents had annual incomes above $75k, which significantly limits the generalizability of these findings to broader populations.
Non-naive Participants Issue
I've noticed a methodological concern regarding participant naivety. Given that Prolific users often complete multiple studies, there's a real risk that participants had prior exposure to similar experimental paradigms, which could confound the results.
RLHF Applicability to This Study Design
The implications for RLHF training pipelines are understated. If we accept the authors' conclusions about preference stability, this has direct consequences for how we should structure reward model training. The temporal decay effect described in Section 4.2 is particularly relevant.
Verify your expertise to join discussion
Create an account and verify your credentials to participate in peer discussions.