Beyond Face Value: Visual and Auditory Signals in Human and Machine Trust Judgments
Authors: N Tyulina, Y Yu, TA Emmanouil, SI Levitan
Published: 2025
Publication: Proceedings of the 7th ACM ..., 2025 - dl.acm.org
Trust judgments are primarily influenced by auditory cues in both humans and multimodal models, though subtle differences in modality weighting exist between them.
Methods: Behavioral experiment with trust ratings of bimodal stimuli across four trust congruence conditions, combined with a multimodal model trained using HuBERT and ResNet-50 with late fusion, analyzed using Permutation Feature Importance (PFI).
Key Findings: The construction of trust from visual and auditory signals in both humans and multimodal models, focusing on modality dominance and feature weighting.
Limitations: Subtle differences between human and model feature interpretations were noted, but further research is needed to generalize findings across broader datasets and real-world scenarios.
Institution: University of Cambridge, University of Bath, University of Edinburgh, New York University
Research Area: Human-AI Interaction, Trust and Perception, Nonverbal Communication
Discipline: Applied Linguistics
Sample Size: 150 participants