Interactive Groupwise Comparison for Faster Reinforcement Learning from Human Feedback

Abstract

Reinforcement Learning from Human Feedback (RLHF) has gained significant attention due to its ability to align AI models with human preferences without designing complicated reward functions manually. However, the traditional RLHF approach via pairwise comparisons is labor-intensive and costly. As tasks become more complex, the amount of required feedback increases, making it difficult to scale. This work presents an interactive groupwise comparison approach for RLHF that exploits human expertise in comprising groups of similar behaviors. To support this concept, the user interface of our approach visualizes contextual pieces of information in a single exploration view, including hierarchical clustering of behaviors and human preferences, and using a hierarchical radial chart with edge bundling to avoid visual clutter. We built a visualization interface comprising two interactively linked views: 1) an exploration view showing a contextual overview of all sampled behaviors in a hierarchical clustering structure, and 2) a comparison view displaying two selected groups of behaviors for user queries. Users can efficiently explore large sets of behaviors by iterating between these two views. We evaluated the effectiveness and efficiency of the proposed approach compared with the traditional approach of pairwise comparisons using a simulated user model and a controlled user study with participants. Using groupwise comparisons, one can increase the number of elicited preferences within the same amount of human time by 71.2% with a lower error rate and obtain a better policy.