Transforming human interactions with AI via reinforcement learning with human feedback (RLHF)

Abstract

This report considers a simple yet important question: can RLHF be developed to transform human experiences with AI without negatively affecting human societies? Analysis of this question is timely and necessary, especially given that research of reward learning methods like RLHF is currently lagging compared to other areas of AI safety. Our objectives are threefold: to provide a systematic study of the social effects of RLHF; to identify key social and ethical issues of RLHF; and to discuss social impacts for stakeholders. While limited by space, we believe it is crucial when evaluating social implications of RLHF to consider the diverse range of areas to which it may be deployed. Guided by the following questions, this report describes the primary ways in which RLHF can influence human society: • How might RLHF affect the integrity of information to which people have access? • How might RLHF reflect values and preferences of target populations? • How might RLHF temper or intensify different axes of social inequality? • How might RLHF alter access different social groups have to AI technologies? • How might RLHF impact cultural and international relations? • How might RLHF enhance industries and transform workforces? We ultimately conclude that RLHF has positive potential to: • Assist in mitigating harmful content generation and improve information integrity. • Serve as an important building block in aligning AI systems with human values. • Reduce bias at multiple levels in the AI production pipeline. • Open the door to democratization of AI technologies to all levels of society. • Transform how we reconcile cross-cultural perspectives and approach peaceful dialogue. • Facilitate development of more adaptable AI systems for use in various industries. • Automate tedious or high-risk portions of manual labor and affect the spatial distribution of jobs. RLHF’s transformative power suggests we will see more resources invested in its development. As RLHF raises concerns that echo those of existing AI technologies for governance, industry, safety, ethics, and the future of global power relations, it will be important for all to be aware and intentional in its adoption.