Participant Interactions with Artificial Intelligence: Using Large Language Models to Generate Research Materials for Surveys and Experiments

Abstract

Researchers are increasingly exploring the use of large language models (LLMs) to develop materials for surveys and experiments. However, clear guidance on effective implementation remains limited. In this paper, we propose a decision-making framework comprising five use cases for integrating large language models into psychological survey and experimental methods: (1) LLM as research assistant; (2) LLM as adaptive content creator; (3) LLM as external resource; (4) LLM as conversation partner, and (5) LLM as research confederate. To support these applications, we introduce the open-source Qualtrics-AI Link (QUAIL), a software designed to integrate content generated by ChatGPT's LLM foundation model into the Qualtrics platform. Across contexts, and for all scenarios involving the use of LLMs in research material creation, we provide guidance on the technical steps necessary to support both internal and external validity. These include effective prompt engineering, model selection, alpha and beta testing, launching, and monitoring. We conclude with a discussion of relevant ethical considerations, cautions, and resources for auditing validity claims. Throughout, we emphasize that good research design and adherence to ethical principles should guide decision-making, and that researcher expertise in both LLMs and research design is essential to ensure valid participant interactions when using LLM-based tools.