Can Generative AI Craft Variable Questions? A Mixed-Method Study on AI’s Capability to Adopt, Adapt, and Create New Scales

Abstract

This study examines the capabilities of Generative artificial intelligence (AI), particularly GPT-4, in transforming research scale development, a process traditionally characterized by extensive time requirements and the potential for human bias. The research aims to clarify whether AI can match and enhance the efficiency and objectivity of research scale creation and adaptation. By adopting a mixed-method design, the study utilizes GPT-4 to generate and modify research scales, which were then rigorously evaluated for reliability and validity and juxtaposed against the scales developed through traditional methodologies. This comprehensive evaluation encompasses quantitative and qualitative assessments and provides a general view of the effectiveness of AI-generated scales. Results revealed GPT-4’s remarkable ability to produce reliable, valid, and comparable research scales that were developed using established methods. Expert feedback further underscores AI’s potential in this field, particularly in reducing human biases and increasing methodological efficiency. A synergistic approach was developed Based on consensus, combining AI’s computational strengths and human oversight and expertise. This study highlights a significant advancement in research methodology and illustrates AI's practical and beneficial integration in scale development. Moreover, it opens new research practice avenues and enables the selection of highly streamlined, unbiased, innovative scale creation processes.