Data-driven hypothesis generation among inexperienced clinical researchers: A comparison of secondary data analyses with visualization (VIADS) and other tools

Abstract

analytic tool for filtering and summarizing large health data sets coded with hierarchical terminologies (VIADS) or other analytical tools routinely used by participants on the same datasets. Methods: We recruited clinical researchers from all over the United States of America and separated them into "experienced" and "inexperienced" groups using predetermined criteria. Within the groups, participants were randomly assigned to a VIADS or non-VIADS groups (control) group. We recruited two participants for the pilot study and 18 for the main study. Fifteen (out of 18) were junior clinical researchers, including seven in the control group and eight in the VIADS group. All participants used the same datasets and study scripts. Each participant conducted a remote 2-hour study session for hypothesis generation. The VIADS groups also had a 1-hour training session. The same researcher coordinated the study session. Two participants in the pilot study were one experienced and one inexperienced clinical researcher. During the session, all participants followed a think-aloud protocol to verbalize their thoughts and actions during data analysis and hypothesis generation. Follow-up surveys were administered to all participants after each study session. All screen activities and audio were recorded, transcribed, coded, and analyzed. Every ten randomly selected hypotheses were included in one Qualtrics survey for quality evaluation. Seven expert panel members rated each hypothesis on validity, significance, and feasibility. Results: Eighteen participants generated 227 hypotheses, of which 147 (65%) were valid based on our criteria. Each participant generated between one and 19 valid hypotheses during the 2-hour session. The VIADS and control groups generated a similar number of hypotheses on average. It took the VIADS group participants approximately 258 seconds to generate one valid hypothesis; for the control group- it took 379 seconds; however, the difference was not statistically significant. Furthermore, the validity and significance of the hypotheses were slightly lower in the VIADS group, though not statistically significant. The feasibility of the hypotheses was statistically significantly lower in the VIADS group than in the control group. The average quality rating of hypotheses per participant ranged from 7.04 to 10.55 (out of 15). In the follow-up surveys, VIADS users provided overwhelmingly positive feedback on VIADS, and they all agreed (100%) that VIADS offered new perspectives on the datasets. Conclusion: The role of VIADS in hypothesis generation trended favorably with respect to the assessment of hypotheses generated; however, a statistically significant difference was not reached, possibly related to sample size or the 2-hour study session being inadequate. Further characterization of hypotheses, including specifics on how they might be improved, could guide future tool development. Larger-scale studies may help to reveal more conclusive hypothesis generation mechanisms.