Data-driven hypothesis generation among inexperienced clinical researchers: A comparison of secondary data analyses with visualization (VIADS) and other tools

Abstract

Objectives: To compare how clinical researchers generate data-driven hypotheses with a visual interactive analytic tool (VIADS, a visual interactive analysis tool for filtering and summarizing large datasets coded with hierarchical terminologies) or other tools. Methods: We recruited clinical researchers and separated them into "experienced" and "inexperienced" groups. Participants were randomly assigned to a VIADS or control group within the groups. Each participant conducted a remote 2-hour study session for hypothesis generation with the same study facilitator on the same datasets by following a think-aloud protocol. Screen activities and audio were recorded, transcribed, coded, and analyzed. Hypotheses were evaluated by seven experts on their validity, significance, and feasibility. We conducted multilevel random effect modeling for statistical tests. Results: Eighteen participants generated 227 hypotheses, of which 147 (65%) were valid. The VIADS and control groups generated a similar number of hypotheses. The VIADS group took a significantly shorter time to generate one hypothesis (e.g., among inexperienced clinical researchers, 258 s versus 379 s, p = 0.046, power = 0.437, ICC = 0.15). The VIADS group received significantly lower ratings than the control group on feasibility and the combination rating of validity, significance, and feasibility. Conclusion: The role of VIADS in hypothesis generation seems inconclusive. The VIADS group took a significantly shorter time to generate each hypothesis. However, the combined validity, significance, and feasibility ratings of their hypotheses were significantly lower. Further characterization of hypotheses, including specifics on how they might be improved, could guide future tool development.

Data-Driven Hypothesis Generation in Clinical Research: What We Learned from a Human Subject Study?

“…Two additional clinical researchers, including one experienced and one inexperienced, participated in the pilot study to help finalize the study flow, scripts, and follow-up surveys before our formal study started. Detailed results can be found in the reference 9 .…”

“…Moreover, the VIADS group had a much smaller standard deviation than the control group regarding the cognitive events used, i.e., 2.43 versus 5.02. More detailed results can be found in the references 9,108 .…”

“…We conducted a 2 × 2 human participant study between August 2021 and November 2022 9 . We recruited clinical researchers and separated them into experienced and inexperienced groups based on predefined criteria 10 .…”

“…Despite the importance of hypothesis generation in scientific studies, the cognitive process of hypothesis generation has not yet been well understood. Our group has conducted a datadriven hypothesis generation study with clinical researchers to explore the process in the clinical research context [6][7][8][9][10] . We developed a visual interactive analytical tool for filtering, summarizing, and visualizing large health data sets coded with hierarchical terminologies-VIADS [11][12][13][14][15][16] , and we compared the hypothesis generation processes among clinical researchers when they used VIADS and any other analytical tools, such as Excel, SPSS, R. The original study protocol 10 and detailed individual aspects of the study results have been published separately, including usability, utility, hypothesis measure instruments, cognitive events [6][7][8][9] .…”

“…Our group has conducted a datadriven hypothesis generation study with clinical researchers to explore the process in the clinical research context [6][7][8][9][10] . We developed a visual interactive analytical tool for filtering, summarizing, and visualizing large health data sets coded with hierarchical terminologies-VIADS [11][12][13][14][15][16] , and we compared the hypothesis generation processes among clinical researchers when they used VIADS and any other analytical tools, such as Excel, SPSS, R. The original study protocol 10 and detailed individual aspects of the study results have been published separately, including usability, utility, hypothesis measure instruments, cognitive events [6][7][8][9] . In this perspective paper, we aim to (1) provide a literature review on the intersectional context of scientific thinking, reasoning, discovery, medical reasoning, and literature-based discovery in clinical research that serves as the background of our study and (2) elaborate on our study, its methods and results, its significance, and its roles within the clinical research context.…”