Open source and reproducible and inexpensive infrastructure for data challenges and education

Abstract

Data sharing is necessary to maximize the actionable knowledge generated from research data. Data challenges can encourage secondary analyses of datasets. Data challenges in biomedicine often rely on advanced cloud-based computing infrastructure and expensive industry partnerships. Examples include challenges that use Google Cloud virtual machines and the Sage Bionetworks Dream Challenges platform. Such robust infrastructures can be financially prohibitive for investigators without substantial resources. Given the potential to develop scientific and clinical knowledge and the NIH emphasis on data sharing and reuse, there is a need for inexpensive and computationally lightweight methods for data sharing and hosting data challenges. To fill that gap, we developed a workflow that allows for reproducible model training, testing, and evaluation. We leveraged public GitHub repositories, open-source computational languages, and Docker technology. In addition, we conducted a data challenge using the infrastructure we developed. In this manuscript, we report on the infrastructure, workflow, and data challenge results. The infrastructure and workflow are likely to be useful for data challenges and education.

Enhancing Early Detection of Cognitive Decline in the Elderly: A Comparative Study Utilizing Large Language Models in Clinical Notes

“…We noticed that Microsoft Azure GPT-4 outperformed AWS EC2 Llama 2 in terms of price, execution speed, and accuracy. However, as an open-source model, Llama 2 may have a better reproducibility 37 while GPT-4 may provide slightly different answers over time due to model updates from OpenAI 38 . Secondly, we tested several prompting strategies, and found doing error analysis using some training data, and then asking LLM to avoid summarized common errors by revising the prompt can be an effective way to improve LLM's performance.…”