Making massive computational experiments painless

Monajemi, Hatef; Donoho, David L.; Stodden, Victoria

doi:10.1109/bigdata.2016.7840870

Cited by 43 publications

(22 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…It is possible that the emerging emphasis on open data will help us detect and characterize instances of nonergodicity broadly in the life sciences. For example, open-data efforts have led to increasing researcher interactions and open conversations about replicability and validity in comparative bioinformatics (40), breast cancer prediction (48), and massive computational experiments (49). At the very least, open-data access would allow researchers to examine the extent to which previously published results may evince varying degrees of nonergodicity.…”

Section: Discussionmentioning

confidence: 99%

Lack of group-to-individual generalizability is a threat to human subjects research

Fisher

Medaglia

Jeronimus

2018

Proc. Natl. Acad. Sci. U.S.A.

809

688

View full text Add to dashboard Cite

Only for ergodic processes will inferences based on group-level data generalize to individual experience or behavior. Because human social and psychological processes typically have an individually variable and time-varying nature, they are unlikely to be ergodic. In this paper, six studies with a repeated-measure design were used for symmetric comparisons of interindividual and intraindividual variation. Our results delineate the potential scope and impact of nonergodic data in human subjects research. Analyses across six samples (with 87-94 participants and an equal number of assessments per participant) showed some degree of agreement in central tendency estimates (mean) between groups and individuals across constructs and data collection paradigms. However, the variance around the expected value was two to four times larger within individuals than within groups. This suggests that literatures in social and medical sciences may overestimate the accuracy of aggregated statistical estimates. This observation could have serious consequences for how we understand the consistency between group and individual correlations, and the generalizability of conclusions between domains. Researchers should explicitly test for equivalence of processes at the individual and group level across the social and medical sciences.

show abstract

Section: Discussionmentioning

confidence: 99%

Lack of group-to-individual generalizability is a threat to human subjects research

Fisher

Medaglia

Jeronimus

2018

Proc. Natl. Acad. Sci. U.S.A.

809

688

View full text Add to dashboard Cite

show abstract

“…It is entirely feasible to extend analyses on NeuroCAAS with more sophisticated methods of parallelization as offered by these tools on a per-analysis basis, as an immutable analysis environment contains all of the infrastructure available in typical computing environments, spanning infrastructure ranging from workstation to compute cluster. Lastly, other tools offer reproducible analyses to researchers as a service (Monajemi et al, 2016,Šimko et al, 2019, Brinckman et al, 2019, https://flywheel.io). NeuroCAAS is again complementary to these methods and services because it is designed explicitly for a heterogeneous community of neuroscientists, giving developers access to all the resources required to create powerful, general-purpose analyses, while simultaneously removing all major barriers of entry to these analyses for a diverse population of users.…”

Section: Discussionmentioning

confidence: 99%

“…For example, limitations on hardware capacity have real consequences for the performance of analysis algorithms (Radiuk, 2017), and can interfere with analysis performance in ways that preclude the most careful attempts to reproduce the appropriate infrastructure. Further, the difficulty of configuring analysis infrastructure drives yet more divergence (Demchenko et al, 2013, Monajemi et al, 2016, and can make errors extremely difficult to detect, let alone repair. This challenge persists even if the original developer provides clear and comprehensive instructions for use (which is rarely the case and an ongoing challenge across science (Zhao and Deek, 2005, Stodden et al, 2018, Raff, 2019.…”

Section: Introductionmentioning

confidence: 99%

Neuroscience Cloud Analysis As a Service

Abe

Kinsella

Saxena

et al. 2020

Preprint

View full text Add to dashboard Cite

A major goal of computational neuroscience is to develop powerful analysis tools that operate on large datasets. These methods provide an essential toolset to unlock scientific insights from new experiments. Unfortunately, a major obstacle currently impedes progress: while existing analysis methods are frequently shared as open source software, the infrastructure needed to deploy these methods -at scale, reproducibly, cheaply, and quickly -remains totally inaccessible to all but a minority of expert users. As a result, many users can not fully exploit these tools, due to constrained computational resources (limited or costly compute hardware) and/or mismatches in expertise (experimentalists vs. large-scale computing experts). In this work we develop Neuroscience Cloud Analysis As a Service (NeuroCAAS): a fully-managed infrastructure platform, based on modern large-scale computing advances, that makes state-of-the-art data analysis tools accessible to the neuroscience community. We offer NeuroCAAS as an open source service with a drag-and-drop interface, entirely removing the burden of infrastructure expertise, purchasing, maintenance, and deployment. NeuroCAAS is enabled by three key contributions. First, NeuroCAAS cleanly separates tool implementation from usage, allowing cutting-edge methods to be served directly to the end user with no need to read or install any analysis software. Second, NeuroCAAS automatically scales as needed, providing reliable, highly elastic computational resources that are more efficient than personal or lab-supported hardware, without management overhead. Finally, we show that many popular data analysis tools offered through NeuroCAAS outperform typical analysis solutions (in terms of speed and cost) while improving ease of use and maintenance, dispelling the myth that cloud compute is prohibitively expensive and technically inaccessible. By removing barriers to fast, efficient cloud computation, NeuroCAAS can dramatically accelerate both the dissemination and the effective use of cutting-edge analysis tools for neuroscientific discovery.

show abstract

“…The statistical performance assessment procedure (TruMAP) and results represent an example of massive computational experiments to quantify the effectiveness of a new algorithm. As Monajemi et al (2016) discussed, with advancement of computational power and tools, researchers are expected to present findings that involve such massive computations (order one million CPU-hours) as opposed to deducing conclusions from a limited number of test cases.…”

Section: Discussionmentioning

confidence: 99%

Closed-loop field development with multipoint geostatistics and statistical performance assessment

Shirangi

2019

Journal of Computational Physics

View full text Add to dashboard Cite

Closed-loop field development (CLFD) optimization is a comprehensive framework for optimal development of subsurface resources. CLFD involves three major steps: 1) optimization of full development plan based on current set of models, 2) drilling new wells and collecting new spatial and temporal (production) data, 3) model calibration based on all data. This process is repeated until the optimal number of wells is drilled. This work introduces a new CLFD implementation for complex systems described by multipoint geostatistics (MPS). Model calibration is accomplished in two steps: conditioning to spatial data by a geostatistical simulation method, and conditioning to production data by optimization-based PCA. A statistical procedure (TruMAP) is presented to assess the performance of CLFD. For performance assessment by TruMAP, the methodology is applied to an oil reservoir example for 25 different true-model cases. Application of a single-step of CLFD, improved the true NPV in 64%-80% of cases. The full CLFD procedure (with three steps) improved the true NPV in 96% of cases, with an average improvement of 37%. These results indicate the effectiveness of performing multiple steps of closed-loop optimization. This massive computational experiment involved about 9.5 million reservoir simulation runs that took about 320,000 CPU hours.

show abstract

Making massive computational experiments painless

Cited by 43 publications

References 12 publications

Lack of group-to-individual generalizability is a threat to human subjects research

Lack of group-to-individual generalizability is a threat to human subjects research

Neuroscience Cloud Analysis As a Service

Closed-loop field development with multipoint geostatistics and statistical performance assessment

Contact Info

Product

Resources

About