Divide and recombine (D&amp;R) data science projects for deep analysis of big data and high computational complexity

Tung, Wen‐wen; Barthur, Ashrith; Bowers, Matthew C.; Song, Yuying; Gerth, John; Cleveland, William S.

doi:10.1007/s42081-018-0008-4

Cited by 8 publications

(10 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The approach in this paper results in confidence intervals that are roughly consistent with those of MK16, although those in MK16 do not reflect the substantial increase in uncertainty during the most recent period: 60.348C (1896-1925), 60.488C (1926-55), 60.418C (1956-85), and 60.718C (1986-2015) (see Table 4) versus 60.58C for all periods in MK16. Last, studies have sometimes found an additional long-memory weather regime at short time scales up to around two weeks, such as in atmospheric convection, clouds, and precipitation (e.g., Tung et al 2004Tung et al , 2018. The FARIMA models can still serve as crude approximations of such processes, with the short-memory part approximating the high-frequency scaling regime and the long-memory part approximating the low-frequency regime.…”

Section: Discussion and Relevance To Earlier Workmentioning

confidence: 99%

“…More recently, evidence of long memory in various meteorological variables has also begun to accumulate; for example, precipitation (Lovejoy and Mandelbrot 1985;Kantelhardt et al 2006), tropical deep convection (Tung et al 2004), general circulation (Tsonis et al 1999;Vyushin and Kushner 2009), and especially surface temperature (Koscielny- Bunde et al 1998;Weber and Talkner 2001;Caballero et al 2002;Eichner et al 2003;Gil-Alana 2005;Huybers and Curry 2006;Vyushin and Kushner 2009;Franzke 2010Franzke , 2012Yuan et al 2015). A stationary long-memory process exhibits power-law decay in its autocorrelation function, that is, R(k) ; k 2d21 (where 0 , d , 1/2 is the long-memory parameter), much slower than the exponential decay of a short-memory process.…”

Section: A Definition Of Short-and Long-range Dependencementioning

confidence: 99%

“…To clarify the interpretation of Eq. ( 5), we compute the empirical variance-time relations (e.g., Tung et al 2004) Figure 3 shows the estimated variance-time curves for the Potsdam temperature anomalies and the four simulated FARIMA series. Since the series are normalized, the variance-time curves all originate with unit variance at n 5 1; however, they assume different shapes depending on their short-and long-range dependence structures.…”

Section: B Fractional Arima Modelsmentioning

confidence: 99%

“…However, another sort of temporal dependence structure called long-range dependence or long memory has since been detected in numerous physical state variables; for example, rainfall (Lovejoy and Mandelbrot 1985;Kantelhardt et al 2006), streamflow (Hurst 1951;Pelletier and Turcotte 1997;Pandey et al 1998), tropical deep convection (Tung et al 2004), general circulation (Tsonis et al 1999;Vyushin and Kushner 2009), surface temperature (Lovejoy and Schertzer 1986;Koscielny-Bunde et al 1998;Pelletier 1998;Huybers and Curry 2006;Vyushin and Kushner 2009;Franzke 2010Franzke , 2012Yuan et al 2015), and even climate proxies like ice cores (Schmitt et al 1995;Ashkenazy et al 2003;Thomas et al 2009), ocean cores (Shackleton and Imbrie 1990), and tree rings (Bowers et al 2013). The long memory implies a slow power-law decay in a process's autocorrelation function, in contrast to the fast exponential decay of short-memory processes.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Variability and Confidence Intervals for the Mean of Climate Data with Short- and Long-Range Dependence

Bowers

Tung

2018

Journal of Climate

Self Cite

View full text Add to dashboard Cite

This paper presents an adaptive procedure for estimating the variability and determining error bars as confidence intervals for climate mean states by accounting for both short- and long-range dependence. While the prevailing methods for quantifying the variability of climate means account for short-range dependence, they ignore long memory, which is demonstrated to lead to underestimated variability and hence artificially narrow confidence intervals. To capture both short- and long-range correlation structures, climate data are modeled as fractionally integrated autoregressive moving-average processes. The preferred model can be selected adaptively via an information criterion and a diagnostic visualization, and the estimated variability of the climate mean state can be computed directly from the chosen model. The procedure was demonstrated by determining error bars for four 30-yr means of surface temperatures observed at Potsdam, Germany, from 1896 to 2015. These error bars are roughly twice the width as those obtained using prevailing methods, which disregard long memory, leading to a substantive reinterpretation of differences among mean states of this particular dataset. Despite their increased width, the new error bars still suggest that a significant increase occurred in the mean temperature state of Potsdam from the 1896–1925 period to the most recent period, 1986–2015. The new wider error bars, therefore, communicate greater uncertainty in the mean state yet present even stronger evidence of a significant temperature increase. These results corroborate a need for more meticulous consideration of the correlation structures of climate data—especially of their long-memory properties—in assessing the variability and determining confidence intervals for their mean states.

show abstract

Section: Discussion and Relevance To Earlier Workmentioning

confidence: 99%

Section: A Definition Of Short-and Long-range Dependencementioning

confidence: 99%

Section: B Fractional Arima Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Variability and Confidence Intervals for the Mean of Climate Data with Short- and Long-Range Dependence

Bowers

Tung

2018

Journal of Climate

Self Cite

View full text Add to dashboard Cite

show abstract

“…10.1029/2020JD033667 5 of 18 the Midwest. The AR detection and detailed analysis were executed via distributed-parallel computing on a high-performance computing cluster with Hadoop system in the backend and the R language-based DeltaRho software in the frontend (Cleveland & Hafen, 2014;Tung et al, 2018).…”

Section: Ar Detection Algorithmmentioning

confidence: 99%

In Search of the Optimal Atmospheric River Index for US Precipitation: A Multifactorial Analysis

2021

Self Cite

View full text Add to dashboard Cite

show abstract

Climatology and Decadal Changes of Arctic Atmospheric Rivers Based on ERA5 and MERRA-2

Zhang¹,

Tung

Cleveland

2023

Preprint

View full text Add to dashboard Cite

We present {the} Arctic atmospheric river (AR) climatology based on twelve {sets of} labels derived from ERA5 and MERRA-2 reanalyses for 1980–2019. The ARs were identified and tracked in the 3-hourly reanalysis data with a multifactorial approach based on either atmospheric column-integrated water vapor ($IWV$) or integrated water vapor transport ($IVT$) exceeding one of the three climate thresholds (75th, 85th, and 95th percentiles). Time series analysis of the AR event counts from the AR labels showed overall upward trends from the mid-1990s to 2019. The 75th $IVT$- and $IWV$-based labels, as well as the 85th $IWV$-based labels, are likely more sensitive to Arctic surface warming, therefore, detected some broadening of AR-affected areas over time, while the rest of the labels did not. Spatial exploratory analysis of these labels revealed that the AR frequency of occurrence maxima shifted poleward from over-land in 1980–1999 to over the Arctic Ocean and its outlying Seas in 2000–2019. Regions across the Atlantic, the Arctic, to the Pacific Oceans trended higher AR occurrence, surface temperature, and column-integrated moisture. Meanwhile, ARs were increasingly responsible for the rising moisture transport into the Arctic. Even though the increase of Arctic AR occurrence was primarily associated with long-term Arctic surface warming and moistening, the effects of changing atmospheric circulation could stand out locally, such as on the Pacific side over the Chukchi Sea. The changing teleconnection patterns strongly modulated AR activities in time and space, with prominent anomalies in the Arctic-Pacific sector during the latest decade. Besides, the extreme events identified by the 95th-percentile labels displayed the most significant changes and were most influenced by the teleconnection patterns. The twelve Arctic AR labels and the detailed graphics in the atlas can help navigate the uncertainty of detecting and quantifying Arctic ARs and their associated effects in current and future studies.

show abstract

Divide and recombine (D&R) data science projects for deep analysis of big data and high computational complexity

Cited by 8 publications

References 32 publications

Variability and Confidence Intervals for the Mean of Climate Data with Short- and Long-Range Dependence

Variability and Confidence Intervals for the Mean of Climate Data with Short- and Long-Range Dependence

In Search of the Optimal Atmospheric River Index for US Precipitation: A Multifactorial Analysis

Climatology and Decadal Changes of Arctic Atmospheric Rivers Based on ERA5 and MERRA-2

Contact Info

Product

Resources

About