Within the Copernicus Climate Change Service (C3S), ECMWF is producing the ERA5 reanalysis which, once completed, will embody a detailed record of the global atmosphere, land surface and ocean waves from 1950 onwards. This new reanalysis replaces the ERA-Interim reanalysis (spanning 1979 onwards) which was started in 2006. ERA5 is based on the Integrated Forecasting System (IFS) Cy41r2 which was operational in 2016. ERA5 thus benefits from a decade of developments in model physics, core dynamics and data assimilation. In addition to a significantly enhanced horizontal resolution of 31 km, compared to 80 km for ERA-Interim, ERA5 has hourly output throughout, and an uncertainty estimate from an ensemble (3-hourly at half the horizontal resolution). This paper describes the general setup of ERA5, as well as a basic evaluation of characteristics and performance, with a focus on the dataset from 1979 onwards which is currently publicly available. Re-forecasts from ERA5 analyses show a gain of up to one day in skill with respect to ERA-Interim. Comparison with radiosonde and PILOT data prior to assimilation shows an improved fit for temperature, wind and humidity in the troposphere, but not the stratosphere. A comparison with independent buoy data shows a much improved fit for ocean wave height. The uncertainty estimate reflects the evolution of the observing systems used in ERA5. The enhanced temporal and spatial resolution allows for a detailed evolution of weather systems. For precipitation, global-mean correlation with monthly-mean GPCP data is increased from 67% This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

The ECMWF twentieth century reanalysis (ERA-20C; 1900–2010) assimilates surface pressure and marine wind observations. The reanalysis is single-member, and the background errors are spatiotemporally varying, derived from an ensemble. The atmospheric general circulation model uses the same configuration as the control member of the ERA-20CM ensemble, forced by observationally based analyses of sea surface temperature, sea ice cover, atmospheric composition changes, and solar forcing. The resulting climate trend estimations resemble ERA-20CM for temperature and the water cycle. The ERA-20C water cycle features stable precipitation minus evaporation global averages and no spurious jumps or trends. The assimilation of observations adds realism on synoptic time scales as compared to ERA-20CM in regions that are sufficiently well observed. Comparing to nighttime ship observations, ERA-20C air temperatures are 1 K colder. Generally, the synoptic quality of the product and the agreement in terms of climate indices with other products improve with the availability of observations. The MJO mean amplitude in ERA-20C is larger than in 20CR version 2c throughout the century, and in agreement with other reanalyses such as JRA-55. A novelty in ERA-20C is the availability of observation feedback information. As shown, this information can help assess the product’s quality on selected time scales and regions.

Members in ensemble forecasts differ due to the representations of initial uncertainties and model uncertainties. The inclusion of stochastic schemes to represent model uncertainties has improved the probabilistic skill of the ECMWF ensemble by increasing reliability and reducing the error of the ensemble mean. Recent progress, challenges and future directions regarding stochastic representations of model uncertainties at ECMWF are described in this article. The coming years are likely to see a further increase in the use of ensemble methods in forecasts and assimilation. This will put increasing demands on the methods used to perturb the forecast model. An area that is receiving greater attention than 5–10 years ago is the physical consistency of the perturbations. Other areas where future efforts will be directed are the expansion of uncertainty representations to the dynamical core and other components of the Earth system, as well as the overall computational efficiency of representing model uncertainty.

A relationship between busted European forecasts, a Rockies trough, and storms over eastern North America suggests the importance of improving quality and use of observations, model depiction of convective systems, and representation of uncertainties.

This article investigates the use of an updated observation-error covariance matrix for the Infrared Atmospheric Sounding Interferometer (IASI) in the European Centre for Medium-Range Weather Forecasts (ECMWF) system. The new observation-error covariance matrix is based on observation-space diagnostics and includes interchannel error correlations, but also assigns significantly altered error standard deviations. The update is investigated in detail in assimilation experiments, including an assessment of the role of error inflation and taking interchannel error correlations into account.The updated observation-error covariance leads to a significant improvement in the use of IASI data, especially in the Tropics and the stratosphere and particularly for humidity and ozone. The benefits are especially strong for short-range forecasts, whereas the impact in the medium range is less pronounced.The study highlights the benefits of taking interchannel error correlations into account, which allows the use of an observation-error covariance for IASI that is overall more consistent with departure statistics. At the same time, the study also demonstrates that error inflation can be used to compensate partially, though not fully, for neglected error correlations. Adjustments such as scaling of the originally diagnosed observation-error estimates are also found to be beneficial when the diagnosed interchannel error correlations are taken into account.

A hybrid assimilation system which uses sample statistics from an Ensemble of Data Assimilations (EDA) to estimate background error variances has been implemented at ECMWF. We show that the new system is beneficial in terms of deterministic forecast skill, provided that random and systematic errors in the estimation of variances are properly accounted for. The mechanisms through which EDA sample variances influence the deterministic analysis are clarified. An interesting aspect is that the use of flow-dependent variances alone is able to introduce a significant degree of flow dependency in the analysis increments. Further possible improvements and extensions to the current methodology are also discussed.

The trend towards using flow‐dependent, ensemble‐based estimates of background‐error covariances has been one of the main themes of atmospheric data assimilation research and development in recent years. In this work it is documented how flow‐dependent ensemble information from the ECMWF ensemble of data assimilations (EDA) has gradually been incorporated into the B model which describes the background‐error covariance matrix at the start of the ECMWF 4D‐Var assimilation window. Starting with background‐error variances for the balanced part of the control vector and observation quality control, the current article extends the flow‐dependency to background‐error variances for the unbalanced part of the control vector and for background‐error correlation structures. The correlations are determined either online from previous days or from a hybrid of climatological and current cycle estimates. Each of these changes is shown to improve both the realism of the modelled B and the accuracy of the analysis and forecast fields produced by the 4D‐Var assimilation cycle which makes use of the improved B. Finally, increasing the resolution at which the EDA 4D‐Vars are run is shown to reduce the underestimation of the EDA‐based error estimates.

Model error is one of the main obstacles to improved accuracy and reliability in numerical weather prediction (NWP) and climate prediction conducted with state-of-the-art, comprehensive high-resolution general circulation models. In a data assimilation framework, recent advances in the context of weak-constraint 4D-Var have shown that it is possible to estimate and correct for a large fraction of systematic model error which develops in the stratosphere over short forecast ranges. The recent explosion of interest in machine learning/deep learning technologies has been driven by their remarkable success in disparate application areas. This raises the question of whether model error estimation and correction in operational NWP and climate prediction can also benefit from these techniques. In this work, we aim to start to give an answer to this question. Specifically, we show that artificial neural networks (ANNs) can reproduce the main results obtained with weak-constraint 4D-Var in the operational configuration of the IFS model of the European Centre for Medium-Range Weather Forecasts (ECMWF). We show that the use of ANN models inside the weak-constraint 4D-Var framework has the potential to extend the applicability of the weak-constraint methodology for model error correction to the whole atmospheric column. Finally, we discuss the potential and limitations of the machine learning/deep learning technologies in the core NWP tasks. In particular, we reconsider the fundamental constraints of a purely data-driven approach to forecasting and provide a view on how to best integrate machine learning technologies within current data assimilation and forecasting methods. Plain Language Summary Model error is one of the main obstacles to improved accuracy and reliability in current numerical weather prediction and in climate prediction. Recent advances in data assimilation at the European Centre for Medium-Range Weather Forecasts (ECMWF) indicate that it is possible to estimate and correct for a large fraction of systematic model error in the stratosphere. The question we address here is whether machine learning techniques can be used alone and in conjunction with standard data assimilation methods to improve on those results. We show that it is indeed possible to extend current data assimilation capabilities in operational state-of-the-art forecast systems using machine learning tools, and we discuss the potential and limitations of future applications of these ideas to other core NWP tasks.