Preparing ECMWF’s Integrated Forecast System (IFS) weather forecast model for Exascale
Authors: George Mozdzynski, ECMWF. Mats Hamrud, ECMWF. Nils Wedi, ECMWF. Jens Doleschal, TUD. Harvey Richardson, Cray.
Today, the European Centre for Medium-Range Weather Forecasts (ECMWF) runs its world leading 16 km global T1279 operational weather forecast model using 1,536 cores of an IBM Power6 system located at its headquarters in Reading, UK.
By 2025 ECMWF expects to be running a 2.5 km global forecast model on an exascale system that should be available, and hopefully affordable, by then. To achieve this would require IFS to run efficiently on about 1000 times the number of cores it uses today. Nothing like a challenge!
But can this be done? Well the good news is that ECMWF, working within the CRESTA project, has already demonstrated IFS running a 10 km global model efficiently on over 50,000 cores of HECToR, the UK’s national HPC resource and a CRAY XE6 based at EPCC, UK. Of course, getting to over a million cores remains a formidable challenge, and many scalability improvements have yet to be implemented.
CRESTA has been essential for the above IFS achievement as it brings together partners such as CRAY, with their experience in PGAS languages, and Technische Universität Dresden, providing exascale enhancements to their profiling tool vampir, just to mention two of the partners in the project.
Now the technical bit. Within CRESTA, ECMWF is exploring the use of Fortran2008 coarrays; in particular, it is possibly the first time that coarrays have been used in a world leading production application within the context of OpenMP parallel regions. The purpose of these optimisations is primarily to allow the overlap of computation and communication, and further, in the semi-Lagrangian advection scheme, to reduce the volume of the data communicated by removing the need for a constant width halo for computing the trajectory of particles of air backwards in time. The importance of this research is such that if these developments are successful, then the IFS model can continue to use the spectral method to 2025 and beyond for the currently planned model resolutions on an exascale sized system. This research is further significant as the techniques used should be applicable to other hybrid MPI/OpenMP codes with the potential to overlap computation and communication.
In a nutshell, IFS is a spectral, semi-implicit, semi-Lagrangian weather prediction code, where model data exists in three spaces, namely, grid-point, Fourier and spectral space. In a single time-step, data is transposed between these spaces so that the respective grid-point, Fourier and spectral computations are independent over two of the three co-ordinate directions in each space. Fourier transforms are performed between grid-point and Fourier space, and Legendre transforms are performed between Fourier and spectral space.
At ECMWF, this same model is used in an Ensemble Prediction System (EPS) suite where today 51 models are run at lower resolution with perturbed input conditions to provide probabilistic information to complement the accuracy of the high resolution deterministic forecast.
The EPS suite is a perfect candidate to run on future exascale systems, with each ensemble member being independent of other such jobs. Increase the number of members and their resolution, and we have the potential to fill an exascale system having an estimated 100 million cores. Hopefully by 2025, a more affordable ($20M) system will have 10 million cores. Even with these huge core counts there will always be a need for a high resolution deterministic forecast which is more challenging to scale than an EPS suite, and the reason for ECMWF’s focus in the CRESTA project.
