Joint Seminar: Extreme data challenges and solutions towards an Exascale weather forecasting system

Starting 2014, ECMWF has embarked on a 10 year research programme on HPC Scalability, aiming to achieve Exascale numerical weather prediction systems by 2025. The programme is now entering its 2nd phase, where focus is now on improvements to the operational forecasting system that incur deeper restructuring changes. ECMWF operational forecast generates massive amounts of I/O in short bursts, accumulating to tens of TiB in hourly windows. From this output, millions of user-defined daily products are generated and disseminated to member states and commercial clients all over the world. These products are processed from the raw output of the IFS model, within the time critical path and under strict delivery schedule. Upcoming rise in resolution and growing popularity will increase both the size and number of these products. Based on expected model resolution upgrades, by 2021 we estimate that each of the 4 daily runs of the operational model will output around 100 TiB which need to be produced and post-processed within 1 hour. Given that the I/O workload is already one of the strongest bottlenecks in ECMWF's workflow, this is one of the main challenges to reach Exascale NWP. We present the latest ECMWF developments in model I/O, product generation and storage, and how we are reworking our operational workflows to adapt to forthcoming new architectures and memory-storage hierarchies. In particular, we present recent developments in the integration of a domain-specific object store to ECMWF's operational pipeline, the challenges that such implementation brought and the opportunities that it opened for future improvements.




15:15 h


Bundesstr. 53, room 022/023
Seminar Room 022/023, Ground Floor, Bundesstrasse 53, 20146 Hamburg, Hamburg


Tiago Quintino, ECMWF, Forecast Department


Reinhard Budich

Attached files

Back to listing