Accelerating New Zealand’s climate research

For climate scientist Dr Jonny Williams of NIWA, the National Institute of Water and Atmospheric Research, global climate models are key to uncovering the future state of New Zealand’s climate. The mountains of numerical data that make up these simulations enable scientists to forecast climate scenarios by tracking and predicting future sea surface temperatures, weather and greenhouse gas levels. Jonny’s research contributes to global projects that compare the modelling data from New Zealand to these other global models.

Fast data transfers for global co-operation

Jonny uses REANNZ and the Globus data management system to shift data worldwide with the click of a button.

“It allows us to transfer data much faster than traditional methods. Because the data volumes that we need to transfer are so large, traditional methods are just simply not fast enough,” he says.

“One simulation that we run with our collaborators can easily produce 50-to-100 terabytes of data. That’s not seen as a large amount of data in the climate research world. On a daily basis, I can deal with individual files that are 100 gigabytes in size.”

Jonny says that this solution has enabled a new workflow that allows multiple tens or hundreds of terabytes to be routinely shifted around the world.

“People used to joke that you shouldn’t underestimate the bandwidth of a horse and cart filled with USB sticks if you just wanted to shift one large amount of data once – but we needed to do this routinely and regularly. If we were still relying on the older computer transfer technologies – such as SCP and rsync – then that workflow would just simply not be possible. It’s not just a case of speeding up something that we were able to do before and making it more efficient, this has actually enabled us to perform completely new workflows and to take part in these very large international collaborations,” he says.

Zooming in on New Zealand

Regional modelling teams have been enhancing the resolution of New Zealand’s climate modelling, contributing their techniques and the data collected to build more accurate models.

“We work very closely with regional modelling teams. Even though they’re modelling a much smaller area, they’re modelling it at a higher resolution. The amount of data that they produce can be even larger,” says Jonny.

The scale of information collected, stored and shared in global climate research makes it extremely data intensive. While comparing a New Zealand model to a UK model, Jonny and his colleagues may shift tens, sometimes hundreds, of terabytes of data across the world. That’s where access to national eResearch infrastructure with international reach becomes crucial.

“Without international collaboration, we wouldn’t be able to do the work we do. The computer models that we run are some of the most complex pieces of software in the world. Our international partners provide us with a code base to work with. The code that we use has been developed by hundreds of scientists over decades, it has a very long history. That is just simply not something that you can do on your own.”

The scale of data being produced in fields like climate research continues to grow. NIWA’s contribution to the latest Intergovernmental Panel on Climate Change Report requires Jonny and his colleagues to share data reliably overseas. Projects like this need to access remote equipment and analysis platforms.

Computing and the network

“The HPC at NIWA produces the data that needs to be analysed, but we also rely heavily on communication with other parts of the world. The ability to share our data comes from the interconnection between the HPC and the network. The international network that REANNZ provides here is essential,” he says.

NIWA is a Crown Research Institute that works on climate, freshwater and ocean science. One aspect of NIWA’s work is in global climate modelling. They collaborate with international partners on simulations that simulate the flow of air, water and sea ice around the world. They also study how these physical processes react and interact with chemical and biological cycles in the air, on the land and in the sea.

REANNZ is New Zealand’s research and education network (NREN). As an NREN, the network enables research, productivity and collaboration through data movement at a scale rarely found outside of the research and education sector.

New Zealand eScience Infrastructure (NeSI) designs, builds, and operates a specialised platform of shared high performance computing (HPC) infrastructure and a range of eResearch services. They work closely with NIWA and REANNZ to provide supercomputing services and expertise to researchers like Jonny