Data mining the sky

We meet Stuart Weston at the lab on a crisp autumn afternoon, just shy of winter. Stuart is a PhD candidate at Aukland University of Technology (AUT) and has been responsible for collecting and distributing the data collected by AUT’s two radio astronomy telescopes in Warkworth.

He’s been working in radio astronomy for over five years, after working in commercial IT in the UK and New Zealand.

Demanding user

We couldn’t be happier that he made the switch from the commercial world to the academic world — Stuart is one of our biggest consumers of bandwidth in the country and has a unique passion and enthusiasm for getting the most he can out of REANNZ.

And it doesn’t stop there: he’s challenging us to push further boundaries, making it even easier for radio astronomers to collaborate internationally.

Advancing algorithms

Stuart’s PhD is titled “Data Mining for Statistical Analysis of the Faint Radio Sky: the Pathway to EMU.” In short, he is collecting data about the universe in multiple frequencies (optical, radio and infrared). Different telescopes are used to observe different frequencies. So when researchers find something of interest in radio, they traditionally embark on a manual and time consuming process to see if the same object has been observed in different frequencies by previous observations.

Stuart is developing a statistical method so computers can algorithmically determine matches across different frequency catalogues, and place the results into a database. This database will make it easier for scientists to process and interact with the data.

The result is scientists spending more time analysing their results to further science, and allowing computers to do what they are good at: finding matches in large data sets.

The difference between a week and a day 

The work is understandably data-intensive, with a specific requirement to be able to give access to this data to local and international colleagues. It wouldn’t be possible on a traditional ISP network. This level of data transfer requires an ultra high speed fibre connection.

Up until recently, it would still take weeks to move the data observed by the telescopes. However, since the 10Gb/s link from Warkworth was installed, Stuart has been able to transfer these same experiment data sets in under a day.

“The difference between a week and a day is huge. It makes a big difference to my work and what I’m capable of achieving,” he says.

The GPS in your smartphone relies on data from AUTs telescope

Part of Stuart’s role at Warkworth is to process the raw data collected by the telescope and perform experiments with the data for organisations and institutions outside of AUT. The geodetic experiments are of particular importance, as they feed directly into the GPS network that we have become accustomed to using everyday, whether we are aware of it or not.

The GPS in your car or on your smartphone relies on the data that is coming out of AUT’s telescope, and it requires very quick turn around. An experiment conducted on Thursday needs to be sent to the data correlator on Friday.

Once processed the data is fed into the GPS system, ensuring the most up to date solutions are available for the following Monday. “Having a good high speed network means we can meet and now even work to improve those turn-around times,” Stuart says.

Looking to the future

Looking into the future, Stuart says he’d like to break some of the laws of physics.

“No matter how fast the network is, we still face latency issues. If we could re-write relativity that would be nice, but that’s probably not going to happen. The REANNZ network is now at a point where I’m not worried about whether it can keep up with the work I’m doing.

“REANNZ is ahead of the ball-game and ahead of the curve. It’s now up to us to come up with ways of using what REANNZ is providing in bigger and better ways.”

That doesn’t stop Stuart from thinking bigger though. He wants to see a shift in network usage across New Zealand. He wants to move his work away from being a ‘data collector’ to a ‘data processor’ and ‘data user’, and he’d like to see the same happening in other areas of data-intensive science.

He would like to see the REANNZ link to Australia and the United States ‘running red’ coming back into New Zealand, rather than only going out. “I’d like New Zealand to be contributing at the science level, not just the data level. That’s my ambition and that’s my goal.”

We finish up our Warkworth visit with lunch at the local vineyard and a sampling of the Ransom Vineyard’s 2009 “Cosmos” Chardonnay. It seems that Stuart is not the only one in the region enamoured by space, and all of the possibilities that sit right at our fingertips.