Data recovery of astronomical proportions

The collapse of the famous Arecibo Observatory Radio Telescope in Puerto Rico on December 1, 2020, was a great loss for science. Still, things could have been a lot worse had it not been for a large-scale data recovery effort.

The collapse of the 57-year-old scientific instrument did not happen entirely out of nowhere. For some time signs of fatigue in the main suspension cables had been apparent, and a large-scale initiative was planned to safe-guard several petabytes of data collected by the telescope and stored locally at the Arecibo Observatory data center.

The recovery operation brought together experts from many organisations, including the University of Central Florida, the Texas Advanced Computing Center, the University of Puerto Rico, the University of Chicago, and the ESnet/Indiana University’s Engagement and Performance Operations Center (EPOC) collaboration. ESnet is the US Department of Energy’s dedicated science network, and others.

Racing against time

Jason Zurawski of ESnet said he and colleagues George Robb and Hans Addleman of Indiana University International Networks became involved in discussions with the observatory and the University of Central Florida in early 2020. The discussions came about during an ESnet requirements review and EPOC Deep Dive support provided to Arecibo.

“In the summer of 2020, these efforts became much more active after the first suspension cable failed (August 10, 2020, ed.), and the Arecibo Observatory began activities to migrate data storage and processing to a commercial cloud,” said Zurawski.

With the failure of a second cable in November 2020, it became obvious that the recovery mission was running short of time.

Because of the urgency, the team decided to move data using physical network attached storage (NAS) appliances; data on tapes and other original sources were loaded onto the NAS at Arecibo. The NAS were then driven to data centers on the island. These centers were connected to the global research and education network infrastructure at 10 Gbps.

A constantly moving supply line

Using Globus data transfer software, the Arecibo Observatory team then began the process of transferring the data to the Texas Advanced Computing Center (TACC). Using multiple devices, and by setting up a constantly moving supply line, they were able to fill a disk, transport to the better-connected locations, start a transfer, take back a completed disk, and return to the data center of the observatory to start the process all over.

ESnet and the entire collaboration team will assure that the data – now stored in the TACC storage cluster – is made widely available to the scientific community to perform new studies.

“The destruction of the Arecibo Observatory Radio Telescope was a catastrophe for global science; however, the quick response of the entire data recovery team helped prevent the loss of much of the valuable data collected by Arecibo over its lifetime. I’m very proud of this accomplishment: the work of the entire ESnet team and our data infrastructure ensured that we had the right capabilities at the right time to make a difference for science,” he said.

The data migration work is set for completion late 2021.

Read the original story on Light Bytes, the ESnet blog.

Image credit: Credit: Michelle Negron, National Science Foundation