International collaboration challenges and solutions for Ukraine’s particle physicists

Technology upgrades can bring both challenges and solutions. For Ukrainian researchers at Kharkiv Institute of Physics and Technology (KIPT), planned changes to the Large Hadron Collider (LHC) run by the European Organisation for Nuclear Research, CERN, will bring a challenge. But a new ‘spectrum’ digital highway provided by URAN and the EU-funded EaPConnect project offers a ‘future-proofed’ solution that will meet KIPT’s growing data flow needs.

PhD Leonid Levchuk is Head of the Department of Experimental Studies in High Energy Particle and Nuclear Physics at the KIPT National Science Centre (NSC) – the only Ukrainian team that is processing data from CERN’s LHC. An important research direction of his department is participating in the large, multi-purpose Compact Muon Solenoid (CMS) experiment. “This work could not be fulfilled without access to the pan-European GÉANT network infrastructure,” Dr Levchuk says.

This access has been provided by URAN, the Ukrainian Research and Academic Network through its partnership in EaPConnect, the EU4Digital: Connecting Research and Education Communities project. A new ’spectrum’ link, operational since September 2021, vastly increases the capacity of the network connection to GÉANT, and is capable of handling further growth in data flow in future. This is a major achievement in the evolution of the networks supporting the KIPT team.

Evolving needs and networks

Before the LHC started up in 2009, NSC KIPT constructed a computer facility commissioned in the CMS grid infrastructure as a ‘Tier-2 site’ in the distributed infrastructure for data processing. Dr. Levchuk’s team must provide a high level of reliability at the facility. This is no simple task: it’s necessary to constantly monitor the operation of the facility and ensure the proper functioning of the sophisticated, specialised software.

“Since then, about 20 Petabytes of experimental information have been transferred for processing to this ‘T2_UA_KIPT’ facility. T2_UA_KIPT operates 24 hours a day with a high level of operational quality, determined by ‘availability/readiness’ metrics. In terms of these metrics, it is one of the best CMS Tier-2 centres”, Dr Levchuk emphasises.

This achievement became possible only thanks to GÉANT network infrastructure access. “For example, KIPT got access to some US CMS sites that only have an Internet connection through Internet2 or ESnet, because these are accessible from the European GÉANT network. Fortunately, we got this access owing to an agreement between the URAN and our academic internet provider (UARNet), and all the needed T2_UA_KIPT network links were commissioned”, Dr Levchuk explains.

Over time, his team has faced the challenge of increasing bandwidth needs. Alexey Kurov, a research associate in Dr Levchuk’s department, described this in a 2019 interview: “Unfortunately, our channels did not have the needed capacity, so we could not achieve the required efficiency of data processing at the facility… We could not resolve this problem on our own and so we went to URAN Association for help”.

Cooperation between URAN and the NSC KIPT team has become direct, close and regular. “Due to this cooperation and the EaPConnect project, our LHCONE/GÉANT channel bandwidth has been gradually increased to 10 Gigabits per second. This has allowed us to provide the network throughput that matches the requirements placed on CMS Tier-2 sites”, Dr Levchuk says.

New challenges and the ‘spectrum’ solution

Data flow from the CMS experiment has continued to grow. Dr Levchuk and his colleagues have sometimes seen saturation of their CMS data flow channel. They have already planned for its extension to around 40 Gigabits per second. But an extra challenge comes from CERN’s plans to upgrade the LHC by 2027. CERN will increase the luminosity of the particle collider by a factor of 5 to 7.5: with greater luminosity comes more experimental information, which has to be transferred through the networks for processing, in particular, at the CMS Tier-2 sites.

This new challenge has to be addressed from two directions, Dr Levchuk reflects: the first is to ensure funding from the Ukrainian government for a substantial upgrade of the system. There are some boundaries set by the speed of input/output operations on the disk storage. During data transfer, such operations normally occur in parallel on several storage disk arrays, but the majority of these arrays at T2_UA_KIPT are already full. “Unfortunately, the T2_UA_KIPT hardware performance, including the storage capacity, is around 3 times below what is called the CMS ‘nominal’ Tier-2 site, and the facility needs to be upgraded”, Dr Levchuk shares his concern.

The second direction needed is to extend the network bandwidth above 100 Gigabits per second. The new network link implemented through EaPConnect already achieves this capacity. And its ‘spectrum’ capability will enable data transmission using different wavelengths of light, so potentially increasing the capacity of the infrastructure to around 400 Gigabit per second.

With this, NSC KIPT can continue to support excellent functionality of its CMS Tier-2 centre, allowing Dr Levchuk’s team to honourably represent Ukraine in the global scientific arena.