At approximately 16:44 on Monday Aug 22nd 2022 power to a network switch was accidentally removed while other works were being carried out. That network switch is located in the Green Data Centre, (GDC), in Aras an Phiarsaigh and is the GDC side network link for the Research IT network between the GDC and the Lloyd Data Centre, (LDC).
The Research IT network uplink to the Trinity network and hence the internet is located in the GDC. Because the link between the two data centres was severed, any systems or services which are hosted in the LDC were inaccessible from the rest of the College network or the internet. The Kelvin, Lonsdale, Parsons and TCIN HPC clusters and Rsync, (our access node), are located in the LDC. Computers on the Research IT network, such as desktops on the 2nd floor of the Lloyd Building, would have been able to connect to systems in the Lloyd Data Centre but not able to connect further upstream.
Systems hosted in the Green Data Centre, such as our private cloud and the other HPC clusters we host were accessible from the rest of the Trinity network. However, the Boole, Boyle, Crusher, Dalton, Pople, Seagull and Tinney HPC Clusters in the GDC were not able to communicate with the Slurm database which is hosted in the LDC. That lead to the nodes in those clusters going down. The LDC hosted HPC Clusters, Kelvin, Lonsdale, Parsons and TCIN were not affected by that and their nodes did not fail.
Power was restored to the affected network switch in the GDC and by about 17:03 on Mon August 22nd the network link between the GDC and LDC was restored. The compute nodes in the affected clusters were then returned to service.
We sincerely apologise to this service interruption.
