At approximately 22:55 on Friday Jul 29th 2022 an equipment failure took place. The affects of this where:
13 Lonsdale nodes lost power and or network access.
rsync.tchpc.tcd.ie lost access to the /home and /projects file system
mounts. The clustered network file system (nfs) nodes that provide those file
system mounts were unavailable because the network switch they were connected
to lost power due to the equipment failure. This would have prevented people
from being able to copy files too and from those file systems via rsync.
Users who logged in with SSH keys would not have been able to use their keys
to login as those keys are saved in the file systems that where then unavailable.
Logins to rsync would have been affected, they would either have been extremely slow or timed out.
The rsync service was failed over to a standby system that does not mount the /home and /projects file system but would allow users to login with their passwords and relay to other clusters without timne outs or being extremely slow.
Status: as of 2022-08-02, 10:30am - rsync has been returned to service. The affected Lonsdale nodes remain down.
We apologise to those affected and are grateful for your patience with this matter.
