Last weekend, AARNET network operations detected that systems running on QRIScloud were directing high volumes of unwanted network traffic towards servers in China. (In network parlance, this was a DDOS attack.) To stop this, AARNET had to institute network blocks on a wide range of the network addresses used in the QRIScloud infrastructure.
The upshot of this is that the following QRIScloud infrastructure is currently not working:
- The DNS server that enables access to QRIScloud RDSI collections is offline. This means that you will not be able to connect to your collection VMs.
- The QRIScloud OpenStack cell is disconnected from the rest of the NeCTAR Research Cloud. This means that instance launching, snapshots and so on will not work.
- One compute node ("CN56") is unavailable.
At this stage, we do not have a good estimate for how long this outage will continue. Unfortunately, we cannot ask AARNET to remove the blocks until we are confident we have identified and shut down all systems that were participating in the attack. The nature of this specific attack makes this difficult.
We will provide regular updates on progress with resolving this outage, by updating this advice and via QRIScloud on Twitter - https://twitter.com/qriscloud.
Finally, we are confident that there has been no damage to RDSI collection data, or to data on NeCTAR VMs in QRIScloud. Indeed, most QRIScloud VMs should be working more or less normally.
If you need more information, please direct questions to <firstname.lastname@example.org>.
UPDATE 11:00am - Our operations staff have identified two systems that were participating in the DDOS attack and they have been isolated. However, we are not yet sure that there were not other systems involved.
UPDATE 11:15am - We are now confident that no other hosts were compromised. We have asked the UQ NOC to ask ARRNET to remove the network blocks.
UPDATE 1:00pm - The network blocks have been removed. We are awaiting confirmation from AARNET that there are no further problems. We anticipate that QRIScloud will rejoin the NeCTAR RC federation at 2pm.
UPDATE 2:20pm - Still waiting on confirmation from AARNET.
UPDATE 3:25pm - We have rejoined the NeCTAR RC federation.