Starting on Monday 17th September, QRIScloud operations staff will be performing essential operating system maintenance and OpenStack upgrades on all regular compute nodes in the QRIScloud Nectar availability zone. Completing these updates will require a reboot and approximately 1 hour of downtime for all instances on each compute node. Given the amount and nature of the work involved, we will be performing these updates over a period of 3 weeks.
The outages will all happen during normal QCIF business hours. The notification procedure for the rolling updates will be as follows:
- We will send an initial notification 24 hours in advance to the Nectar tenant managers for each affected instance or group of instances that are running on the compute nodes designated to be updated the following day.
- Please check and ensure that your instances are properly backed up, ahead of the outage. Information on how to do this can be found here.
- If necessary, please advise your users of the impending outage.
- If your instances have special shutdown or restart requirements, please ensure that they are shut down ahead of time. We will not restart them automatically and leave them for you to restart.
- We will send a 2nd notice just prior to the start of the outage for each compute node.
- We will send a 3rd notice when the compute node has been updated and the instances have been restarted.
Due to the number of compute nodes and instances involved, it is impractical to produce a detailed downtime schedule ahead of time.
We apologize in advance for any inconvenience this may cause to you and your users.
2018-09-17 - Compute nodes CN21 & CN22 to be done today.
2018-09-17 - 12:50 - Done.
2018-09-17 - 13:00 - Compute nodes CN23 - CN28 and CN32 - CN36 to be done on Tuesday.
2019-09-18 - 12:15 - CN23,24 done. CN25,26 underway.
2019-09-19 - 10:00 - All nodes scheduled for Tuesday were done. Today we will be doing CN20, CN38, CN39, S1-CN01 - S1-CN07, QFAB-MEM1 (and possibly MEM3, GPU1 and GPU2).
2019-09-20 - 12:40 - Today we are doing CN40-CN56. Most are done.
2019-09-21 - 17:30 - On Friday, we completed CN01-CN19 and CN29-CN31. That completes the "regular" compute nodes. All that remain are the NSP (QRIScloud-HA) and QFAB nodes.
2019-09-27 - Today we attempted to upgrade the NSP nodes but ran into a hardware related issue. The remaining NSP upgrades have been postponed to 2018-10-02.
2019-10-03 - The NSP nodes have been completed.