Advice: Rolling outages for Nectar instances in QRIScloud: 2018-09-17 onwards

Follow

Starting on Monday 17th September, QRIScloud operations staff will be performing essential operating system maintenance and OpenStack upgrades on all regular compute nodes in the QRIScloud Nectar availability zone.  Completing these updates will require a reboot and approximately 1 hour of downtime for all instances on each compute node.  Given the amount and nature of the work involved, we will be performing these updates over a period of 3 weeks.

The outages will all happen during normal QCIF business hours. The notification procedure for the rolling updates will be as follows:

  1. We will send an initial notification 24 hours in advance to the Nectar tenant managers for each affected instance or group of instances that are running on the compute nodes designated to be updated the following day.
    • Please check and ensure that your instances are properly backed up, ahead of the outage. Information on how to do this can be found here.
    • If necessary, please advise your users of the impending outage. 
    • If your instances have special shutdown or restart requirements, please ensure that they are shut down ahead of time. We will not restart them automatically and leave them for you to restart.
  2. We will send a 2nd notice just prior to the start of the outage for each compute node.
  3. We will send a 3rd notice when the compute node has been updated and the instances have been restarted.

Due to the number of compute nodes and instances involved, it is impractical to produce a detailed downtime schedule ahead of time.

We apologize in advance for any inconvenience this may cause to you and your users.


PROGRESS

2018-09-17 - Compute nodes CN21 & CN22 to be done today. 

2018-09-17 - 12:50 - Done.

2018-09-17 - 13:00 - Compute nodes CN23 - CN28 and CN32 - CN36 to be done on Tuesday.

2019-09-18 - 12:15 - CN23,24 done.  CN25,26 underway.

2019-09-19 - 10:00 - All nodes scheduled for Tuesday were done.  Today we will be doing CN20, CN38, CN39, S1-CN01 - S1-CN07, QFAB-MEM1 (and possibly MEM3, GPU1 and GPU2).

2019-09-20 - 12:40 - Today we are doing CN40-CN56.  Most are done.

Have more questions? Submit a request

Comments

Powered by Zendesk