Reminder: QRIScloud NeCTAR spring cleaning

Follow

There are various tasks that users of the NeCTAR Research Cloud need to perform regularly to avoid problems for themselves and for other users. This article is intended to remind you of what they are.

Please register as a QRIScloud User

QCIF now requires every user of QRIScloud resources (including users of resources provided under the NeCTAR program) to register an account using the QRIScloud portal.  The procedure is simple:

  1. Visit the QRIScloud portal - https://qriscloud.org.au - in your web browser.
  2. Click on the red "Account" button.
  3. Perform an AAF login using the same AAF identity that you use to access the NeCTAR dashboard.  (As an existing NeCTAR user, we assume that you already understand this procedure.)
  4. Fill in the registration form with your contact details.
  5. Read and agree to the QRIScloud Terms and Conditions
  6. Submit the registration form.

You should then receive an automatically generated "support ticket" from the QRIScloud User Support system with some useful information on QRIScloud support resources and how to request access to other QRIScloud resources.

Releasing unused and under-used NeCTAR resources

In accordance with the QRIScloud Acceptable Use Policy, if you have NeCTAR instances, volumes and objects that you no longer require, please release them.

  • Unwanted instances should be Terminated.  If you want to be able to relaunch later, taking an instance snapshot first is advisable, though you need to be aware that a snapshot will not preserve your ephemeral file system or your instance's IP and MAC addresses.
  • Unwanted volumes and objects should be deleted.  If the data needs to be preserved, consider copying it to an off-NeCTAR storage system. (Note that NeCTAR storage is provided primarily to support computation and online services, not as a long-term data storage platform.)
  • Unwanted images and snapshots should be deleted.  In particular, if you have previously made images "public" and you are no longer prepared to maintain and support them, please withdraw them or delete them entirely.

Note that an instance in paused, suspended or shutdown/shutoff/powered off state still either uses or reserves VCPUs, memory and disk space. Such an instance will still be accruing VCPU-hours usage against your tenant.  The instance's resources are only released by terminating the instance.

Security patching

The security of your NeCTAR instances depends on you applying security patches on a regular basis.  On RHEL, CentOS or Scientific Linux systems this is done by running:

  $ sudo yum update

For Ubuntu and Debian systems, you should run:

  $ sudo apt-get update
  $ sudo apt-get upgrade

For recent Fedora systems, you should run:

  $ sudo dnf update

In all cases you will be given a list of the packages that are going to be updated / upgraded, and you will be asked for a confirmation before the packages are changed.  Note that the above commands will apply all available updates to software that you have installed via the package management system ("yum", "apt-get" or "dnf"), not just security updates.  This is advisable, though there is a small risk of a non-security update breaking some software that you.

It is also possible to configure an instance to automatically apply updates.  Please check the Linux distribution's documentation on how to set this up.

Operating System upgrades

Linux operating systems distributions have a finite support lifetime. When a distribution goes beyond its "end of life" date, the supplier ceases providing updates for security problems.  This means that normal security patching is ineffective.  Instead, you need to upgrade the operating system.

There are two possible approaches to OS upgrades:

  1. You can attempt an in-place upgrade of the operating system. The procedure for doing this varies from one distro to the next.  If you take this approach, beware that:
    • While in-place release upgrade processes are relatively reliable, recovering a NeCTAR instance from a failed upgrade would be difficult.  Therefore, you should make sure that you have a backup of everything important on the instance. (An instance snapshot is NOT a good option in this case, because extracting files from a snapshot image would not be straightforward.)
    • Depending on the distro, you may find that it is not possible to skip versions with an in-place upgrade. If you are a number of versions behind, the upgrade path could entail a number of steps.
    • If you do plan to take this approach, it is essential that you read the OS upgrade documentation carefully.
  2. A (generally) better option is to launch a new instance from an image using the newer distro, install your service and application software into the new instance, and then transfer the files over from the old instance.  The downsides are:
    • You need enough quota to be able to run the old and new instances size by side.
    • The new instance will have a new IP address and a new MAC address.
    • If you did not keep track of how you originally installed your services and applications in the old instance, building the new one may not be straightforward.
    • If you are using a 3rd-party image as your base image, you are reliant on the provider of that image (not NeCTAR) to provide a new image with an updated operating system.

As of December 5th, 2016, the following standard NeCTAR images are beyond their "end-of-life" and need to be updated.

  • All versions of CentOS prior to CentOS 5 are beyond "end-of-life".  Note also that CentOS 5 "end-of-life" is March 31, 2017, so an early update is advisable.
  • All versions of Scientific Linux prior to SL 5 are beyond "end-of-life".  Note also that SL 5 "end-of-life" is March 31, 2017, so an early update is advisable.
  • All releases of Fedora prior to Fedora 23 are beyond "end-of-life", and Fedora 23's "end-of-life" is December 22nd, 2016.
  • All releases of Ubuntu prior to Ubuntu 14.04 (LTS) and also Ubuntu 15.04 and 15.10 are beyond "end-of-life".  For Ubuntu 14.04, see below.
  • All releases of Debian prior to Debian 7 are beyond "end-of-life".

Kernel upgrades for Ubuntu 14.04

The standard NeCTAR images for Ubuntu 14.04 tracked the official Ubuntu "point releases"; i.e. 14.04.0, 14.04.1 through to 14.04.5. The 14.04.5 release became the official long term support (LTS) release. Unfortunately, during the life of Ubuntu 14.04, the suppliers (Canonical Inc) changed Linux kernel version 4 times, and in some cases a version of the kernel without long-term support was used.

Ubuntu  release Linux kernel
version number
Ubuntu kernel package End of life Update required?
14.04.0 v3.13 Trusty 14.04 GA Kernel April 2019 No (optional)
14.04.1 v3.13 Trusty 14.04 GA Kernel April 2019 No (optional)
14.04.2 v3.16 Utopic 14.10 HWE Kernel August 2016 Yes
14.04.3 v3.19 Vivid 15.04 HWE Kernel August 2016 Yes
14.04.4 v4.2 Wiley 15.10 HWE Kernel August 2016 Yes
14.04.5 (LTS) v4.4 Xenial 16.04 HWE Kernel April 2019 No

If your system is based on 14.04.2, 14.04.3 or 14.04.4, kernel patches ceased in August, and it will be open to attack based on unpatched kernel vulnerabilities.

To determine (definitively) what version of the Linux kernel your instance is using, examine the "kernel release" using the following command:

  $ uname -r
    2.6.32-21-generic

The kernel version will be the first two components of release string; e.g. "2.6" in the above.  Next, check in the table above to see if an update is required.  If an update is required, then the recommended approach is to install the HWE kernel from Ubuntu 16.04 as follows:

  $ sudo apt-get update
  $ sudo apt-get install --install-recommends linux-generic-lts-xenial

Note: It is advisable to ensure that your backups are up to date first. If you run a desktop environment on your instance, the HWE upgrade is a bit more complicated. Please refer to the Ubuntu LTS Enablement Stack page for a more detailed explanation.  The Ubuntu Kernel Support page shows the support schedule graphically.

(The table above says that it is "optional" to update the kernel for 14.04.0 and 14.04.1. The update is not necessary for security purposes, but that there may be advantages to you in doing it anyway. However, we do not advise doing it as a matter of course. There is a small risk that the kernel upgrade might break something in your instance's application stacks, or affect its ability to use special purpose hardware.)

Tenant allocation and member review

If you are a Tenant Manager, please login to the NeCTAR Dashboard and check the current end date on your allocation(s).

  • If you need to extend the project, please submit an amendment / extension request.  This can then be reviewed.
  • If you no longer require a project, please submit a request to either NeCTAR Support or QRIScloud Support to let us know.

If you intend your project to continue, then you should also review the project's tenant manager(s) and members.

  • Anyone who is no longer associated with the project should be removed:
    • Tenant Manager can remove tenant members themselves. 
    • Changes to a tenant's Tenant Manager, require a support request. Ideally the request should be submitted by a current Tenant Manager of the tenant.
  • If a user has moved to a different institution, either remove the old identity and add the new one, or get the user to submit a support request to have their identity updated (in Keystone).
  • If it would be more appropriate for someone else to be Tenant Manager, request that this be changed.

Note: It is not safe to leave old members in a project. Even if the user has lost their AAF access they (or someone else) can still potentially:

  • use the user's OpenStack password to use or interfere with the project's resources, or
  • use the user's SSH keys to login to instances.

Even if the person is still considered to be trustworthy, they may no longer have physical control of all of the computer systems on which they places copies of the passwords and keys.

To deal with the last bullet point, you need to check the "~/.ssh/authorized_keys" files on all relevant instances and remove any of the user's SSH keys that should no longer be there.  (But before you change any "authorized_keys" file, make sure that you have a "root" password set; see below.)

If the user has been using project-wide shared SSH keys, then you should treat all of those keys as compromised, and generate new key-pairs. In the future, don't share key-pairs if you can avoid it.  Ideally, everyone should have personal SSH keys.

Other House-keeping

If your NeCTAR instances, volumes and object storage contains important data that you cannot afford to lose, then you should have regular backup procedures in place.

  • If you haven't set up regular backups, we recommend that you attend to this.
  • If you do have backup procedures, we recommend that you check that they are working. The best way to do this is to attempt to build a clone of your systems from the backups.

(Note that while object storage is replicated across multiple sites, this does not protect against accidental deletion due to user actions, or buggy applications and scripts. Off-NeCTAR backup is advisable to protect against such eventualities.)

We recommend that you review the Security Groups and Access Rules for your instances. If you have ports open that should no longer be open, or if access is provided to many IP addresses or networks, consider tightening the rules. 

We recommend that you place a strong passphrase on each of your important private keys. This protects against the scenario where the computer where you store your keys is lost, stolen or hacked.  A passphrase can be set on an existing private key using your system's SSH key management tools.

We recommend that you set a password on the "root" account of all of your instances. This will allow you to login as "root" on the instance's VNC console. However, it is advisable to use a good password, and it is critical that you don't allow "root" to login over SSH.

Finally, if you (or the people who built your system for you!) have set up things so that you can login over SSH using a password, that is a major security problem. You / they should disable password-based authenticate.  If that is not practical, then you / they should restrict SSH access to IP addresses or networks where all users are relatively trustworthy.  An open SSH port with password authentication is an invitation to hackers.

 

 

 

Have more questions? Submit a request

Comments

Powered by Zendesk