QRISdata Collection Access Migration

Follow

Introduction

QCIF is currently in the process of rolling out the next generation of QRISdata (RDSI) collection access services in Brisbane. The new services will enhance collection security by providing individual accounts, providing more control for collection owners, and they will address existing performance and resource efficiency issues.

We are aiming to complete the roll-out in Q1 2016.

Personal QRISdata collection access accounts

The QRIScloud Portal provides each registered user with personal credentials (an account and password) that can be used to authenticate against QRIScloud services.  Registered users can reset their access password at any time via the Portal.  Users will have only one password to remember, rather than separate passwords for each collection they manage or use.

The collection's designated managers will be able to grant collection access to any registered QRIScloud user, and revoke that user's access (say, when a research team member leaves) without affecting other users. A collection manager will be able to do this directly, rather than having to lodge a support request to ask us to do it.

Access Methods

The old approach of using a separate collection virtual machine (VM) for each collection is being phased out, and being replaced by a more efficient service implementation model.

The following access methods will be available.  Please note that they are mutually exclusive.

Standard Access Methods

This is the access method offering that we anticipate that most people will want.  It offers the following:

  • SSH-based access methods: SCP, SFTP and RSYNC.
  • Globus GridFTP
  • Aspera Shares and Drive
  • Direct file system access on Flashlite / Euramoo.

Other access protocols are being investigated as part of the Standard Access Methods offering.  For example, we may be able to expose an entire collection for anonymous (read-only) HTTP / HTTPS or FTP access.

NFS-only Access Method

This allows a collection to be NFS read-write mounted by NeCTAR Instances in one or more designated NeCTAR Tenants.  You can implement other access methods yourself in the NeCTAR VM(s), but you will need to implement your own identity and access control schemes.

Mediaflux Access Method

Your collection will be held in a Mediaflux server running in Polaris or the JCU data centre.  Mediaflux provides rich data and metadata management facilities, and you can choose to expose your data to your users in a variety of ways.  The downside is that there is a significant learning curve.

Tier 3 (HSM) collections

QRISdata Tier 3 collections are implemented using a hierarchical storage management (HSM) system that transparently migrates files between disk and tape storage on demand.

A careful review of the current and proposed collection access methods has led us to the conclusion that we should not support direct NFS access for HSM collections.  It is likely to trigger a sequence of tape retrievals which would cause severe performance degradation and a poor user experience, e.g. long pauses and web browser timeouts.

Please refer to the Virtual Wranglers "Using DMF" page for a deeper explanation of the problems with using QRISdata HSM, and advice on how to get acceptable performance.

Change to NFS mounting of QRISdata collections to NeCTAR instances

It is currently possible to NFS mount a QRISdata collection on NeCTAR instances within a designated NeCTAR tenant. Unfortunately, NFS mounting of collections presents security difficulties because there are two independent identity management regimes involved:

  • On the collection side, access control is based on "user" and "group" identities that are represented using numeric IDs managed by the QRIScloud identity management (IDM) infrastructure.
  • On the NeCTAR instance side, identity management is entirely under the control of the people who manage each in instance.  Put simply, if you have "root" access, you can create local "users" and "groups" with any numeric IDs that you want.

When we export a QRISdata collection to a NeCTAR VM, the mapping of QRIScloud identities to instance-local identities is under the control of the VM itself. This means that anyone who has (or can gain) control of a NeCTAR VM will be able to access or update all files in a mounted collection. This happens irrespective of the access controls provided to the collection manager provided by the QRIScloud Portal.

Unfortunately, we have not been able to identify a workable solution that supports both read-write access methods with per-user access controls (using QRIScloud IDM), and NFS mounting. We have therefore decided to make NFS and other access methods an "either / or" choice:

  • If you (or your users) want to use the "standard" access services for your collection, you will not be permitted to NFS-mount the collection.
  • If you want to NFS-mount your collection, you (and your users) will not be able to access it via the "standard" access services, and the corresponding access groups will be moot.

It is worth noting that many of the access methods are easy to implement yourself within your own NeCTAR VM.  For example, you could install Apache and enable its WebDAV module, and then set up your own web accounts and access controls.

Finally, we may be able to provide read-only NFS mounts on request.  This has not been decided.

Mediaflux Collections

Mediaflux collections are now available. Mediaflux provides collection owners with new options for cataloging and describing their data assets, implementing fine-grained access control, and exposing their data using easy-to-configure web portals.

Mediaflux has other capabilities that may be made available in the future.  For example, it is capable of accessing HSM storage, and of storing collection files in an encrypted store.

No WebDAV

Unfortunately, we have not been able to identify an implementation of WebDAV that we are confident can be implemented securely in a Standard collection.  For this reason, WebDAV access will not be supported.  There are a couple of alternatives available to you:

  • You can NFS mount your collection on your own NeCTAR VM (subject to the caveats above) and implement your own WebDAV server there.
  • It might be possible to set up WebDAV access for a Mediaflux collection.

Transitional Arrangements

Every existing QRISdata collection needs to be transitioned from the existing (legacy) model (with shared RW and RO accounts) to per-user credentials.

We will first contact all collection owners / managers and ask them to register QRIScloud portal accounts, and generate QSAC credentials.  As this is done, we will associate each collection owner / manager with their respective collections' access control groups.

The next step is for the collection owner to choose between the Standard access methods, Mediaflux or NFS-only for each collection. Then we can schedule downtime to convert each collection from "legacy" to the chosen collection type:

  • For NFS-only collections, no downtime will be required.
  • For Standard collections, some downtime is required to change file system mounts, and update file permissions and ACLs.
  • For Mediaflux collections, longer downtime is required because all files in the collection will need to be physically copied by the "ingestion" process into Mediaflux.

Collection Owner Considerations

Collection owners / managers should familiarize themselves with the QRIScloud collection access management.  The user interface and procedures are described in the "Managing Collection Access" document. The document describes:

  • how to invite a user to apply for access,
  • how to grant access,
  • how to revoke access,
  • how to delegate access grant / revoke rights to other users.

Note: a person normally needs to belong to an AAF member organization (e.g. an Australian University, CSIRO, etc) before they can register a QRIScloud account. If you want to provide access to your collection to someone who does not have an AAF account, we (QCIF) can facilitate this.)

When your collection is transitioned, the old shared RW and RO credentials will cease to work.  If you have shared those credentials with other people, and you want them to continue to have access, you can send them an "invitation" URL as described in the "Managing Collection Access" document.

The decision of whether to choose Standard, Mediaflux or NFS-only access needs to be made by you, depending on how you, your colleagues and your systems use or intend to use the collections.  If you need advice, contact one of the QCIF eResearch Analysts or QRIScloud Support.

Finally, you should take time to familiarize yourself with the relevant QRIScloud Policies.

 

Have more questions? Submit a request

Comments

Powered by Zendesk