Section 3: Basic QRISdata questions

Q3.1 - What is cloud storage?

By analogy with cloud computing, cloud storage is where you store your data on a infrastructure run by someone else "on the internet".

Q3.1.1 - Is QRISdata really cloud storage?

Yes it is. The models for resourcing and allocating QRISdata storage are different from commercial cloud storage providers, but QRISdata storage qualifies as cloud storage.

Q3.1.2 - Why should I use QRISdata storage?

QRISdata storage is held in physically secure data centres in southeast Queensland, on systems that are managed by IT professionals. It provides a more reliable place to store research data than putting it on USB drives and portable hard disks.

QRISdata is designed to complement well-managed data storage provided by your institution.

Q3.1.3 - How do I get data storage?

If you are from UQ, you should typically apply for UQ RDM storage via the RDM portal.

If you are from UQ and your use-case doesn't suit RDM, or you are from JCU or USC, then

read on.

You can start the process of applying for QRISdata storage via the QRIScloud Portal's Services page. Someone from the QCIF eRA team will then contact you to discuss your needs, and prepare storage application.

Q3.1.4 - How much storage should I apply for?

You can apply for as much storage as you need: a few gigabytes to hundreds of terabytes, or even more. It is a relatively simple process to increase the allocation size, if you find that you need more space; see FAQ 3.9 on how collection usage limits are implemented.

However there are a couple of practical caveats:

Requests for large amounts of storage are subject to capacity constraints.
For really large requests, we are only able to provide HSM collections; see FAQ 3.3.4.

Please do not apply for more space then you need. Our stakeholders measure us on actual data stored, not on allocated storage. If you are allocated storage and don't use it within the agreed time-frame, we reserve the right to take it back.

Q3.1.5 - What happens after my application is submitted?

Your application will assessed to determine if your request can be fulfilled.

Your storage will then be provisioned and we will send you collection's details and links to the QRIScloud collection storage documentation.

Q3.1.6 - Can I request QRISdata storage via the NeCTAR Dashboard?

No. QRISdata and NeCTAR resources are requested, allocated and accounted using different processes and mechanisms.

Q3.2 - What is a Collection?

There is no clear official definition that says what a Collection is. However, we take it to encompass a collection of data (e.g. files) that is related to scientific / academic research activities.

Q3.3 - What kind of QRISdata storage is available?

We currently offer one type of QRISdata storage:

GPFS-based disk storage. It has HSM characteristic, with the advantage that each collection's on-disk cache size and residency policy can be controlled individually.

We should note that NeCTAR Object Storage and NeCTAR Volume Storage are covered by QRIScompute FAQs.

Q3.3.1 - What is a Storage Pool?

The storage that holds collection data is divided into Storage Pools. Each pool corresponds to a file system on one of the NFS servers.

The "GPFS" pools are for GPFS storage, they are much larger can accommodate collections of indefinite size, with the caveat that larger collections cannot be 100% on-disk.

Q3.3.3 - Why can't I have a really large on-disk collection?

We don't have enough disk space to give everyone what they would like. Not even on GPFS. Building and running a large-scale disk array costs a significant amount of money. Since QRIScloud is not operated on a "cost recovery" basis, the economics means that resources need to be rationed.

Q3.3.4 - What are the implications of using HSM?

HSM stands for Hierarchical Storage Management. With HSM, the primary copy of your files will be held on tapes, with a cache copy held on disk for faster access. The problem is that the HSM disk caches are relatively small compared to the amount of data help on tape. If you try to access a file that is no longer in the cache it has to be retrieved from tape.

As a consequence, classic HSM is not suitable if you require fast access to your files at all times

Q3.4.1 - What is the collection identifier?

Each collection has a unique identifier of the form "Qnnnn" where "nnnn" is a 4 digit number.

Q3.4.2 - What are access methods?

The access methods determine how you and your collaborators will be able to access the data in your collection. This needs to be specified when a collection is provisioned. There are currently two access methods to choose from for new collections Standard and NFS-only.

The access method also determines how per-user access control works:

For Standard collections, per-user access is implemented using access groups,
For NFS-only collections, access control is your responsibility.

Q3.4.3 - What is Standard access?

The Standard access methods allow you to read and write your data using a variety of tools. These tools include Cyberduck, Filezilla and WinSCP, and various command-line utilities.

In addition, collections with Standard access are NFS mounted on Bunya. This will allow users of these systems who are members of the appropriate access groups to be able to access the collection data via the file system.

Q3.4.4 - What is NFS-only access?

The NFS-only access method allows you to NFS mount the collection on NeCTAR instances running in the QRIScloud availability zone. You need to nominate the NeCTAR tenants that can mount the collection, and then you need to configure the mounts on each instance. Once you have done that, you can implement whatever data access services and access controls you want.

For more information, please refer to NFS access to QRIScloud Collection

Q3.4.6 - Can I change my mind about my collection's Access methods?

Yes, you can change your mind.

Switching between "Standard" and "NFS-only" is relatively straight-forward, but it does entail a collection outage.

Q3.5 - How is access to my collection controlled?

There is a document called Guide to Managing Collection Access that explains how a collection administrator can manage user access. Note that this only applies to collection configured with Standard or Mediaflux access methods.

Q3.5.1 - What are the access groups?

An access group is essentially a managed list of (specific) people with access to a (specific) collection. Each collection has two access groups: a "read-only" group and a "read-write" group. Within each group, a person can have one of three roles: "user", "administrator" or "owner".

Q3.5.2 - How do I grant someone collection access?

The procedure for "inviting" a person is described in the Guide to Managing Collection Access. This procedure works for any person with current AAF access. If you wish to "invite" people who does not have AAF access, please contact QRIScloud Support.

Q3.5.3 - How do I revoke someone's collection access?

The procedure is described in the Guide to Managing Collection Access.

Q3.5.5 - What do I do when I get an "invitation" URL?

If you were expecting to be invited, copy-and-paste the URL to your web browsers. If you were not expecting it, either ignore it (do nothing!) or reach out to the person who (apparently) sent it to you.

A valid "user" invitation URL will look like this:

    https://services.qriscloud.org.au/access/<hex-digits>/user

where <hex-digits> is a string of 32 digits and letters ('a' through 'f'). If it looks different, be suspicious.

Q3.5.6 - What happens when I load an "invitation" URL?

The first thing that happens is that you are directed the QRIScloud Portal's AAF login page:

If you belong to an AAF member organization (e.g. any Australian University), login as described in FAQ 1.5.4.
If you do not have an AAF login, you can apply to QCIF for an AAF VHO account as described in FAQ 1.5.6. Once that has been set up, start this procedure again, using the AAF VHO as your organization.

If you don't have QRIScloud account associated with your AAF identity, the next thing that happens is that you will be asked to register, and acknowledge the QRIScloud Terms and Conditions.

Finally, you will be sent to a form for requesting access to the collection. Fill it in and submit it, and your request will be sent to the collection's owners / administrators for approval.

Q3.5.7 - What happens when my request is approved?

When your request is approved, you will receive an email from the QRIScloud portal with instructions on accessing the collection.

In your MyServices page, you can see all of the collection access groups that you belong to. If you click on the link for a collection group, you will get a page that gives some details on accessing the collection.

Q3.6 - Will my collection be backed up?

No. QRISdata does not provide a conventional backup service for collection data (see Q3.6.1),

Instead of backup, we replicate the data to protect against operator errors, storage system failures and major data center catastrophes.

Q3.6.1 - How is replication different to backup?

With a backup system, you would have a reasonable expectation that we could restore your files if you accidentally deleted or overwrite them. A typical backup system guarantees to keep old versions of files for months or years, and provides mechanisms that allow the backup administrator to restore them.

In a replication system, the primary goal to keep copies of files so that we can restore to the most recent "known good" state of a collection. Restoration of files from older states may be possible, but is not the primary goal.

Q3.6.2 - How does replication work?

For HSM collections, the replicas are created by the HSM system itself. The normal replication policy is to create two tape replicas of each file in the tape store in the Polaris data centre, and a third replica in the tape store in the St Lucia data centre.

Q3.6.3 - How long will it be before my data is replicated?

The design goals for QRISdata collection replication state that the first on-site tape replica should be completed within 24 hours, and that the off-site replica should be completed within 48 hours.

Q3.6.4 - How long are the replicas retained?

The design goals for QRISdata collection replication state that on-site replica of a file should be retained for 4 weeks after deletion, and the off-site replica should be retained for 12 weeks. (Currently we are retaining data for 6 months, though this is subject to change.)

Q3.6.5 - Why doesn't QRIScloud implement backup for me?

Implementing the kind of backup system that a typical user desires is extremely expensive at the scale that QRISdata operates; i.e. multiple petabytes of data and hundreds of million files. The bottom line is that if we implemented "time machine" style backup and restore for QRISdata, we could probably only afford to store 10th of the data that we currently store.

Q3.7 - How can I access data in a collection?

Collection data can be accessed in the following ways, depending on the collection's access method:

For Standard access:

Using the "data.qriscloud.org.au" access system; see Q3.7.1.
Using SSH based protocols such as "scp", "rsync" and "sftp" via the above system; see Q3.7.1.
Via auto-provisioned file-system mounts on Bunya.
Via campus Medici caches.

For NFS-only access; see Q3.7.4

Via an NFS mount on a NeCTAR VM
Using data access services that you set up on such a VM

Q3.7.1 - How do I access data via SSH-based data access services?

The "data.qriscloud.org.au" system: this machine allows you to read and write files using SSH and SSH-based file transfer protocols.

You can login to the "data" using an SSH client such as the "ssh" command on Mac OSX and Linux, or "putty" on Windows:
- Use your QSAC to login, or your UQ credentials if the account names match: see Q1.2.6, Q1.2.7 & Q1.2.8.
- Once logged in, you will have a standard Linux command environment, similar to what you have when you connect to a NeCTAR VM and typical HPC systems.
- The files for each collection are auto-mounted as "Qnnnn" directories beneath the "/data" directory. Access is restricted to users who are members of the respective collection access groups.
You can use a desktop file transfer command or tool to copy files between the your desktop and the collection via the access machines:
- On Windows you can use Cyberduck, Filezilla, WinSCP among others.
- On Mac OSX or Linux you can use Cyberduck (Mac OSX only), Filezilla and command line tools such "scp", "rsync" or "sftp".
- The path to your collection will be as above; i.e. "/data/Qnnnn", where "Qnnnn" is your collection's identifier.
If you have an account on Bunya that mounts the collections (i.e. Euramoo, Flashlite and Tinaroo) you can access the files via the "/QRISdata" directory.

Note: if you are transferring large numbers of small files, you will typically get better performance using "rsync" rather than "scp" or "sftp". The latter need to create a separate SSH-enabled TCP connection for each file transferred. When the files are small, the connection overhead is large compared to the time taken to transfer the bytes.

Q3.7.5 - How do I get NFS access to my data?

The first step is that the collection manager needs to lodge a QRIScloud support request to "export" the collection to a specified NeCTAR project. Once that has been done, the NFS access to QRIScloud Collections document explains how to set up an NFS mount on a NeCTAR instance.

You are free to install and use other software on your NeCTAR instance, and use that to manage your collection data. However, the onus will be on you to manage security and access control.

Note: NFS access is only available for "NFS-only" collections. NFS-only and Standard Access are mutually exclusive, and NFS-only collections are NOT auto-mounted on the HPC / HTC systems.

Q3.7.6 - Is NFS access secure?

That is a complicated question.

On the one hand, exporting your collection does not directly expose it to the internet. The exported collection is only directly accessible via a private IP address, and it should be impossible for anything outside of the Polaris data centre to access it.
On the other hand, once your collection has been attached to an instance in your NeCTAR project, anyone who has (or can gain) privileged access to the instance has unfettered access to that data.

Thus, while providing NFS access to your collection is not insecure per se, it is definitely increasing the risk to your data.

Q3.8 - How do I get data into my collection?

The normal method for "ingesting" data into a collection is to go to the system where the data currently lives and "upload" it to the collection using one of the supported access methods; see Q3.7.

If your data is held on removable media (e.g. external hard drives, memory sticks, optical discs) you will need to plug them in one at a time.

Q3.8.1 - How do I ingest huge amounts of data?

If you have really large amounts of data to ingest, then conventional upload is liable to be problematic. Transferring terabytes of data over the network takes a long time, especially if your local networking is slow. Some of the alternatives that we can try include:

Running transfers as background processes.
Optimize transfer patterns; e.g. instead of downloading files from somewhere to your laptop or PC and then uploading them, transfer them directly.

If have lots of really small files to ingest / upload then you are going to be in for a hard time, no matter what approach you take. We strongly advise you NOT to do this. Instead, we recommend that you use a utility like "zip" or "tar" to bundle up the files into larger "archive" files before you upload them. If you need to compute against the little files, there are two approaches:

Copy or download the ZIP / TAR archive file to a local file system on the machine where you are doing the computation, unzip / untar the bundle into a local directory (i.e. not NFS mounted!) and compute against that tree.
Modify your application so that it can open and use the archive file directly. (There are standard runtime libraries for doing this in most mainstream programming languages.

Q3.8.3 - Can you help me with ingestion?

If you need advice or assistance with ingesting your data, please contact QRIScloud support.

Q3.9 - Is my QRISdata usage controlled?

Q3.9.1 - How are disk quotas implemented?

A different approach is used for GPFS. There are quotas on the total storage usage, but the more critical issue is how much cache disk space your collection uses:

For GPFS collections, each collection has its own cache, and the dimensions of that cache can be controlled on a per collection basis. We will adjust collection GPFS cache sizes up and down according to your access patterns, and general demand for disk space. Work is underway to move away for this to a system where disk cache is shared between collections.

Q3.10 - How do I share my data with other people?

If your collection is configured with "Standard" access, then you can grant read-only or read-write access to your collection to anyone who has a QRIScloud account. (Anyone with AAF access can get a QRIScloud account.)
If your collection is configured as "NFS only", it is up to you implement your own collection access controls.

Q3.10.1 - Does a collection have a DOI or other permanent identifier?

No. Usually your institution's library manages Digital Object Identifiers (DOI) and other identifiers for publications and data. Please contact your library for institution-specific questions.

Q3.10.2 - Can I make my QRIScloud data open access?

If your collection is NFS-only, then you are free to expose the data as you see fit. However, you need to be mindful of data security and privacy concerns.
It is possible to make a collection available for public consumption via data.qld.edu,au. In this situation the entire collection is readable. If you only want to publish part of your collection, you need to request another collection, copy the data that you wish to make public to it, and then request that other collection is published via data.qld.edu.au.

Q3.10.3 - Can I provide access to different files to different people?

This is not something that is currently supported. There are some other options:

You could NFS mount your collection on a NeCTAR instance, then implement a portal that does fine-grained access control.
It is technically possible to split a collection into sub-collections with separate read-write and read-only access groups. However, this is results in overhead for QCIF operational and administrative staff.

Q3.10.4 - Can I change my collection's metadata?

Your collection's metadata includes things such as:

The collection's title and description
The collection's custodian, requester and technical contacts.
The collection's FoR codes.
The organization and organizational unit that the collection belongs to.
The URL of a public portal for the collection.
The URL of a public metadata record for the collection.

If you need changes to be made to the metadata for your collection, please submit a QRIScloud support request.

Q3.11 - How do collection file and directory permissions work?

Collections are stored as files and directories on a POSIX compliant file system. This that the low-level access control mechanisms for files and directories in a collection are based on POSIX users and groups, POSIX file permissions and the POSIX ACL (access control list).

By contrast, the QRIScloud collection access model is based on each collection having one group of users with read-write (RW) access to the collection, and a second group of users who have read-only (RO) access. Ideally, the person who created the file is not supposed to have more access than anyone else.

The QRIScloud access model is implemented as follows:

The RW and RO groups are implemented as POSIX groups. Thus collection Q0042 has POSIX groups called Q0042RW and Q0042RO.
Each collection user maps to a distinct POSIX user.
There is a distinct collection user identity for each collection (e.g. Q0042) which is used when the actual user identities is not available. For example, files and directories ingested using Aspera are owned by the collection's user identity.
Each file or directory should have "rwx" as its owner and group access settings, and "---" for "other" access.
Each file or directory should have the collection's RW group as its POSIX group.
All directories should have the sticky "inherit group" access bit set, so that newly created subdirectories are also owned by the RW group.
All files and directories should inherit ACLs that:

Grant "r-x" access to members of the collection's RO group.
Provide default group access and umask settings.
Forbid access to "other" users.

Q3.11.1 - Are there any permissions gotchas?

Unfortunately, there are:

The person who creates a file or directory will be the POSIX owner of the object. That means they will be able to change the access bits and the ACLs.
If a file or directory (somehow) has access other than "rwxrwx---" / "drwxrws---", or has its POSIX group set incorrectly, or has its ACLS set incorrectly, it could prevent users in the RW or RO from doing what they should be able to do. These problems can spread to newly created subdirectories of a "broken" parent directory.
For files and directories in GPFS, problems can arise if user and group identities or ACLS are not mapped consistently by all caches.

Q3.11.2 - How do I fix incorrect permissions in my collection?

If the owners, permissions or ACLs on a file or directory are incorrect, you have two options:

If the file system recognizes you the owner of the file or directory (according to the "ls -l" command) you may be able to use "chgrp" to fix incorrect groups, "chmod" to correct permissions, and "setfacl" command to add missing ACLs.
If you are not the owner then you should raise a support ticket.

It is also worth noting is that fixing a collection takes time in proportion to the number of files. This is another case where having lots of small files causes pain.

Q3.12 - What is a collection migration?

Sometimes QRIScloud operations staff need to move a collection from one physical storage medium to another. Unfortunately, our infrastructure does not allow this to be done without an outage. It is also sometimes necessary to make access control changes that impact on your ability to use the collection.

If we need to migrate your collection, we will contact you and provide you with a detailed description of the migration procedure. There are some points in the procedure where we need to coordinate with you (the collection owner) via a support ticket. We would ask you to respond to migration tickets promptly.

Q3.13 - Are there other things that I need to know to use my collection?

We have written a "QRIScloud Collections Dos and Don'ts" document with recommendations on how to use QRISData collections safely, and without causing operational problems.

Comments