Unfortunately, due to inadequate resourcing levels, RCC are generally unable to meet requests from individual users to install applications. (Feel free to ask, but the answer will typically be "No"). Instead, we recommend that users try to "self service" and install the applications that they need for themselves.
This page is intended to give some general guidance for people building and installing software.
(Some of the links below refer to RCC pages that may be restricted to UQ users only. Non-UQ users needing access should submit a request to <firstname.lastname@example.org> requesting a copy.)
The first two steps to installing software are:
- Find the source code bundle for the software you want to install.
- Find and read the (correct) build and installation instructions. You may be able to find them using a Google search. If not, the source code bundle (e.g. a "tar" or "tar.gz" file) should include instructions in the top directory: look for a README file or similar.
Generally speaking, installing binaries is likely to be problematic. You will get better results by trying to build from source.
- You will not have sufficient access to install software into the standard locations; e.g. /usr/bin, /usr/lib, /usr/man and so on. If the instructions tell you to login as "root" or use the "sudo" command, they won't work for you.
- If the installation instructions tell you how to install as a non-privileged user (i.e. when you don't have "root" access), follow those instructions.
- If the instructions don't explicitly say what to do if you don't have root access, the following sections give some tips on what to do.
Where to install software to
As stated above, you do not have permission to install software into the standard Linux locations, or to the local locations that are used for sharing; e.g. "/opt", "/usr/local" or "/sw". There are good reasons for this, and no exceptions will be made.
Instead, you should use one of the following:
- Your "home" directory (preferred)
- Your "30days" or "90days" directories ... if you only want to use the software for a relatively short time
- A "file share" collection.
- Some shared space that has been set up for you.
Tips for installing packages; i.e. how to handle "apt-get", "yum", "dnf", etc
If the instructions tell you to install a package (for example a library dependency) using "apt-get", "yum" or "dnf". This will typically require root access, which you don't have. (And which won't be granted ... even if you ask.)
The best / simplest option is find the source code for the packaged software, then download and build it. Then install it into the directory where you are putting your local installs.
Alternatively, submit a support request to have the package installed. If a package is of general interest / use, and if there is a suitable version available, RCC may agree to install it for you.
Tips for "configure" and similar
A lot of C and C++ software uses a "configure" script to analyse the platform dependencies, and generate a Makefile. You then use the generated Makefile to build and install the software.
If the build instructions say to run something like the following:
cd some-package ./configure # this is the key step make make install
you will run into problems in the last step. The final "make install" step is going to attempt to install into directories where you do not have write permission. The simple way to get "make install" to install somewhere else is to set a "prefix" when running configure. For example, the following will install into your home directory.
cd package ./configure --prefix=$HOME make make install
The prefix can be any directory that you can write to. The "make install" step will typically create subdirectories called "bin", "lib", "man" and so on in the prefix directory.
Sometimes the instructions may tell you to use "autoconf" to create the "configure" script. Other times, the script is called "configure.sh" rather than "configure". In either case, the approach of adding "--prefix=..." is likely to work. However, read the supplied build documentation.
Tips for hand-written Makefiles
If you have a C or C++ application or library that does not use a configure script, it probably has a hand-written Makefile. A common convention in hand-written makefiles is to use the DESTDIR variable to specify the installation directory. (The semantics are typically the same as "--prefix=..."; see above. Here is a recipe:
- Open the top-level Makefile in a text editor
- Look to see if it contains references to "$(DESTDIR)"
- Make the software using "make" as per the build instruction
- Install the software like this: "make DESTDIR=$HOME install". Note the addition of "DESTDIR=...".
(If you are uncertain, you may want to run the "make" commands with the "-n" option first. That should tell you what running "make" would do ... without doing it.)
Tips for Python
There are four approaches that you could use to installing python applications when you don't have root privilege.
- Don't install it at all. A lot of python scripts do not need to be installed.
- In addition to the standard system places (as configured by python), the python tool-chain will look for packages in your "user site-packages" directory. Run "python -m site --user-site" to print its location. The location of this directory can be changed in various ways as described in the Python Reference Manual. If you are installing packages using "pip", add the "--user" option to tell it to install packages into your "user site-packages" directory.
- Use Anaconda to install and manage the python applications that you need. See the (UQ only) RCC Anaconda page.
- In theory, you could use the "virtualenv" tool to build your own personal Python installation. In practice, we don't offer this:
- Each Python virtual environment uses a lot of file-space.
- By default, a Python virtual environment does not share the standard system-wide packages. Thus, changes that the operators apply to the primary installation will not be applied to an application in a virtual environment.
- It is also not clear if "virtualenv" would play nicely with "module load python", especially if the HPC operators set up multiple versions of the "python" module.
Generally speaking (excluding Anaconda!), the procedure for installing a Python application is as follows:
- Load the python module first: "module load python". The "python" modules provides python 2.7 and also one or more python 3.x versions. The versions of python available through the "python" modules are likely to be newer than the versions installed in the base OS.
- Fully and carefully read the application's installation documentation.
- Check to see if the dependencies are already installed. Many dependencies will already be available, though some packages may be available as (or as part of) separate modules; e.g. in "pylibs", "scipy", "pyasm" and so on.
- If you can't find a dependent Python package, try using "pip" (with the "--user" option; see above) to install it. If the package is known, it can download it from the standard PyPi repositories. It should inform you of any secondary dependencies that need to be installed as well, and make a good attempt at building any native code components.
- If your python code needs a native shared library that is available as a host OS package, threat this as per "Tips for installing packages" above. If you request installation of a package for a shared library, be sure to identify whether you need the standard package or the "devel" version which typically provides the C / C++ include files.
Tips for R
The R ecosystem has a powerful but easy to use framework for installing code into your R environment as packages. The short version is:
- You can install packages from CRAN with
- You use them in R with
- You get help on them with
help(package = "x").
- You can find the documentation via the web; e.g. https://rdrr.io/.
There is a comprehensive eBook on R Packages by Hadley Wickham.
Also, check the "/opt/R/local/lib" for R library packages that have already been installed.
Finally, check the RCC page about Personal R Libraries.
Tips for Matlab
File space for software
Some people run into problems with file space or file count quotas when building and installing into their HPC home directories. If this happens to you, please raise an RCC support request.
If you want to share software you have built with other users, please raise an RCC support request asking for a group area and group file storage quotas to be
Q: Can I have root access?
A: No you cannot. We cannot allow that. Besides, it is not necessary.
Q: Will you install software for me?
A: It depends. If the version of the package that you require is available via the Linux package manager or as a "Roll", then we will consider it. However, doing this may not be as simple as you think, for various reasons. We may be able to suggest an alternative.
Q: Can I use a Docker container on HPC?
A: No you cannot. Docker is not installed for security reasons. The HPC systems provide Singularity as an alternative; see the RCC Containers User Guide.
Q: What do I do if I can't get the build to work?
A: Here are some suggestions:
- Do you have local IT support who could help? Or an expert in your research group?
- Raise an RCC support request. They may be able to advise, or arrange a consultation.
- You may be able to get help at your local Hackyhour (UQ and Griffith ... more coming)
- You may be able to find a solution using Google.