Tips for building software on Awoonga, Flashlite & Tinaroo

Follow

Background

Unfortunately, due to inadequate resourcing levels, we (QCIF) are generally unable to meet requests from individual users to install applications.  (Feel free to ask, but the answer will typically be "No").  Instead, we recommend that users "self service" and install the applications that they need for themselves.

This page is intended to give some general guidance for people building and installing software.

General instructions

The first two steps to installing software are:

  1. Find the source code bundle for the software you want to install.
  2. Find and read the (correct) build and installation instructions.  You may be able to find them using a Google search.  If not, the source code bundle (e.g. a "tar" or "tar.gz" file) should include instructions in the top directory: look for a README file or similar.

Generally speaking, installing binaries is likely to be problematic. You will get better results by trying to build from source.

Notes: 

  1. You will not have sufficient access to install software into the standard locations; e.g. /usr/bin, /usr/lib, /usr/man and so on. If the instructions tell you to login as "root" or use the "sudo" command, they won't work for you.
  2. If the installation instructions tell you how to install as a non-privileged user (i.e. when you don't have "root" access), follow those instructions.
  3. If the instructions don't explicitly say what to do if you don't have root access, the following sections give some tips on what to do.

Where to install software to

As stated above, you do not have permission to install software into the standard Linux locations, or to the local locations that are used for sharing; e.g. "/opt", "/usr/local" or "/sw". There are good reasons for this, and no exceptions will be made.

Instead, you should use one of the following:

  • Your "home" directory (preferred)
  • Your "30days" or "90days" directories ... if you only want to use the software for a relatively short time
  • A "file share" collection.
  • Some shared space that has been set up for you.

Tips for installing packages; i.e. how to handle "apt-get", "yum", "dnf", etc

If the instructions tell you to install a package (for example a library dependency) using "apt-get", "yum" or "dnf". This will typically require root access, which you don't have.

The best / simplest option is find the source code for the packaged software, then download and build it.  Then install it into the directory where you are putting your local installs.

Alternatively, submit a support request to have the package installed. If a package is of general interest / use, and if there is a suitable version available, we may agree to install it for you.

There are other ways that a non-root user can deal with software in packages (e.g. here or here) but they are complicated, and not recommended unless you are a Linux expert.

Tips for "configure" and similar

A lot of C and C++ software uses a "configure" script to analyse the platform dependencies, and generate a Makefile.  You then use the generated Makefile to build and install the software.

If the build instructions say to run something like the following:

    cd some-package
    ./configure      # this is the key step
    make
    make install

you will run into problems in the last step.  The final "make install" step is going to attempt to install into directories where you do not have write permission.  The simple way to get "make install" to install somewhere else is to set a "prefix" when running configure.  For example, the following will install into your home directory.

    cd package
    ./configure --prefix=$HOME
    make
    make install

The prefix can be any directory that you can write to.  The "make install" step will typically create subdirectories called "bin", "lib", "man" and so on in the prefix directory.

Sometimes the instructions may tell you to use "autoconf" to create the "configure" script.  Other times, the script is called "configure.sh" rather than "configure".  In either case, the approach of adding "--prefix=..." is likely to work.  However, read the supplied build documentation.

Tips for hand-written Makefiles

If you have a C or C++ application or library that does not use a configure script, it probably has a hand-written Makefile.  A common convention in hand-written makefiles is to use the DESTDIR variable to specify the installation directory.  (The semantics are typically the same as "--prefix=..."; see above.  Here is a recipe:

  1. Open the top-level Makefile in a text editor
  2. Look to see if it contains references to "$(DESTDIR)"
  3. Make the software using "make" as per the build instruction
  4. Install the software like this: "make DESTDIR=$HOME install".  Note the addition of "DESTDIR=...".

(If you are uncertain, you may want to run the "make" commands with the "-n" option first. That should tell you what running "male: would do ... without doing it.)

Tips for Python

There are three approaches that you could use to installing python applications when you don't have root privilege.

  1. Don't install it at all. A lot of python scripts do not need to be installed.
  2. In addition to the standard system places (as configured by python), the python tool-chain will look for packages in your "user site-packages" directory.  Run "python -m site --user-site" to print its location.  The location of this directory can be changed in various ways as described in the Python Reference Manual.  If you are installing packages using "pip", add the "--user" option to tell it to install packages into your "user site-packages" directory.
  3. In theory, you could use the "virtualenv" tool to build your own personal Python installation.  In practice, we don't offer this:
    • Each Python virtual environment uses a lot of file-space.
    • By default, a Python virtual environment does not share the standard system-wide packages.  Thus, changes that the operators apply to the primary installation will not be applied to an application in a virtual environment.
    • It is also not clear if "virtualenv" would play nicely with "module load python", especially if the HPC operators set up multiple versions of the "python" module.

Generally speaking, the procedure for installing a Python application is as follows:

  1. Always load the python module first: "module load python".  The base version of Python provided by the operating system is python 2.6.  That is unsuitable for a lot of recent applications.  The "python" module provides python 2.7 and also python 3.5.
  2. Read the application's installation documentation. 
  3. Check to see if the dependencies are already installed.  Many dependencies will already be available, though some packages may be available as (or as part of) separate modules; e.g. in "pylibs", "scipy", "pyasm" and so on.
  4. If you can't find a dependent Python package, try using "pip" (with the "--user" option; see above) to install it.  If the package is known, it can download it from the standard PyPi repositories.  It should inform you of any secondary dependencies that need to be installed as well, and make a good attempt at building any native code components.

Tips for R

Tips for Matlab

File space for software

Some people run into problems with file space or file count quotas when building and installing into their HPC home directories. If this happens to you, please raise a support ticket.

If you want to share software you have built with other users, we can provide a "file share" collection to enable this.  Please raise a support ticket if you want to request one.

FAQs:

Q: Can I have root access? 

A: No you cannot.  We cannot allow that.  Besides, it is not necessary.

Q: Will you install software for me? 

A: It depends.  If the version of the package that you require is available via the Linux package manager or as a "Roll", then we will consider it.  However, doing this may not be as simple as you think, for various reasons.  We may be able to suggest an alternative.

Q: What do I do if I can't get the build to work?

A: Here are some suggestions:

  • Do you have local IT support who could help?  Or an expert in your research group?
  • Raise a QRIScloud (or RCC) support request. We may be able to advise, or arrange a consultation.
  • You may be able to get help at your local Hackyhour (UQ and Griffith ... more coming)
  • You may be able to find a solution using Google.
  • Persevere.

 

Have more questions? Submit a request

Comments

Powered by Zendesk