How to use (FAQ)

Once you know how to submit jobs to the cluster using Slurm, the trickiest thing to work out is how to get your science done as effectively as possible. This isn’t a trivial task – firstly you need to know what your jobs are doing so that you can understand where the bottlenecks are and then you need to figure out how to eliminate those bottlenecks. Thankfully, we provide a simple, powerful tool to help you do just that. 

All jobs on our cluster are automatically run through a lightweight job profiler. Job profiling is a technique in which a small helper program tracks how much time, CPU and memory your job uses and provides statistics when it completes. You’ll find these statistics automatically added to the end of your job output. Using this, you can submit a single job with your best guesses, look at the output, and then have a reasonable idea of what to use for subsequent jobs of a similar type.

Slurm job profiling - an introductory guide

Once you understand the output, you can move on to optimising your jobs for time, CPU and memory. This is something that varies depending on the programs you’re using and the data they’re processing, so you’ll probably end up being your own expert in the long run. At the same time, we've been able to put together a set of general guidelines (see: Slurm top tips) that you can apply to get your work done as speedily as possible whilst also making sure that you don't accidentally ask for resources that don't get used. These should make a good starting point.

BASIC USAGE

If you just want to get up and running with our curated set of commonly used bioinformatics packages, you can do so with a single command:
$ module load python-cbrg
or
$ module load R-cbrg

Note that the Spyder IDE is included in the standard Python installations, and that R-Studio is a separate module.
To see all modules available for use, use the command:
$ module avail

ADVANCED USAGE

The setup uses the following system:

• python-base and R-base contain fixed, unchanging installations of the base languages. This is for safety – they cannot be accidentally overwritten causing unexpected changes of behaviour.
• python-cbrg and R-cbrg contain separate package and library repositories for each version of Python and R. Because packages and library versions also change over time, we take a snapshot of the state on a monthly basis and then lock this to prevent changes causing unexpected behaviour. A single current version for each provides a continual rolling ‘head’ where changes are applied.

Loading the python-cbrg or R-cbrg module will automatically pull in the latest stable base and all packages or libraries:
$ module load python-cbrg
Loading python-cbrg/current
  Loading requirement: python-base/3.8.3

$ module list
Currently Loaded Modulefiles:
  1) python-base/3.8.3(default) 2) python-cbrg/current(default)

However, if you want to use a different version of the base, you can do that by loading it manually first:
$ module load python-base/3.6.10
$ module load python-cbrg

$ module list
Currently Loaded Modulefiles:
   1) python-base/3.6.10 2) python-cbrg/current(default)