On this system, users can send their processes to be computed where each lab has queue size according to their necessity.

Technical specifications:

• sNow! 1.0.3
• marvin.s.upf.edu frontend and login node.
• 27 compute nodes with 720 cores an 7.4 TB of RAM.
• NFS server node with 180TB.
• All nodes connect to IBM Spectrum Scale with Infiniband (56Gbs).
• 660 TB Parallel FIle System (IBM Spectrum Scale - GPFS)

Index

0. Brief cluster description

1. Getting an Account

2. Connection

2.1 Connecting outside the PRBB

3. Directory structure & Quota

4. Software

     5.1 Running Programs

     5.2 Submitting to the queue

     5.3 queue information

     5.4 Watching over your jobs

     5.5 Job output

6. Job arrays

               GNU Parallel: An alternative to job arrays

7. Interactive Jobs

8. Paralel Jobs

     8.1 Using a bash script and using the sbatch command

     8.2 Using the srun command directly

9. GPU Resource

10. Advice and Good clustering practices

11. MATLAB

12. Priorities

13. Inspiration

 

0. Brief cluster description

MARVIN has:

- 12 computing nodes with 16 cores and 128 GB RAM.

- 13 computing node with 32 cores and 256 GB RAM.

- 1 computing node with 48 cores and 512 GB RAM.

- 1 computing node with 64 cores and 512 GB RAM.  

 

Our system has a total of 27 computing nodes, which all in all sum 720 cores. And the disk space available for MARVIN and the nodes are approximately 7.4 TB. The Global Disk Status is divided into three main groups:

 

homes: 20 TB

scratch: 283 TB

projects: 357 TB

 

All of them IBM Spectrum Scale. Thus meaning high performance access. The OS running in the computing nodes is CentOS 7.x x86_64

 

1. Getting an account

In order to get an account, send an email containing your full name from the academic email account where you want to receive any cluster notification to [email protected].

2. Connection

The first thing to do is connect to the cluster. Depending what O.S. you are using it will be different.

Windows:

Install Putty (https://www.chiark.greenend.org.uk/~sgtatham/putty/download.html) Type in the “Host name”: [email protected] type your password and login.

Linux / MacOS:

Open a terminal and type: ssh [email protected]

 

If you want to use programs which have a GUI (Graphical User Interface) you will need to do the following:

Windows:

Install Xming server or any other X-Window server (you will probably need to install a fonts package), run it and when you run putty, go to the left options tree-menu and check: Connection – SSH – X11 – Enable X11 forwarding.

Linux / MacOS:

Open a terminal and type: ssh [email protected] -X (its an upper-case X, lower case x does the oposite)

2.1 Connecting outside the PRBB

The cluster front-end “mr-login” doesn’t have direct access to the internet, so in order to connect to marvin from OUTSIDE the PRBB you’ll have to install the UPF’s VPN client. You can get it here: https://www.upf.edu/bibtic/serveis/xarxes/vpn/

3. Directory structure & quota

Inside the cluster there are three main filesets:

 

 

4. Software

By default all of the installed software is not directly available at the command prompt. There are a large number of software applications, libraries, tools, and compilers installed on Marvin. Some of the installations require environment settings that are incompatible with other installations. The Marvin-Cluster ‘Environment Modules’ are used to activate (load) software required by the user. The module commands allow the user to load/unload software, and the modules packages modify the user environment variables necessary to run the software. The modules package is used to customize user environment settings. It allows the user to control what versions of a software package will be used when the user compiles or runs a program in their current login session or batch job.

Software Available For a current list of all software available enter the command “module avail” at the command prompt. Software Install Request To submit a request to have software installed on the cluster send us an e-mail to [email protected] with the subject “SOFTWARE INSTALL REQUEST” with the software name, link to download, and any concern you have about it.

Summary of Basic Module Commands

For a current list of all available modules (software) installed use the command:

 module avail

Basic help on each module:

module help

Display details for each module:

module display

For additional information:

man module

 

Intel Compilations

By default the software you can see with modules are compiled with GCC toolchains. If you wish to compile your software with the Intel compiler for a better performance please add this requirement when you send the email for your install request.

 

To show the software compilated with intel type the folowing:

module load modulepath/arch

module av

 

To go to the previous modules, loaded with the toolchains which contain GCC type:

module load modulepath/noarch

 

When you change from no arch to arch modules or conversely, the loaded modules will remain inactive.

5.1 Running Programs

Once you are connected to marvin, please, DO NOT RUN ANYTHING THERE. No really, this is a cluster, and you are suposed to run things here, but “here” it is not “the cluster” yet. Now you are connected to “mr-login”, which is the “front end” of the cluster. This means that this computer is only a shared “entry point” to the whole cluster. It is shared by all the users and all the proceses, so, if you run things here, you will go slower, and you’ll slow other people too (and we can stop without warning anything we find running this way). If you want to launch anything, you have to submit it to a queue, as explained below.

 

5.2 Submitting a job to the queue system

To use the queue system all you need to do si:

sbatch scriptname

 

Where scriptname is a script that contains whatever you need in order to run the programs you want to.

 

Example:

 

Let’s say we’re already connected to marvin:

 

 

In our case, we just want to check the date, sleep and re-check the date:

Now let’s send it:

 

Notice that the squeue command has given us info about our job

As we’ve tell the queue system that we want to receive a mail at start, abortion or end of execution, once we receive the mail we can check the results:

  

man sbatch, man squeue and man sinfo will give you more information about how to use those commands to submit your jobs and monitor them.

5.3 Queue information

5.4 Watching over your jobs

           Useful commands of the queue system:

 

squeue: Shows your currently waiting or running queued jobs.

 

sinfo: view information about Slurm nodes and partitions

 

sacct -j jobid: Shows information about already finished jobs

 

scancel jobid: Deletes from the queue a job. You can delete only your own jobs

 

smem: Shows information about the cluster occupacy.

https://www.upf.edu/web/sct-sit/software

https://github.com/situpf/smem

 

sit-web.upf.edu/sitservices: Web interface where you can see the status of the cluster.

 

Examples of useful commands

 

 

5.5 Job output

 

By default standard error and standard output of your job will go to the same file, called slurm-%j.out , where %j is the JOBID number of your job.

 

standard error contains all the messages that your job has sent to the error channel, i.e. “command not found”, “no such file or directory”…

 

standard output contains all the messages that your job has sent to the standard output messages, i.e. anything you see written.

 

It is highly advisable that you control where standard error and standard output go. You can do it using sbatch flags -e for standard error and -o for standard output.

 

6. Job arrays

 

SLURM allows you to submit a number of "near identical" jobs simultaneously in the form of a job array. To take advantage of this, you will need a set of jobs that differ only by an "index" of some kind.

For example, say that you would like to run tophat, a splice-aware transcript-to-genome mapping tool, on 30 separate transcript files named trans1.fq, trans2.fq, trans3.fq, etc.

 

First, construct a SLURM batch script, called tophat.sh, using special SLURM job array variables:

 

tophat.sh

 

Then launch the batch process using the --array option to specify the indexes.

 

 

In the script, two types of substitution variables are available when running job arrays. The first, %A and %a, represent the job ID and the job array index, respectively. These can be used in the sbatch parameters to generate unique names. The second, SLURM_ARRAY_TASK_ID, is a bash environment variable that contains the current array index and can be used in the script itself. In this example, 30 jobs will be submitted each with a different input file and different standard error and standard out files.

GNU Parallel: An alternative to job arrays

 

There is another way to submit a large number of jobs in parallel which files’ names differs in more than one number (like 1-30 in the example below). With GNU parallel, the user should check how many CPUs is able to use (replace lab_sit with your lab name):

 

 

In this example, we want to run STAR to map 84 samples (paired-end FastQ = 168 files) to a reference, and we created this script:

 

run_all_in_nodes.sh

 

Each job (or task) uses 8 threads (or CPUs), so we should calculate how many tasks we can ask to run the maximum number of jobs simultaneously (in this example, c=cpus-per-task=8)

 

 

Pay attention to the maximum CPUs per user that we saw before!!! We are going  to use  29x8=232 CPUs (<450 CPUs available per user).

Now, we prepare the script:

 

parallel.sbatch

 

And finally, run multiple jobs that can be controlled by the same job ID:

 

For more information about the GNU parallel options and advantages, please visit https://www.upf.edu/web/sct-sit/gnu-parallel-tutorial

7. Interactive jobs

 

Although Slurm has a command designed to run interactive jobs (salloc), we strongly recommend to use a tool named ‘interactive’ that is available in Marvin. Exist several ways to run interactive jobs (normaly are different combinatios between the salloc and the srun command) but the use for the ‘interactive’ tool help to the final user to do it in the correct way:

 

 

Examples:

 

1.- Run an interactive job with a resource reservation of 4 nodes with 8 cores and 4Gb of RAM for each core.

 

[msanchez@mr-login ~]$ interactive -N 4 -T 8 -m 4 -e [email protected] -J test-interactive-job

 

2.- Run an interactive jobs with 32 cores (we don’t mind the number of cores and how many cores we have in each node). We also ask for 8 GB in each core:

[msanchez@mr-login ~]$ interactive -N 4 -n 32 -m 8 -e [email protected] -J test-interactive-job

8. Parallel jobs

The most important issue that we have to take in account when we want to run a parallel job in slurm is to use always the ‘srun’ command.

 

We have two ways to run a parallel job:

9. GPU Resource

Marvin offers as an extra resource the possibility of use GPU to run jobs. To do that first we need to have a program previously compiled with CUDA (nVidia GPU compilers):

 

At the moment, we only have one computer with 4 GPU (mr-00-12).

 

To ask for a GPU resource we have to add this flag to our job script:

 

#SBATCH --gres=gpu:2

 

Below we have a completed job script that asks for 2 GPUs.

 

 

10. Advice and Good clustering practices

• Keep your home directory as clean as possible. An ideal home directory would be clean of files. It would only have there a directory for each of your projects. Once done that, whenever you submit a job,     use -e/-o options in order to send those files to the directory of the Project they belong.

 

• Don't use the projects filesystem to work with files that require a very intense use of the disk, this means to create and delete thousands of small files, pipelines on large files, etc. For this use you should use the scratch filesystem. The scratch filesystem is more efficient for this kind of operations. If you have your files in projects, copy them to scratch, work with them in scratch, and when you have the result, copy the result into the projects. If you use projects filesystem for disk intensive operations, commands as ls, ll, rm, bash tab completion as well as your jobs can take a lot of time.

 

• Use sbatch for anything that it is not editing, moving or zipping files. But if the files are pretty big, then sbatch also the zips.

 

• Test your jobs before launching them. A programm could work in your computer or in another cluster and fail here. Maybe here we need to install a perl module, a library or maybe you got a typo in the input file path. Submiting small test executions is fast and really useful.

 

• You should have a roughly idea of the amount of resources your job needs: time, memory, hard disk space.If not, you're increasing the risk of a job failure due to walltime, too much swapping or lack of disk space. Which will be a waste of time for you and anyone else using the cluster.

 

• Use sbtach. Once you’re in marvin you are in the login node, this is not the cluster, it is only the front end and queue manager computer. If you run heavy things here the computer will go slower not only to you, but to anyone else using it.

 

• Use  the “--mem=XXX” option. It is needed by the scheduler system in order to know how much memory will need your job.

 

• Use a reasonable amount of memory. Any extra memory you are requesting, is memory that other users (or even you) will not be able to use, and make everything slower (jobs not running because, even though there are free cpu slots, there is not enough free memory).

 

• We do have more than 700 CPUs. That mean we can run 700 jobs     simultaneously. Yeah, it’s not Marenostrum but… use it!!! I mean, if you have to run a process that will last 20 hours, but you manage it to become 20 processes of 1 hour each… It will work 20 times faster!!! Sometimes it is not possible to break it into pieces     that way, but much others it is.

 

• But keep in mind: N cores => No more than 30xN simultaneous sbatch jobs. as mentioned, it is ok to split a 200h job in 20 x 10h jobs, or even in 100 x 2h jobs, but it doesn’t make sense to try to split in 20000 jobs.

 

• It doesn't make sense to send jobs shorter than 1'. Because even when a queue is empty it takes some seconds  for a job to start. And when it's finished it takes again some time to leave the queue, inform the master, write accounting logs, and so on. Therefore, if your job takes only a few seconds of computation, it spends most of the time geting in and out of the queue, instead of computing.

 

• As an extension of the previous point, if you want to run an installed program, and that program has a parallel version, let us known. They tend to be much faster and efficient.

 

• If you need to run several similar jobs, here is an easy way to do it, making a script that sends jobs with different parameters, but try to avoid neverending self-calling loops. They are pretty nasty to kill :S

 

11. MATLAB

 

In order to use the Matlab licenses into the cluster, the Slurm queue system provides resources to use this license:

• To show the current status of licenses:

scontrol show lic

• If you want to use 2 licenses as resource:

sbatch -L matlab:2 <job-script>.sh

      or inside the bash script:

#SBATCH -L matlab:2

 

12. Priorities

The calculation of the priorities is the same for all the users and labs in Marvin.

 

Marvin use the standard way of Slurm to decide which priority is applied to each job. The priority is based in a arithmetic calculation that depend of six factors.

 

The six factors that influence in the job priority are:

 

Age

The length of time a job has been waiting in the queue, eligible to be scheduled. A job reaches the max value for this factor after waiting in the queue for 1 weeks.

 

Fair-share
The amount of resources that has been consumed in the previous 28 days.

 

Job size
Job time limit. if you know how many time is going to take your job, you can add this info to your slurm script using the ‘--time’ directive to help to gain more priority, specially if you know that your job is going to take a few hours or even minutes (in addition, you also have to take in a account that after this time, if your job is still alive, it will be killed by Slurm).

 

Partition
A factor associated with each node partition. As in Marvin there are only on partition this value is not relevant because all jobs are going to have the same value in this priority factor.

 

QOS
A factor associated with each Quality Of Service. In Marvin the QOS doesn’t have any priority factor associated so this factor is not relevant in the priority calculation.
 

TRES
Each TRES Type has it's own factor for a job which represents the number of requested/allocated TRES Type in a given partition. Actually Marvin is using the default configuration for this priority factor.

  

To check the value of each factor for each job that is pending in the queue, we can use the slurm commands ‘sprio -j --long’ and ‘sshare’.

13. Inspiration

Marvin is the ship's robot aboard the starship Heart of Gold. Originally built as one of many failed prototypes of Sirius Cybernetics Corporation's GPP (Genuine People Personalities) technology, Marvin is afflicted with severe depression and boredom, in part because he has a "brain the size of a planet" which he is seldom, if ever, given the chance to use. Indeed, the true horror of Marvin's existence is that no task he could be given would occupy even the tiniest fraction of his vast intellect. Marvin claims he is 50,000 times more intelligent than a human, (or 30 billion times more intelligent than a live mattress) though this is, if anything, an underestimation. When kidnapped by the bellicose Krikkit robots and tied to the interfaces of their intelligent war computer, Marvin simultaneously manages to plan the entire planet's military strategy, solve "all of the major mathematical, physical, chemical, biological, sociological, philosophical, etymological, meteorological and psychological problems of the Universe except his own, three times over", and compose a number of lullabies. 

https://en.wikipedia.org/wiki/Marvin_the_Paranoid_Android