🚩IMPORTANT
When you login into Marvin you are in mr-login which is the landing node. Once you are connected to marvin, please, DO NOT RUN ANYTHING THERE. No really, this is a cluster, and you are suposed to run things here, but “here” it is not “the cluster” yet. Now you are connected to “mr-login”, which is the “front end” of the cluster. This means that this computer is only a shared “entry point” to the whole cluster. It is shared by all the users and all the proceses, so, if you run things here, you will go slower, and you’ll slow other people too (and we can stop without warning anything we find running this way). If you want to launch anything, you have to submit it to a queue, as explained below.
Submitting a job to the queue system
A job consists in two parts: resource requests and job steps. Resource requests consist in a number of CPUs, computing expected duration, amounts of RAM or disk space, etc. Job steps describe tasks that must be done, software which must be run.
The typical way of creating a job is to write a submission script. A submission script is a shell script, e.g. a Bash script, whose comments, if they are prefixed with SBATCH, are understood by Slurm as parameters describing resource requests and other submissions options. You can get the complete list of parameters from the sbatch manpage:
man sbatch
To use the queue system all you need to do is:
sbatch scriptname
Where scriptname is a script that contains wharever you need in order to run the programs that you what.
!🚩IMPORTANT
The SBATCH directives must appear at the top of the submission file, before any other line except for the very first line which should be the shebang (e.g. #!/bin/bash).
Example:
Let’s say we’re already connected to marvin:
|
-bash-4.2$ pwd /homes/users/username |
In our case, we just want to check the date, sleep and re-check the date:
|
-bash-4.2$ cat exemple.sh #!/bin/bash
#set the job name #SBATCH --job-name= example
#set the number of CPUS per task #SBATCH --cpus-per-per-task = 1
# job output file information #SBATCH -o slurm.%j.out
# job errors file #SBATCH -e slurm.%j.err
# set the partition where the job will run #SBATCH --partition=normal # set the number of nodes #SBATCH --nodes=1 # set max wallclock time #SBATCH --time=1:00:00 # mail alert at start, end and abortion of execution #SBATCH --mail-type=ALL # send mail to this address #SBATCH --mail-user= [email protected] # run the application DIR=/homes/users/username/test-slurm date > $DIR/testing.log sleep 60 date >> $DIR/testing.log -bash-4.2$ |
Now let’s send it:
|
-bash-4.2$ sbatch exemple.sh Submitted batch job 4196 -bash-4.2$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 4196 normal exemple. username R 0:03 1 mr-00-01 -bash-4.2$ |
Notice that the squeue command has given us info about our job
As we’ve tell the queue system that we want to receive a mail at start, abortion or end of execution, once we receive the mail we can check the results:
|
-bash-4.2$ ls test-slurm/ testing.log -bash-4.2$ cat test-slurm/testing.log Tue Feb 7 16:06:47 CET 2017 Tue Feb 7 16:07:47 CET 2017 -bash-4.2$ |
man sbatch, man squeue and man sinfo will give you more information about how to use those commands to submit your jobs and monitor them.
Watching over your jobs
Useful commands of the queue system:
squeue: Shows your currently waiting or running queued jobs.
sinfo: view information about Slurm nodes and partitions
sacct -j jobid: Shows information about already finished jobs. This command shows information like the name of the job, the state ...
Example:
seff jobid: Shows information about already finished jobs. It shows information like who execute the job, cores used ...
Example:
scancel jobid: Deletes from the queue a job. You can delete only your own jobs
smem: Shows information about the cluster occupacy.
https://www.upf.edu/web/sct-sit/software
https://github.com/situpf/smem
sit-web.upf.edu/sitservices: Web interface where you can see the status of the cluster.
Examples of useful commands
|
command |
description |
|
squeue -u <username> |
List all current jobs for a user |
|
squeue -u <username> -t RUNNING |
List all running jobs for a user |
|
squeue -u <username> -t PENDING |
List all pending jobs for a user |
|
showq-slurm -o -U -q <partition> |
List priority order of jobs for the current user (you) in a given partition |
|
scontrol show jobid -dd <jobid> |
List detailed information for a job (useful for troubleshooting) |
|
sstat --format=AveCPU,AvePages,AveRSS,AveVMSize,JobID -j <jobid> --allsteps |
List status info for a currently running job |
|
scancel -t PENDING -u <username> |
To cancel all the pending jobs for a user |
Job output
By default standard error and standard output of your job will go to the same file, called slurm-%j.out , where %j is the JOBID number of your job.
standard error contains all the messages that your job has sent to the error channel, i.e. “command not found”, “no such file or directory”…
standard output contains all the messages that your job has sent to the standard output messages, i.e. anything you see written.
It is highly advisable that you control where standard error and standard output go. You can do it using sbatch flags -e for standard error and -o for standard output.
Job arrays
SLURM allows you to submit a number of "near identical" jobs simultaneously in the form of a job array. To take advantage of this, you will need a set of jobs that differ only by an "index" of some kind.
For example, say that you would like to run tophat, a splice-aware transcript-to-genome mapping tool, on 30 separate transcript files named trans1.fq, trans2.fq, trans3.fq, etc.
First, construct a SLURM batch script, called tophat.sh, using special SLURM job array variables:
tophat.sh
|
#!/bin/bash #SBATCH -J tophat # A single job name for the array #SBATCH -n 1 # Number of cores #SBATCH -N 1 # All cores on one machine #SBATCH -p fast # Partition #SBATCH --mem 4000 # Memory request (4Gb) #SBATCH -t 0-2:00 # Maximum execution time (D-HH:MM) #SBATCH -o tophat_%A_%a.out # Standard output #SBATCH -e tophat_%A_%a.err # Standard error module load TopHat/2.0.11 tophat /whatever/Mus_musculus/mm10/chromFatrans"${SLURM_ARRAY_TASK_ID}".fq |
Then launch the batch process using the --array option to specify the indexes.
|
sbatch --array=1-30 tophat.sh |
In the script, two types of substitution variables are available when running job arrays. The first, %A and %a, represent the job ID and the job array index, respectively. These can be used in the sbatch parameters to generate unique names. The second, SLURM_ARRAY_TASK_ID, is a bash environment variable that contains the current array index and can be used in the script itself. In this example, 30 jobs will be submitted each with a different input file and different standard error and standard out files.
GNU Parallel : An alternative to job arrays
There is another way to submit a large number of jobs in parallel which files’ names differs in more than one number (like 1-30 in the example below). With GNU parallel, the user should check how many CPUs is able to use (replace lab_sit with your lab name):
|
-bash-4.2$ sacctmgr show qos lab_sit format="Name,GrpTRES,MaxTRESPerUser" Name GrpTRES MaxTRESPU ---------- ------------- ------------- lab_sit cpu=450 cpu=450 |
In this example, we want to run STAR to map 84 samples (paired-end FastQ = 168 files) to a reference, and we created this script:
run_all_in_nodes.sh
|
#!/bin/bash module load parallel module load STAR/2.5.2b-foss-2016b ### input file = FW / define RV file fw=$1 rv=`echo $fw | sed 's/R1_001/R2_001/'` ### print sample identifier and the node where the job is running echo "Running sample $2 in `hostname`" ### STAR index / prefix to output files ind=/path/to/STAR/index/ pref=output_path/$2 ### STAR command STAR --runThreadN 8 --genomeDir $ind --readFilesIn $fw $rv --readFilesCommand zcat --outFileNamePrefix $pref --outSAMtype BAM SortedByCoordinate |
Each job (or task) uses 8 threads (or CPUs), so we should calculate how many tasks we can ask to run the maximum number of jobs simultaneously (in this example, c=cpus-per-task=8)
|
-bash-4.2$ c=8 -bash-4.2$ sinfo | gawk -v c=$c 'BEGIN{n=0}!/^NODELIST/{split($3,x,"/"); split(x[2]/c,y,".");n+=y[1]}END{print n-1}' 29 |
Pay attention to the maximum CPUs per user that we saw before!!! We are going to use 29x8=232 CPUs (<450 CPUs available per user).
Now, we prepare the script:
parallel.sbatch
|
#!/bin/bash ### Here, we are saying to use all the nodes availables and the number of CPUs per job #SBATCH --nodes=1-18 #SBATCH --cpus-per-task=8 ### Define number of tasks: ### 1. Number of tasks from the calculation done before #SBATCH --ntasks=29 ### 2. (Not used) Define how many tasks per node in all requested nodes. It’s a fixed number, it will only run 2 tasks per node when it’s possible (if a node has only 8 CPUs, it will not use them; and if a node has 32 CPUs it will only use 16) # #SBATCH --ntasks-per-node=2 module load parallel if [ ! -d star ];then mkdir star;fi ### execute each job as a task in a node, with 8 cpus srun="srun -N1 -n1 -c8" ### parallel how many jobs to start: -j 0 -> run as many jobs as possible parallel="parallel --delay .2 -j 0" ### run the parallel command ls path/to/fastq/*R1*.fastq.gz | $parallel "$srun ./run_all_in_nodes.sh {} {/.} " |
And finally, run multiple jobs that can be controlled by the same job ID:
|
-bash-4.2$ sbatch parallel.sbatch Submitted batch job 110766 -bash-4.2$ j JOBID PARTITION NAME USER TIME_LEFT TIME_LIMIT START_TIME ST NODES CPUS NODELIST(REASON 110766 normal parallel. mtormo UNLIMITED UNLIMITED 2017-08-31T09:45 R 12 232 mr-00-[03-11,14 -bash-4.2$ sacct -j 110766 JobID JobName Partition Account AllocCPUS State ExitCode ------------ ---------- ---------- ---------- ---------- ---------- -------- 110766 parallel.+ normal lab_genom+ 232 RUNNING 0:0 110766.0 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.1 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.2 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.3 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.4 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.5 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.6 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.7 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.8 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.9 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.10 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.11 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.12 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.13 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.14 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.15 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.16 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.17 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.18 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.19 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.20 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.21 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.22 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.23 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.24 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.25 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.26 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.27 run_all_i+ lab_genom+ 8 RUNNING 0:0 110766.28 run_all_i+ lab_genom+ 8 RUNNING 0:0 |
For more information about the GNU parallel options and advantages, pelase visit https://www.upf.edu/web/sct-sit/gnu-parallel-tutorial