There are three ways of submitting jobs with slurm, using either sbatch, srun or salloc:
=== "sbatch (passive job)"
bash ### /!\ Adapt <partition>, <qos>, <account> and <command> accordingly sbatch -p <partition> [--qos <qos>] [-A <account>] [...] <path/to/launcher.sh>
=== "srun (interactive job)"
bash ### /!\ Adapt <partition>, <qos>, <account> and <command> accordingly srun -p <partition> [--qos <qos>] [-A <account>] [...] ---pty bash
srun is also to be using within your launcher script to initiate a job step.
=== "salloc (request allocation/interactive job)"
bash # Request interactive jobs/allocations ### /!\ Adapt <partition>, <qos>, <account> and <command> accordingly salloc -p <partition> [--qos <qos>] [-A <account>] [...] <command>
sbatch is used to submit a batch launcher script for later execution, corresponding to batch/passive submission mode.
The script will typically contain one or more srun commands to launch parallel tasks.
Upon submission with sbatch, Slurm will:
- allocate resources (nodes, tasks, partition, constraints, etc.)
- runs a single copy of the batch script on the first allocated node
- in particular, if you depend on other scripts, ensure you have refer to them with the complete path toward them.
When you submit the job, Slurm responds with the job's ID, which will be used to identify this job in reports from Slurm.
# /!\ ADAPT path to launcher accordingly
$ sbatch <path/to/launcher>.sh
Submitted batch job 864933srun is used to initiate parallel job steps within a job OR to start an interactive job
Upon submission with srun, Slurm will:
- (eventually) allocate resources (nodes, tasks, partition, constraints, etc.) when run for interactive submission
- launch a job step that will execute on the allocated resources.
A job can contain multiple job steps executing sequentially or in parallel on independent or shared resources within the job's node allocation.
salloc is used to allocate resources for a job
in real time. Typically this is used to allocate resources (nodes, tasks, partition, etc.) and spawn a
shell. The shell is then used to execute srun commands to launch
parallel tasks.
You should use the helper functions si, si-gpu, si-bigmem to submit an interactive job.
For more details, see interactive jobs.
| Command | Description |
|---|---|
sacct [-X] -j <jobid> [...] |
display accounting information on jobs. |
scontrol show [...] |
view and/or update system, nodes, job, step, partition or reservation status |
seff <jobid> |
get efficiency metrics of past job |
smap |
graphically show information on jobs, nodes, partitions |
sprio |
show factors that comprise a jobs scheduling priority |
squeue [-u $(whoami)] |
display jobs[steps] and their state |
sstat |
show status of running jobs. |
You can view information about jobs located in the Slurm scheduling queue (partition/qos), eventually filter on specific job state (R:running /PD:pending / F:failed / PR:preempted) with squeue:
$ squeue [-u <user>] [-p <partition>] [---qos <qos>] [--reservation <name>] [-t R|PD|F|PR]To quickly access your jobs, you can simply use sq
You can use the scurrent (for current interactive job) or (more generally) scontrol show job <jobid> to collect detailed information for a running job.
??? example "scontrol show job <jobid>"
console $ scontrol show job 2166371 JobId=2166371 JobName=bash UserId=<login>(<uid>) GroupId=clusterusers(666) MCS_label=N/A Priority=12741 Nice=0 Account=ulhpc QOS=debug JobState=RUNNING Reason=None [...] SubmitTime=2020-12-07T22:08:25 EligibleTime=2020-12-07T22:08:25 StartTime=2020-12-07T22:08:25 EndTime=2020-12-07T22:38:25 [...] WorkDir=/mnt/irisgpfs/users/<login>
Use the slist helper for a given job:
# /!\ ADAPT <jobid> accordingly
$ slist <jobid>
# sacct -j <JOBID> --format User,JobID,Jobname%30,partition,state,time,elapsed,\
# MaxRss,MaxVMSize,nnodes,ncpus,nodelist,AveCPU,ConsumedEnergyRaw
# seff <jobid>You can also use sreport o generate reports of job usage and cluster utilization for Slurm jobs. For instance, to list your usage in CPU-hours since the beginning of the year:
$ sreport -t hours cluster UserUtilizationByAccount Users=$USER Start=$(date +%Y)-01-01
--------------------------------------------------------------------------------
Cluster/User/Account Utilization 2021-01-01T00:00:00 - 2021-02-13T23:59:59 (3801600 secs)
Usage reported in CPU Hours
----------------------------------------------------------------------------
Cluster Login Proper Name Account Used Energy
--------- --------- --------------- ---------------------- -------- --------
iris <login> <name> <firstname>.<lastname> [...]
iris <login> <name> project_<acronym> [...]Use seff to double check a past job CPU/Memory efficiency. Below examples should be self-speaking:
=== "Good CPU Eff."
console $ seff 2171749 Job ID: 2171749 Cluster: iris User/Group: <login>/clusterusers State: COMPLETED (exit code 0) Nodes: 1 Cores per node: 28 CPU Utilized: 41-01:38:14 CPU Efficiency: 99.64% of 41-05:09:44 core-walltime Job Wall-clock time: 1-11:19:38 Memory Utilized: 2.73 GB Memory Efficiency: 2.43% of 112.00 GB
=== "Good Memory Eff."
console $ seff 2117620 Job ID: 2117620 Cluster: iris User/Group: <login>/clusterusers State: COMPLETED (exit code 0) Nodes: 1 Cores per node: 16 CPU Utilized: 14:24:49 CPU Efficiency: 23.72% of 2-12:46:24 core-walltime Job Wall-clock time: 03:47:54 Memory Utilized: 193.04 GB Memory Efficiency: 80.43% of 240.00 GB
=== "Good CPU and Memory Eff."
console $ seff 2138087 Job ID: 2138087 Cluster: iris User/Group: <login>/clusterusers State: COMPLETED (exit code 0) Nodes: 1 Cores per node: 64 CPU Utilized: 87-16:58:22 CPU Efficiency: 86.58% of 101-07:16:16 core-walltime Job Wall-clock time: 1-13:59:19 Memory Utilized: 1.64 TB Memory Efficiency: 99.29% of 1.65 TB
=== "[Very] Bad efficiency"
This illustrates a very bad job in terms of CPU/memory efficiency (below 4%), which illustrate a case where basically the user wasted 4 hours of computation while mobilizing a full node and its 28 cores.
console $ seff 2199497 Job ID: 2199497 Cluster: iris User/Group: <login>/clusterusers State: COMPLETED (exit code 0) Nodes: 1 Cores per node: 28 CPU Utilized: 00:08:33 CPU Efficiency: 3.55% of 04:00:48 core-walltime Job Wall-clock time: 00:08:36 Memory Utilized: 55.84 MB Memory Efficiency: 0.05% of 112.00 GB
This is typical of a single-core task can could be drastically improved via GNU Parallel.
Note however that demonstrating a CPU good efficiency with seff may not be enough!
You may still induce an abnormal load on the reserved nodes if you spawn more processes than allowed by the Slurm reservation.
To avoid that, always try to prefix your executions with srun within your launchers. See also Specific Resource Allocations.
Use susage to check your past jobs walltime accuracy (Timelimit vs. Elapsed)
$ susage -h
Usage: susage [-m] [-Y] [-S YYYY-MM-DD] [-E YYYT-MM-DD]
For a specific user (if accounting rights granted): susage [...] -u <user>
For a specific account (if accounting rights granted): susage [...] -A <account>
Display past job usage summaryAlternatively, you can use sacct (use sacct --helpformat to get the list of) for COMPLETED or TIMEOUT jobs (see Job State Codes).
??? example "using sacct -X -S <start> [...] --format [...],time,elapsed,[...]"
ADAPT -S <start> and -E <end> dates accordingly - Format: YYYY-MM-DD.
hint: $(date +%F) will return today's date in that format, $(date +%Y) return the current year, so the below command will list your completed (or timeout jobs) since the beginning of the month:
console $ sacct -X -S $(date +%Y)-01-01 -E $(date +%F) --partition batch,gpu,bigmem --state CD,TO --format User,JobID,partition%12,qos,state,time,elapsed,nnodes,ncpus,allocGRES User JobID Partition QOS State Timelimit Elapsed NNodes NCPUS AllocGRES --------- ------------ ------------ ---------- ---------- ---------- ---------- -------- ---------- ------------ <login> 2243517 batch normal TIMEOUT 2-00:00:00 2-00:00:05 4 112 <login> 2243518 batch normal TIMEOUT 2-00:00:00 2-00:00:05 4 112 <login> 2244056 gpu normal TIMEOUT 2-00:00:00 2-00:00:12 1 16 gpu:2 <login> 2246094 gpu high TIMEOUT 2-00:00:00 2-00:00:29 1 16 gpu:2 <login> 2246120 gpu high COMPLETED 2-00:00:00 1-02:18:00 1 16 gpu:2 <login> 2247278 bigmem normal COMPLETED 2-00:00:00 1-05:59:21 1 56 <login> 2250178 batch normal COMPLETED 2-00:00:00 10:04:32 1 1 <login> 2251232 gpu normal COMPLETED 1-00:00:00 12:05:46 1 6 gpu:1
sinfo allow to view information about partition status (-p <partition>), problematic nodes (-R), reservations (-T), eventually in a summarized form (-s),
sinfo [-p <partition>] {-s | -R | -T |...}
We are providing a certain number of helper functions based on sinfo:
| Command | Description |
|---|---|
nodelist |
List available nodes |
allocnodes |
List currently allocated nodes |
idlenodes |
List currently idle nodes |
deadnodes |
List dead nodes per partition (hopefully none ;)) |
sissues |
List nodes with issues/problems, with reasons |
sfeatures |
List available node features |
We have defined several custom ULHPC Slurm helpers defined in /etc/profile.d/slurm.sh to facilitate access to account/parition/qos/usage information.
They are listed below.
| Command | Description |
|---|---|
acct <name> |
Get information on user/account holder <name> in Slurm accounting DB |
irisstat, aionstat |
report cluster status (utilization, partition and QOS live stats) |
listpartitionjobs <part> |
List jobs (and current load) of the slurm partition <part> |
pload [-a] i/b/g/m |
Overview of the Slurm partition load |
qload [-a] <qos> |
Show current load of the slurm QOS <qos> |
sbill <jobid> |
Display job charging / billing summary |
sjoin [-w <node>] |
join a running job |
sassoc <name> |
Show Slurm association information for <name> (user or account) |
slist <jobid> [-X] |
List statistics of a past job |
sqos |
Show QOS information and limits |
susage [-m] [-Y] [...] |
Display past job usage summary |
| Command | Description |
|---|---|
scancel <jobid> |
cancel a job or set of jobs. |
scontrol update jobid=<jobid> [...] |
update pending job definition |
scontrol hold <jobid> |
Hold job |
scontrol resume <jobid> |
Resume held job |
The scontrol command allows certain charactistics of a job to be
updated while it is still queued (i.e. not running ), with the syntax scontrol update jobid=<jobid> [...]
!!! important
Once the job is running, most changes requested with scontrol update jobid=[...] will NOT be applied.
# /!\ ADAPT <jobid> and new time limit accordingly
scontrol update jobid=<jobid> timelimit=<[DD-]HH:MM::SS># /!\ ADAPT <jobid>, <qos>, <resname> accordingly
scontrol update jobid=<jobid> qos=<qos>
scontrol update jobid=<jobid> reservationname=<resname>If you forgot to specify the expected project account:
# /!\ ADAPT <jobid>, <account> accordingly
scontrol update jobid=<jobid> account=<account>!!! note "" The new account must be eligible to run the job. See Account Hierarchy for more details.
Prevent a pending job from being started:
# /!\ ADAPT <jobid> accordingly
scontrol hold <jobid>Allow a held job to accrue priority and run:
# /!\ ADAPT <jobid> accordingly
scontrol release <jobid>Cancel a specific job:
# /!\ ADAPT <jobid> accordingly
scancel <jobid>??? info "Cancel all jobs owned by a user (you)"
scancel -u $USER
This only applies to jobs which are associated with your
accounts.