Basic Slurm Terminology

Partition (a.k.a. Queue)

A partition is a logical grouping of compute nodes, similar to a queue. Each partition may have different hardware, time limits, or access policies. Use sinfo to list available partitions.

To check the current status of available queues and nodes, use the sinfo command. This command provides a snapshot of which nodes are idle, allocated, or down, along with their associated partitions and time limits.

Example output:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
compute*     up 3-00:00:00      1  alloc larcc-cpu3
compute*     up 3-00:00:00      9   idle larcc-cpu[1-2,4-10]
gpu          up 2-00:00:00      3  alloc larcc-gpu[1-3]
gpu          up 2-00:00:00      7   idle larcc-gpu[4-10]

In this example:

  • The compute partition has 10 nodes, 1 of which is currently in use (alloc) and 9 are idle.

  • The gpu partition has 10 nodes, with 3 allocated and 7 idle.

  • The TIMELIMIT column shows the maximum wall time allowed per job in each partition.

Node

A node is a physical machine in the cluster. Each node has a specific number of CPU cores, memory, and possibly GPUs.

To get detailed information about a specific node, use:

scontrol show node <nodename>

For example:

scontrol show node larcc-cpu1

This will return detailed specs and current usage for the node:

NodeName=larcc-cpu1 Arch=x86_64 CoresPerSocket=64
   CPUAlloc=0 CPUEfctv=128 CPUTot=128 CPULoad=0.00
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=(null)
   NodeAddr=larcc-cpu1 NodeHostName=larcc-cpu1 Version=24.11.4
   OS=Linux 5.14.0-503.38.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 16 16:38:39 UTC 2025
   RealMemory=515002 AllocMem=0 FreeMem=505959 Sockets=2 Boards=1
   State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=compute
   BootTime=2025-05-15T13:30:56 SlurmdStartTime=2025-07-30T13:57:36
   LastBusyTime=2025-09-22T16:10:19 ResumeAfterTime=None
   CfgTRES=cpu=128,mem=515002M,billing=128
   AllocTRES=
   CurrentWatts=240 AveWatts=180

Key fields to note:

  • CPUTot: Total number of CPU cores available.

  • RealMemory: Total memory available on the node.

  • State: Current status (e.g., IDLE, ALLOC, DOWN).

  • Partitions: Indicates which queue(s) the node belongs to.

Job

A job is a unit of work submitted to Slurm. It can be a single task or a collection of tasks (e.g., simulations, data processing).

Jobs are submitted using sbatch (batch jobs) or srun (interactive jobs).

A Job can have 5 states:

  • PD (Pending): Waiting for resources.

  • R (Running): Currently executing.

  • CG (Completing): Finishing up.

  • CD (Completed): Finished successfully.

  • F (Failed): Encountered an error.

To view jobs currently in the queue or running, use: squeue.

Example output:

                JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
             3777   compute matlab_p jd01  R    4:02:04      1 larcc-cpu3
3852_[226-241%20]       gpu Ecthelio jd02 PD       0:00      1 (QOSMaxNodePerUserLimit)
         3852_224       gpu Ecthelio jd02  R       2:00      1 larcc-gpu3
         3852_225       gpu Ecthelio jd02  R       2:00      1 larcc-gpu3
         3852_222       gpu Ecthelio jd02  R       2:01      1 larcc-gpu2
         3852_223       gpu Ecthelio jd02  R       2:01      1 larcc-gpu2
             3343       gpu sno_pv80 jd03  R    4:02:05      1 larcc-gpu1

Explanation of columns:

  • JOBID: Unique identifier for each job.

  • PARTITION: Queue the job was submitted to.

  • NAME: Job name.

  • USER: Submitting user.

  • ST: Job status (R = Running, PD = Pending).

  • TIME: Runtime so far.

  • NODELIST(REASON): Node(s) assigned or reason for pending status.

Job Steps

A job step is a unit of work executed within a running job. While a job defines the overall resource allocation (e.g., number of nodes, CPUs, memory, time), job steps are the actual commands or tasks that run using those resources.

Key Characteristics of a Job Step:

  • Executed with srun: Job steps are typically launched using the srun command inside a job script or interactively.

  • Shares job resources: All job steps run within the resource allocation defined by the parent job (sbatch).

  • Can run in parallel: Multiple job steps can be launched simultaneously, each using a subset of the allocated resources.

  • Useful for multi-task jobs: Ideal when you want to run several independent tasks (e.g., simulations, analyses) within a single job submission.