Skip to content

General rules for writing a job script

The sbatch command is used for submitting batch jobs to the cluster (same as Torque's qsub). sbatch accepts a number of options either from the command line, or (more typically) from a batch job script. In this section, we will show you a simple job script and how to submit it to SLURM. Note the sbatch command only runs on development and compute nodes - it will not work on any gateway node.

Job script

A job script contains two parts: #SBATCH lines for resource request and command lines for job running. The script should be in plain text format. Below is an example used to explain what each lines means.

Note that the job script below is not intended to be used as your job template (more on that later)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
#!/bin/bash --login
########## SBATCH Lines for Resource Request ##########

#SBATCH --time=00:10:00             # limit of wall clock time - how long the job will run (same as -t)
#SBATCH --nodes=1-5                 # number of different nodes - could be an exact number or a range of nodes (same as -N)
#SBATCH --ntasks=5                  # number of tasks - how many tasks (nodes) that you require (same as -n)
#SBATCH --cpus-per-task=2           # number of CPUs (or cores) per task (same as -c)
#SBATCH --mem-per-cpu=2G            # memory required per allocated CPU (or core)
#SBATCH --job-name Name_of_Job      # you can give your job a name for easier identification (same as -J)

########## Command Lines for Job Running ##########

module load GCC/6.4.0-2.28 OpenMPI  ### load necessary modules.

cd <path_to_code>                   ### change to the directory where your code is located.

srun -n 5 <executable>              ### call your executable. (use srun instead of mpirun.)

scontrol show job $SLURM_JOB_ID     ### write job information to SLURM output file.
js -j $SLURM_JOB_ID                 ### write resource usage to SLURM output file (powertools command).

As was previously said, the above job script is not intended to be used as your job template. Your job is most likely going to use much simpler resource specifications than shown above. For purpose of illustration, this job requests 10 minutes walltime, at least 1 at most 5 different nodes, a total of 5 parallel tasks (processes) in distributed memory, 2 cores per task for parallel threads in shared memory and 2 GB memory per core (total 2 GB x 5 tasks x 2 cpus-per-task = 20 GB) with job name "Name_of_Job". After this job starts, it first loads two modules: GCC/6.4.0-2.28 and the default version of OpenMPI. Then, change the directory to the path of the code and run the specified executable in 5 parallel tasks. After running the program, it outputs the job information and quits.

By default, SLURM will try to use the settings: --nodes=1, --tasks-per-node=1, --cpus-per-task=1, --time=00:01:00 and --mem-per-cpu=750 for each job if any of them can not be acquired from the job specifications. Also, the job script must begin with a specification of a shell type or an interpreter on the first line, such as #!/bin/bash or '#!/usr/bin/python' . All lines starting with #SBATCH need to be placed above the first command line in the script. If they are below the line, the job controller will not execute them and lead to unexpected results.

Batch job submission

Once the job script has been created, the job can be submitted using the sbatch command. If the command has been submitted successfully, the job controller will issue a job ID on the screen:

1
2
$ sbatch myjob.sb
Submitted batch job 8929

Optionally, any job specification (by #SBATCH line) can also be requested by sbatch command line with an equivalent option. For instance, the #SBATCH  --nodes=1-5 line could be removed from the job script, and instead be specified from the command line:

1
$ sbatch --nodes=1-5 myjob.sb
Command line specifications take precedence over those in the job script.

Advanced: job with multiple tasks

In the SLURM job options, the number of tasks (--ntasks) is used for jobs running multiple tasks with distributed memory. It has the same meaning as the number of processes (-np) used in mpirun application. This is similar to number of nodes (nodes=) used in a Torque (#PBS) job script.

Each task could also use more than 1 core in shared memory (controlled by --cpus-per-task similar to ppn= in #PBS job script) and each node could run more than 1 task (controlled by --tasks-per-node). By default, SLURM will use 1 core per task if --cpus-per-task (or -c) is not specified.

It is better for multiple-task jobs to use srun command instead of mpirun to launch a software application. More details about srun, check out the SLURM web site.