Job Arrays - Run multiple similar jobs simultaneously
Job array is an efficient way to submit and manage a collection of jobs that differ from each other by only a single index parameter. All jobs in a job array should have the same resource request (ex. size, time, etc.). For example, in the task below we wish to run python script "hello.py" ten times, each with a different input parameter for. Instead of creating 10 separate Slurm job scripts and submitting them separately, we can create an array job in a script and submit it once for all.
-
Creat Python script file "hello.py" as show below.
hello.py
#!/usr/bin/env python
# import sys library (needed for accepted command line args)
import sys
# example of "hello world!" in Python
print('Hello World! This is the task number', sys.argv[1])
-
Creat a job array script file "hello.sb" as show below.
hello.sb
#!/bin/bash
# Example of running python script in a batch mode
#SBATCH -J hello.py # job name
#SBATCH -c 1 # use one CPU core
#SBATCH --mem=1M # request memory
#SBATCH -t 10:00 # 10 minutes time limit
#SBATCH -o hello-%A_%a.out # output file name pattern
# %A is jobID and %a is task index.
#SBATCH --array=1-10 # run array of 10 tasks with index 1, 2, ... 10.
echo "This script is from ICER's job array tutorial"
# Load default version of Python
module purge
module load Python
# Run array of python script
python hello.py $SLURM_ARRAY_TASK_ID
- Submit the job array as shown in following line:
sbatch hello.sb
Now you can see there are 10 jobs submitted into your job queue.
This example was modified from the example at: https://rcpedia.stanford.edu/topicGuides/jobArrayPythonExample.html
User can also download job array examples from our collection of examples by running following commands:
module load powertools
getexample basic_array_job
A directory named "basic_array_job" containing two simple examples of job array will be downloaded to user's current directory.