Skip to content

Job Arrays - Run multiple similar jobs simultaneously

Job array is an efficient way to submit and manage a collection of jobs that differ from each other by only a single index parameter. All jobs in a job array should have the same resource request (ex. size, time, etc.). For example, in the task below we wish to run python script "hello.py" ten times, each with a different input parameter for. Instead of creating 10 separate Slurm job scripts and submitting them separately, we can create an array job in a script and submit it once for all.

  • Creat Python script file "hello.py" as show below.

    hello.py

1
2
3
4
5
6
7
    #!/usr/bin/python

    # import sys library (needed for accepted command line args)
    import sys

    # example of "hello world!" in Python
    print('Hello World! This is the task number', sys.argv[1])

  • Creat a job array script file "hello.sb" as show below.

    hello.sb  

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
    #!/bin/bash

    # Example of running python script in a batch mode

    #SBATCH -J hello.py                     # job name
    #SBATCH -c 1                            # use one CPU core
    #SBATCH --mem=1M                        # request memory
    #SBATCH -t 10:00                        # 10 minutes time limit
    #SBATCH -o hello-%A_%a.out              # output file name pattern
                                            # %A is jobID and %a is task index.

    #SBATCH --array=1-10                    # run array of 10 tasks with index 1, 2, ... 10.

    # Load default version of Python
    module purge
    module load Python

    # Run array of python script
    python hello.py $SLURM_ARRAY_TASK_ID
  • Submit the job array as shown in following line: 
1
    sbatch hello.sb

Now you can see there are 10 jobs submitted into your job queue.

This example was modified from the example at: https://rcpedia.stanford.edu/topicGuides/jobArrayPythonExample.html

User can also download job array examples from our collection of examples by running following commands:

1
2
    module load powertools
    getexample basic_array_job

A directory named "basic_array_job" containing two simple examples of job array will be downloaded to user's current directory.