Skip to content

How Jobs are Scheduled

Schedulers

SLURM schedules jobs in two ways: the main scheduler and the backfill scheduler. The main scheduler constantly tries to start high priority jobs. The backfill scheduler considers all jobs, and starts any jobs that won't defer the start time of a higher priority job.

Scheduler Function When it Runs Run Time
Main Launches high priority jobs that can start immediately. Stops evaluating jobs once it encounters a job that cannot be started.  About every 2 seconds 0.08-2 seconds
Backfill Evaluates the entire queue. Launches jobs that won't interfere with the start time of a higher priority job. Sets jobs' StartTime and SchedNodeList. 20 seconds after the last backfill cycle completes 2-15+ minutes

StartTime and SchedNodeList

The backfill scheduler sets the StartTime and SchedNodeList parameters on jobs that can start within the next 7 days. These parameters can be viewed in the output of scontrol show job <jobid>. StartTime estimates when a job will start and SchedNodeList shows the nodes this job might start on. StartTime is only an estimate. These values are updated every time the backfill scheduler runs and may change as running jobs complete and new jobs are submitted.

Minimum Job Requirements to Avoid Deferment

Jobs must meet certain criteria before the backfill scheduler will avoid potentially deferring them through starting lower priority jobs. These thresholds allow the backfill scheduler to cycle faster and maintain high system utilization.

Criteria Minimum Description
Priority 3000 Jobs require a minimum priority of 3000 is require to avoid potential deferment in scheduling. Buy-in account jobs are never below this threshold.

Job Priority Factors

A job's priority is determined by a combination of several priority factors. Age, size, fairshare, and whether it was submitted to a buy-in account all contribute to the job’s priority.

Priority Factor Description Maximum Contribution to Priority
Age Starts at zero at job submission, then increases linearly to a maximum of 60000 after 30 days 60000 after 30 days
Fairshare Starts at 60000 and decreases and users' recent usage goes up. Usage for this calculation is decayed 50% each day 60000 for no recent cluster usage
Size Scales linearly with the amount of CPU and memory requested by a job. 100 per CPU, 20 per GB. 52000+ depending on memory requested
QOS Adds 3000 to buy-in jobs to ensure they are always above backfill schedulers minimum priority for reserving resources 3000

FairShare

The FairShare priority factor is calculated based on recent usage compared to overall cluster usage and gives a priority boost to users who have consumed fewer resources than others. The exact amount of usage it takes to take fairshare priority down to zero varies with overall system system usage, but can be exhausted after two days with sustained usage of ~1000 CPUs or 40 GPUs.

Usage is tracked based on the cluster's configured Trackable Resource (TRES) billing weights. Weights are configured for CPUs, memory, and each type of GPU. Every second a job consumes each resource is multiplied by these weights and added together to get a combined measure of TRES seconds that are tracked for each user/account pair.

The formula used to calculate the FairShare factor from accumulated TRES seconds and is as follows:

F - FairShare priority factor U - Usage as a proportion of total cluster usage S - User's configured "share" of the cluster W - FairShare priority weight (60000)

F = (2 ^ (-U/S)) * W

When consumed TRES seconds is equal to the share of TRES seconds relative to the entire cluster, the FairShare factor will be 30000, and when consumed TRES seconds exceeds double the share for the entire cluster, the FairShare factor will be 15000, tapering off towards zero as usage increases. Usage accrual for this calculation decays with a half life of one day and the effect of this decay is recalculated every five minutes.

To view the exact state of the FairShare configuration on the cluster, you can use the fairshare_info powertool.

input
module load powertools
fairshare_info
output
Resource Weights:

  CPU               = 1
  Memory (GB)       = 0.19999
  GPU (Unspecified) = 50
  GPU (v100)        = 50
  GPU (v100s)       = 50
  GPU (a100)        = 75
  GPU (a100_slice)  = 10
  GPU (l40s)        = 25
  GPU (gh200)       = 50
  GPU (h200)        = 100

Current Total Cluster Usage:  3879799 Trackable Resource Hours
FairShare Ratio Per User:     0.00240
User Portion of Cluster:      9312 Trackable Resource Hours

FairShare is calculated every 00:05:00 and your usage decays 50% every 1-00:00:00

The following usage will reduce FairShare priority to 1/2. Twice this usage will reduce it to 1/4:

      9312 CPU Hours
       186 GPU Hours
     46561  GB Hours

      7760 1 Hour / 1 CPU / 1 GB Jobs
       182 1 Hour / 1 CPU / 1 GB / 1 GPU (Unspecified) Jobs
        93 1 Hour / 1 CPU / 1 GB / 1 GPU (h200) Jobs

Current FairShare Priority Status:

  Account             Priority  Usage (TRES Seconds)
  --------------------------------------------------
  general                60000                    0
  scavenger              60000                    0