Skip to content

Buy-In Accounts with SLURM

If you want to have priority access to our clusters, you can purchase buy-in nodes. Users who run on buy-in nodes receive priority access to their nodes within 4 hours and are exempt from the 1 million CPU hour per year limit.

Types of Partitions

There are three types of partitions configured in SLURM.

Name Purpose
Buyin Partition (names vary) A partition is created for each buy-in account. Each buy-in partition includes all non-buy-in nodes, allowing buy-in jobs to span buy-in and non-buy-in nodes. These jobs get equal consideration for scheduling on non-buy-in nodes as jobs in general-long. When jobs submitted to these partitions request a wall time of four hours or less, they are also submitted to the general-short, enabling them to use other available buy-in nodes and ensuring they are scheduled as fast as possible.
general-short This partition includes all nodes--buy-in and non-buy-in--and runs jobs that request a wall time of four hours or less. Jobs in this partition are considered for scheduling after jobs in buy-in partitions and the general-long partition. To prevent these jobs from being continuously bumped by general-long/buy-in jobs, they are also submitted to general-long. Jobs with a requested wall time of four hours or less are automatically submitted to this partition.
general-long This partition includes non-buy-in nodes and allows jobs to run for up to seven days. Jobs with a requested wall time of four hours or less are also submitted to the general-short partition.
general-long-bigmem This partition includes non-buy-in nodes with more memory and CPU cores than most nodes and allows jobs to run for up to seven days. Jobs requesting more than 256GB or 40 CPUs per node are automatically submitted to this partition. This partition ensures large jobs get priority access to large nodes over jobs that can run elsewhere. Jobs with a requested wall time of four hours or less are also submitted to the general-short partition.
general-long-gpu This partition contains non-buy-in nodes with GPUs and allows jobs to run for up to seven days. Jobs requesting GPUs are automatically submitted to the partition. This partition ensures jobs requesting GPUs get priority access to nodes with GPUs over jobs not requesting GPUs. Jobs with a requested wall time of four hours or less are also submitted to the general-short partition.

In most cases, you will not need to specify a partition when submitting a job. Given a specified (or default) account, SLURM's job submit plugin will assemble the partition list automatically.

Default Account

When submitting a job without specifying an account, your default account is used. You can check your default account using the buyin_status power tool.

1
2
3
4
5
6
$ module load powertools
$ buyin_status -l

User:     fordste5
Accounts: test1_classres test1 classres
Default:  classres

User fordste5 has a default account of classres. If fordste5 submits a job without specifying an account, it will be submitted to the classres partition. If the job requested four hours or less of wall time, it will be submitted to the general-short partition as well. Jobs submitted by users with a default account of general will be queued in the general-long and general-short partitions.

Default Account Wall Time<=4 Hours Wall Time<=4 Hours and >=256GB or 40CPU Wall Time<=4 Hours and GPUs Reqested Wall Time>4 Hours  Wall Time>4 Hours and >=256GB or 40CPU Wall Time>4 Hours and GPUs
general general-long,general-short general-long-bigmem,general-short general-long-gpu,general-short general-long general-long-bigmem general-long-gpu
<buyin> <buyin>, general-long, general-short <buyin>, general-long-bigmem, general-short <buyin>, general-long-gpu, general-short <buyin>, general-long <buyin>, general-long-bigmem <buyin>, general-long-gpu

where <buyin> is the name of buyin account.

Using Non-Default Accounts

You can see what accounts you have access to using the buyin_status powertool:

1
2
3
4
5
6
$ module load powertools
$ buyin_status -l

User:     fordste5
Accounts: test1_classres test1 classres
Default:  classres

This output shows that fordste5 is a member of the test1_classres, test1, and classres buy-in accounts. Both test1 and classres correspond to partitions with the same name.

Because fordste5 has a default account of classres, if they want to submit jobs to test1, they will have to explicitly specify the test1 account at submission. This is done by using either using the -A test1 option for srun/sbatch/salloc, or by adding the #SBATCH -A test1 directive to their batch script.

Checking the Status of Buy-Ins

The status of buy-ins can be viewed with the buyin_status powertool.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ module load powertools
$ buyin_status -a classres

User:     fordste5
Accounts: test1_classres test1 classres
Default:  classres


Buyin: classres
  JOBID      STATE      USER       CPUS PRIORITY  TIME_LIMIT           START_TIME
  17401      RUNNING    fordste5   10   101          1:00:00  2018-08-09T12:55:17
  17409      RUNNING    changc81   4    100          1:00:00  2018-08-09T12:57:54

  Partition: classres
    lac-421 (down*)
    csn-020 (down*)
    csp-018 (down*)
    css-034 (down*)
    css-035 (down*)
    css-079 (down*)
    css-080 (down*)
    css-033 (allocated)
      JOBID      ACCOUNT    USER       CPUS       TIME  TIME_LEFT
      17402      general    fordste5   10         4:36      55:24
      17401      classres   fordste5   10         4:36      55:24

This tool will list all jobs that are queued for a buy-in account, all nodes associated with the buy-in account–including node status–and a list of jobs running on each node. Because jobs with a wall time of four hours or less can run on any buy-in node, it is possible to see jobs running on buy-in nodes from other accounts.

Buy-in partitions have the same name as a given buy-in account

Managing Buy-In Account Membership

SLURM allows for the configuration of account coordinators. Account coordinators can add and remove users to accounts that they coordinate. Buy-in account owners are configured as coordinators of their buy-in accounts and can request that other users also be added as coordinators.

If you had a buy-in account configured in the old Moab scheduler, account coordinators for your SLURM buy-in account were copied from the managers listed on the Moab account.

View what users have access to your buy-in account:

1
$ sacctmgr show association where account=<account>

Add a user to your buy-in account:

1
$ sacctmgr add user name=<userid> account=<account>

Remove a user from your buy-in account:

1
$ sacctmgr delete user <userid> where account=<account>