SLURM resource request guide
This guide will help you identify ways you can improve your SLURM job submission scripts to request the appropriate resources for your jobs. This will help your jobs queue efficiently.
Note
Walltime is the time taken measured by a "clock on the wall", i.e. it is the time taken for your code to run, and the time you will request in a SLURM script. CPU time is the amount of time used per CPU, so if 5 CPUs are used for the entire walltime, the CPU time will be 5 times the walltime.
Measure your resource requirements
The first step to obtain the correct resource requirements is to understand your code's resource usage. There are many ways to do this, so this guide will not be exhaustive.
Code documentation
If your code has documentation, this should be the first place to go. The code documentation or descriptive journal article may have guidelines for resource requirements such as memory usage, walltime, and number of CPU cores. If the code is currently maintained or you have a support agreement with the software producer, you can contact the developers to ask them about their experience with HPCC resource requirements.
Local testing
If your code can be run on a local computer such as your laptop, you can easily
estimate the required walltime by timing a run of the code. You may also be able
to estimate the required CPU cores and memory using your computer's resource
monitor (Task Manager for Windows, Activity Monitor on Mac). For timing on Linux systems,
you can use the time command. This is run as time <your process name> and will return 3 time
measurements: real, user and sys. real is the equivalent of walltime,
while user and sys are CPU time measurements. See this Wikipedia article
for more information: https://en.wikipedia.org/wiki/Time_%28Unix%29.
Development node testing
Note
Development nodes have a maximum CPU time of 2 hours for processes. They are also used by other users which will affect CPU and GPU usage estimates.
Our development nodes are a potentially useful place to investigate your code's resource requirements for short jobs (< 2 CPU hours). Note that each additional CPU you use reduces your total allowed process time. Testing is best done when the dev node reports low usage.
You can use the Linux tool top to measure memory and CPU usage. Some
development nodes (dev-amd20-v100 and dev-amd24-h200) have
access to GPUs to help you determine GPU resource requirements.
As mentioned in Local testing you can use the time command
to measure walltime and CPU time, though this may be inaccurate in some cases.
Just going for it
If all else fails, you can just try submitting your job and seeing what happens! You will need to use a permissive resource request with a large amount of memory and walltime so that you can measure your code's needs. You should expect the queue time for this test job to be long. For faster queuing of test jobs, request walltimes of less than 4 hours.
Refining your resource request
Once you have a solid estimate (or wild guess) as to your code's resources, you can test them by submitting a SLURM job. Once this job completes (or all the walltime is used), you can use the following techniques to refine your request.
Reviewing job efficiency
seff and reportseff are useful tools for investigating your resource request
efficiency. They will provide statistics for individual jobs (seff <job id>)
or a report of multiple jobs (reportseff -u <user name>).
You can use these tools to get a quick measurement of your resource request usage.
An example seff output is as follows:
seff
Job ID: 17792078
Cluster: msuhpcc
User/Group: kopenhaf/pa
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 64
CPU Utilized: 02:14:21
CPU Efficiency: 94.70% of 02:21:52 core-walltime
Job Wall-clock time: 00:02:13
Memory Utilized: 151.68 GB
Memory Efficiency: 59.25% of 256.00 GB
This seff output tells us the requested CPU configuration and total memory as well
as how efficiently each of these were actually utilized by the job. It also gives the
total walltime of the job. A job does not have to complete successfully for seff to be used;
seff can even be used on a running job but may report misleading information.
The reportseff command can give you information on multiple jobs. Like seff, it
reports CPU and memory efficiency but it also reports the wall time efficiency.
With no additional arguments, reportseff looks for slurm output files in your current directory:
ls slurm*.out
slurm-17662021.out
slurm-17792028.out
slurm-17792078.out
reportseff
JobID State Elapsed TimeEff CPUEff MemEff
slurm-17662021.out FAILED 00:02:14 0.9% 92.9% 59.3%
slurm-17792028.out CANCELLED 00:01:07 0.5% 92.2% 59.3%
slurm-17792078.out COMPLETED 00:02:13 0.9% 94.7% 59.2%
You can also use reportseff's options to filter past jobs from a given period of time.
The following pulls all completed jobs from the last week:
reportseff --since w=1 --state COMPLETED
JobID State Elapsed TimeEff CPUEff MemEff
16207131 COMPLETED 1-19:59:19 26.2% 99.8% 13.4%
16207136 COMPLETED 1-02:36:40 15.8% 99.6% 12.5%
16207137 COMPLETED 1-08:10:49 19.2% 99.6% 12.3%
16207140 COMPLETED 1-01:44:58 15.3% 99.6% 12.0%
16207143 COMPLETED 3-12:52:47 50.5% 99.7% 20.6%
16207145 COMPLETED 2-14:04:05 36.9% 99.7% 15.3%
16207146 COMPLETED 1-08:47:16 19.5% 99.7% 13.8%
16207151 COMPLETED 2-08:52:28 33.9% 99.7% 14.6%
You can read more about using reportseff on the tool's (GitHub README)[https://github.com/troycomi/reportseff].
Note
reportseff can sometimes generate malformed output if the number of lines overflow the terminal window. In such cases, pipe the output of 'reportseff' through less: reportseff <options> | less. You can exit this output by hitting q.
Adding more CPUs to longer jobs
If your code can take advantage of parallelism, adding more CPUs won't always make it run faster. This is because of something called "scaling." At a certain point, adding more CPUs will give diminishing returns. Where this point is depends on your code and you'll need to perform a scaling measurement to help decide.
To measure scaling, you can run your code a few times with increasing CPU requests each time and measure how long it takes. Then you can fit a simple linear or exponential function to these points and approximate intermediate requests, or extrapolate larger resource requests.
This strategy only makes sense if your CPU usage is already relatively high.
If tools like seff and reportseff show low CPU utilization, your code is not
equipped to take full advantage of the CPUs you requested and you should reduce
your resource request accordingly.
Auto-reducing your memory allocation
Sometimes memory utilization can be hard to predict when we use known software on new problems.
SLURM now offers the mem-update option which allows SLURM to free up
unused memory that you requested mid-job. While this option does not affect your queue time
(as that depends on the initial request) it does release unused resources for others to
take advantage of.
The mem-update option is included in your SLURM resource request. After a certain specified delay period, SLURM
measures the current resident physical (RSS) memory being used by your job.
The allocation is then adjusted to be a specified percentage above this maximum if the new limit would be below the original limit.
When using this option, you should have a good idea of when your program should reach its maximum memory usage. Let's say we have a Python script that reads in a large dataset during its first few operations. In this hypothetical situation, we expect the maximum memory usage to be reached after 10 minutes. Since we filter the data into smaller sub-datasets, we decide that we should give ourselves 30% above the measured maximum. The SLURM request line would be:
#SBATCH --mem-update=30@00:10:00
SLURM will never increase your memory allocation, even if the new calculated maximum would exceed your initial request. It will also not reduce your memory allocation to less than 10% of the original request amount in order to avoid unreasonably small memory limits. The memory allocation will only be updated at most once.