Warning
This is a Lab Notebook which describes how to solve a specific problem at a specific time. Please keep this in mind as you read and use the content. Please pay close attention to the date, version information and other details.
Difficulties working with GPUs
As interest in AI and LLMs grows, we have seen increased use of GPU resources on the HPCC. Typically, long running (>4 hours) GPU jobs take longer to queue than similar CPU jobs and we have received reports that it is difficult to develop code which use GPUs because GPU dev nodes (h200 in particular) are constantly in use. While ICER intends to obtain more GPUs during the next cluster purchase, for the immediate future these resources are fixed in quantity. To relieve some of the pressure on the limited number of GPUs available on HPCC, we are providing the following guide for scheduling GPU jobs and accessing GPU for the purpose of development.
Planning GPU jobs on HPCC
Before you schedule a job on HPCC, it's important to be aware of available resources on the HPCC, and the resource requirements of your job. ICER provides a table of available resources on each cluster which includes the nodes with GPUs, number of GPUs per node, as well as the type and memory size of each GPU. When estimating the needs of a job, dedicated tools will often describe their needs in the documentation and there are online tools for estimating the resource needs of commonly used open-weight LLMs.
Where possible, don't limit jobs to only the most recent GPUs. While the h200 GPUs can support larger models with higher performance, reasonable results can also be achieved with the a100 or v100 GPUs. For example, according to its own documentation, AlphaFold3 runs about twice as fast on an h200 compared to an a100, but both GPUs will give results in a reasonable amount of time. Additionally, smaller models (including open-weight) continue to improve in performance while becoming more feasible to run effectively with fewer computational resources (Stanford HAI, 2025; Mozilla's State of Open Source AI report, 2026).
The HPCC also has dedicated support for data intensive research in fields where computational resources have historically not been used, in the form of the Data Machine. The Data Machine has several high memory nodes with multiple a100 GPUs that are exclusive to its queue. Currently, access to the Data Machine is available only on request, but ICER is seeking more application for the space.
Working interactively with GPUs
Currently, there are two development nodes on HPCC with GPU resources, dev-amd24-h200 and dev-amd20-v100. Each of these nodes have four GPUs of the specified type and all jobs on dev nodes are limited to 2 hours of CPU run time. However, there are alternative ways to work interactively with GPU resources on HPCC.
- You can request an Interactive jobs with the GPU, memory, and time tailored to your development needs. ICER has documentation on available GPU Resources] as well as how to request them.
- Through OnDemand you can run development environments and agents directly on GPU nodes, including:
Getting help
If you are new to HPCC or GPUs, ICER offers in person and online training, including a GenAI on the HPCC workshop