Skip to content

The HPCC's Layout

The HPCC is comprised of three different kinds of nodes: the "gateway" and "rsync" entry nodes, development nodes, and compute nodes. In a typical workflow, users will connect to the HPCC through the an entry node, then connect to a development node to compile and test code before submitting jobs to the SLURM queue to be run on a compute node. This workflow is demonstrated in the diagram below.

Each node type is explained in more detail in the following sections. Information on the HPCC's filesystems is available within a separate section.

Diagram of the ICER HPCC layout. From left to right are the user's computer, the gateway nodes, the development nodes, and the compute nodes. These layers are connected by right-pointing arrows.

Entry Nodes

The gateway and rsync nodes are the only nodes directly accessible over the internet. Users connect to these nodes from their personal computers using ssh before accessing other parts of the HPCC.

Gateway Nodes

These nodes are the default accessed via ssh <username> as in the top fork of the diagram above. The gateway nodes are not meant for compilig or running software, accessing the scratch space, or connecting to compute nodes. Users should only use the gateway nodes to ssh to development nodes. Alternatively, users may set up SSH tunneling to automate the process of passing through the gateway to a development node.

Rsync Nodes

These nodes are accessed via ssh <username> as in the bottom fork of the diagram above. Unlike the gateway nodes, the scratch file system is accessbile from the rsync nodes. This is because these nodes are primarily intended for file transfer. Large amounts of data should be transferred via the rsync nodes to avoid slowing down the gateway nodes.

These nodes are named for the popular command line file transfer utility rsync. Users can use this utility to transfer files via the rsync gateway by following this command pattern: rsync <local path> <username><remote path>. Other commands such as scp may also be used with the rsync gateway, or a GUI such as MobaXterm may be used instead.

Development Nodes

From the gateway node, users can connect to any development node to compile their jobs or and run short tests. They may also access files on the scratch file system. Jobs on the development nodes are limited to two hours of CPU time. More information is available on the development node page.

Each development node is configured to match the compute nodes of the same cluster. If you would like your job to be able to run on any cluster (as is the default for the queue; see the section on Automatic Job Constraints) you should not compile with architecture-specific tuning (e.g. -march or -x).


Code compiled on older development nodes (dev-intel14 and dev-intel14-k20) may have errors when running on the latest clusters due to an outdated instruction set. To avoid this, compile your code on a newer development node or specify --constraint=intel14 in your SLURM batch script.

Compute Nodes

ICER maintains several clusters worth of compute nodes. Users submit jobs to the SLURM scheduler which assigns compute nodes based on the resources requested.

A user may see which nodes their job is running on using squeue -u <username>. Not providing a username to squeue will show all jobs currently running on the system. Users may ssh directly to a compute node only if they have a job running on that node. See our page on connecting to compute nodes for more.

Comparison to a personal computer

Laptop/Desktop HPCC Clusters
Number of Nodes 1 979
Sockets per node 1 2, 8
Cores per node 4, 8, or 16 20, 28, 40, 128 or 144
Cores total 4, 8, or 16 50,084
Core Speed 2.7 - 3.5 ghz 2.5-3 ghz
RAM memory 8, 16 or 32 GB 64, 128, 92, 500 GB or 6TB
File Storage 250, 500 GB or 1TB 1TB(Home), 50TB(Scratch)
Connection to other computers Campus ethernet1 Gbit/sec "Infiniband" 100 Gbit/sec
Users 1 ~2,000
Schedule On Demand 24/7 via queue