Local File Systems

The local file systems are available on each cluster compute node and development node. These file systems are directly connected to each node and may be faster than the home, research, and scratch filesystems which must be accessed over the network.

There are two kinds of local file systems: the local hard drive (accessible from either /tmp or /mnt/local) and RAMDISK space (accessible from /dev/shm). Read more about each type of storage below.

Warning

Please limit the use of the local spaces. It is also used for MPI by the MPI runtime to implement fast communication between MPI processes. Users are advised to clean up the space after use. Files that over 2 weeks old will be removed without notice. If the space is over 90% full, we may clean up unused files without notice.

Local Hard Drive

Each node has local hard drive storage accessible from either /tmp or /mnt/local. The files on this space can be accessed locally on each node without going through network. This space is a good choice for jobs using a single node or multiple nodes where I/O is processed only on each node's local file.  When network traffic is high, using this space will likely allow your program to run faster than running on Home, Research or Scratch space.

Please note that this local space is shared with all processes running on the same node and there is no direct I/O from other nodes. The space also has no auto backup. It should be used as temporary storage space. When the execution of programs in a job is completed, any useful files in this space should be saved back to Home or Research space.

To ensure the continued, reliable operation of each node, all user accounts will be prevented from writing data to local storage that consumes more than 95% of the /tmp directory. Once a user account's usage of the /tmp directory exceeds 95% or the total available space for that directory, it will be locked and prevented from writing any additional data to the /tmp directory until files and data are removed bringing the user account back under 95% space utilization.

Note

In addition to the space restrictions on /tmp, local user account quotas are also in place for another shared directory, the /var directory. This directory is rarely accessed directly by user accounts, and shouldn't be a concern for most users of the HPCC; however, if you receive any errors that your user account has exceeeded the /var directory quota, please contact us right away, and we'd be glad to help troubleshoot the error with you.

$TMPDIR

The environment variable $TMPDIR references a directory that is automatically created when your job starts on the local node. This directory is automatically deleted after the job finishes. It is accessible from both /mnt/local/$SLURM_JOBID and /tmp/local/$SLURM_JOBID. Since local hard drive storage is shared amongst all jobs on a node, using $TMPDIR is a convenient way to keep your files separated and organized.

Any files saved to $TMPDIR should be transferred to a Home or Research space before the job ends.

RAMDISK

RAMDISK space is a “logical” storage space; it sits inside a node’s RAM, not the hard drive. Linux supports a system tool that provides an interface for users to intercept the I/O requests to /dev/shm with memory operations.  We may think of it as a virtual disk in memory.

Due to this nature, access to this space is actually access to RAM. Since the bandwidth of the access is much higher,  the I/O operations are considerably faster than the local hard drive space. However, since programs take up some of the node's memory, the usable RAM space for program execution becomes less.

This space is good for programs that do not require large memory and perform very frequent I/O on small files.