Skip to content

Parallel Filesystems and I/O

The HPCC provides shared, high-performance storage using parallel filesystems, including IBM Storage Scale (GPFS), and non-parallel filesystems such as VAST. While VAST is not a traditional parallel filesystem, both of these systems are designed to allow many compute nodes and users to access large amounts of data concurrently, and it can be thought of as such for the purposes of this documentation.

Parallel filesystems like GPFS provide much greater scalability and aggregate performance than storage attached to an individual computer. However, their performance characteristics are also different from a local disk. The way an application reads and writes files can have a significant impact not only on that application, but also on other users of the shared storage system.

What is a Parallel Filesystem?

On a typical desktop or workstation, an application accesses a filesystem backed by storage dedicated to that system. On the HPCC, thousands of processes running across many compute nodes may access the same shared filesystem at the same time.

A parallel filesystem is designed to distribute this workload across multiple storage resources. This allows the system to support:

  • Large datasets
  • Many compute nodes accessing storage simultaneously
  • High aggregate read and write throughput
  • Files that need to be accessible from multiple systems
  • Large parallel applications

This does not, however, mean that every type of file operation becomes faster simply because the filesystem is parallel.

In particular, there is an important difference between transferring a large amount of data efficiently and performing a very large number of individual filesystem operations.

Large I/O vs. Frequent Small I/O

Consider two applications that each write 100 GB of data.

The first application writes a relatively small number of large files:

output/
├── result-001.dat   25 GB
├── result-002.dat   25 GB
├── result-003.dat   25 GB
└── result-004.dat   25 GB

The second application writes the same amount of data using hundreds of thousands of small files or repeated small writes:

output/
├── result-000001.dat
├── result-000002.dat
├── result-000003.dat
├── ...
└── result-500000.dat

Although the total amount of data may be similar, these workloads place very different demands on the filesystem.

Large sequential reads and writes can generally take better advantage of the bandwidth available from a parallel filesystem. A workload consisting of many small or frequent operations requires the filesystem to perform substantially more work managing those operations.

Filesystem Operations Involve More Than Data

Writing a file does not only mean writing its contents.

The filesystem may also need to perform operations involving:

  • File creation
  • Directory updates
  • File metadata
  • Permissions and ownership
  • File size and timestamps
  • Allocation of storage
  • File locking and coordination
  • Opening and closing files
  • Renaming or deleting files
  • Synchronizing changes between components of the storage system

These operations are commonly referred to as metadata operations.

For example, creating 100,000 one-kilobyte files involves much more filesystem activity than writing the same amount of data into a small number of larger files.

Similarly, repeatedly opening a file, writing a small amount of data, closing it, and reopening it may create substantially more filesystem activity than buffering those results and writing them less frequently.

Why This Matters on a Shared System

GPFS and VAST are shared resources.

Your application may be using only a few compute nodes, but its filesystem requests are being handled by storage infrastructure that is simultaneously serving many other jobs and users.

An application that generates an unusually high rate of filesystem operations can therefore affect more than its own job.

Possible symptoms include:

  • Slow file creation
  • Delayed directory listings
  • Slower reads or writes
  • Applications taking longer to open files
  • Interactive commands such as ls, find, or du responding slowly
  • Other jobs experiencing unexpected I/O delays
  • Reduced filesystem performance across multiple compute nodes

This is one reason a job can cause storage performance problems even when it is not writing an especially large amount of data.

The rate and pattern of I/O operations can be just as important as the total number of bytes being transferred.

Parallel Jobs Can Amplify I/O Problems

Parallel applications require particular care because an inefficient I/O pattern can be multiplied by the number of processes in the job.

For example, suppose a job starts 1,000 processes and every process writes a status message to a separate file once per second.

Instead of one write per second, the shared filesystem may receive approximately:

1,000 processes × 1 write/second = 1,000 writes/second

If those writes also involve opening files, updating metadata, closing files, or creating new files, the filesystem workload can be considerably greater than the amount of data being written would suggest.

Increasing the number of CPUs used by an application can therefore make an I/O problem worse rather than faster if every process independently accesses the shared filesystem.

Common examples include:

  • Each process maintaining its own log file
  • Thousands of workers creating individual output files
  • Frequently updated checkpoint files
  • Repeated creation and deletion of temporary files
  • Software repeatedly opening and closing the same files
  • Large directory trees containing enormous numbers of small files
  • Many processes simultaneously accessing the same file or directory
  • Workflows that continuously scan large directory trees

A Common Misconception

It is tempting to assume:

More processes means more parallelism, so file operations should also become faster.

That is not necessarily true.

CPU parallelism and filesystem parallelism are different resources.

A parallel filesystem can provide very high throughput when a workload is able to use it efficiently. However, increasing the number of processes performing small filesystem operations can eventually create contention rather than additional useful throughput.

For this reason, an application that scales well computationally from 10 processes to 1,000 processes may not have an I/O design that scales equally well.

Improving I/O Behavior

When possible, applications should reduce unnecessary filesystem operations.

Some useful strategies include:

Combine small writes

Rather than writing every individual result immediately, accumulate results in memory and write them in larger blocks when the application allows it.

Reduce the number of output files

If an application generates hundreds of thousands or millions of small files, consider whether those results can be combined into fewer, larger files.

Depending on the software and data format, formats designed for structured or parallel data may also be more appropriate than creating one file for every individual result.

Avoid excessive logging

Logging from every worker in a large parallel job can generate an unexpectedly large I/O workload.

Consider:

  • Reducing log verbosity
  • Writing status information less frequently
  • Having a smaller number of processes perform logging
  • Combining logs after processing

Avoid unnecessary open and close operations

If possible, keep a file open while it is actively being used rather than repeatedly opening, modifying, and closing it.

Limit checkpoint frequency

Checkpointing is valuable for long-running jobs, but extremely frequent checkpoints can generate substantial filesystem traffic.

Select a checkpoint interval that balances recovery requirements against the cost of writing the checkpoint.

Be careful with temporary files

Some applications create very large numbers of temporary files while processing data.

When appropriate, use temporary or job-local storage for intermediate data if such storage is available, and copy the final results to shared storage when processing is complete.

Avoid unnecessarily large directory scans

Commands and applications that repeatedly traverse very large directory trees can generate significant metadata activity.

For example, repeatedly running operations similar to:

find /path/to/large/dataset ...

against a directory containing millions of files requires the filesystem to examine a very large amount of metadata each time.

File Size Is Not the Only Consideration

There is no single file size or number of files that is always problematic.

Filesystem performance depends on factors such as:

  • Number of files
  • Size of the files
  • Frequency of operations
  • Number of processes performing I/O
  • Whether processes access the same files or directories
  • Read versus write patterns
  • Directory layout
  • Other activity occurring on the storage system

For example, creating one million files gradually over several days is very different from thousands of compute processes attempting to create those files simultaneously.

For this reason, I/O rate and access pattern should be considered in addition to total storage usage.

Designing Workflows for Shared Storage

When developing or configuring software for the HPCC, consider how its I/O pattern will change as the job becomes larger.

A workflow that performs well with:

4 processes

may behave very differently with:

4,000 processes

If each process performs its own filesystem operations, scaling the computation by a factor of 1,000 may also scale the number of filesystem requests by approximately the same amount.

Before running a workflow at very large scale, it can be useful to test it with a smaller number of processes and examine:

  • How many files are created
  • How frequently files are modified
  • How much data is written
  • Whether every process performs I/O
  • Whether temporary files are necessary
  • Whether output can be consolidated

Remember: The Filesystem Is Shared

The HPCC storage systems are designed to support demanding research workloads, and high levels of I/O are expected.

The goal is not to avoid using the filesystem.

Instead, users should be aware that GPFS and VAST are shared infrastructure. Inefficient I/O patterns can consume filesystem resources disproportionately and may affect jobs belonging to many other users.

For large-scale workflows, especially those involving thousands of processes, large numbers of files, or very frequent writes, consider the I/O design of the application as carefully as its CPU and memory requirements.

If you are unsure whether a workflow may place an unusual load on the shared filesystems, contact HPCC support before running it at large scale. We can help identify potential I/O issues and suggest approaches that are better suited to the storage environment.