Skip to content

File Backups

There are multiple locations to store your files on the HPCC. While some file systems take automated snapshots of files, none of the file systems on the HPCC offer backups.

If you have not implemented your own backup solution, there is no guarantee that your files can be recovered in the case of modification or deletion.

Backup strategies

All important data should be backed up according to the "3-2-1 rule":

  • You should make 3 copies of any important file.
  • These copies should be stored on at least 2 storage devices.
  • 1 of these copies should be offsite (i.e., in a physically different location than where the original is).

Example of the 3-2-1 rule

You have following three copies of an important dataset:

  1. In a home or research space on the HPCC
  2. On your personal computer
  3. In your MSU OneDrive storage

This situations satisfies the 3-2-1 rule because:

  • You have 3 copies
  • Each copy is in different location on a different type of machine (this satisfies the "different storage devices" and "offsite" requirements at once)

You can use Globus to implement this strategy on the HPCC:

NON-example of the 3-2-1 rule

You have the following three copies of an important dataset:

  1. In your HPCC home space
  2. In an HPCC research space
  3. In your HPCC scratch space

While you (seemingly) have three different copies, none are stored offsite (i.e., in a different location than the HPCC). Additionally, your copy stored in scratch will be purged after 45 days, so it doesn't count!

This could also be considered a violation of the "two different storage devices" rule. While the file systems are technically different, a disaster that affects one could just as easily affect the other two.

NON-example of the 3-2-1 rule

You have the following copies of an important dataset:

  1. On your personal computer
  2. In an HPCC research space

While this meets the "two different storage devices" and "one offsite copy" rules, you still need a third copy of your data.

Importantly, these copies should reflect the state of the files when you want to recover them. If you have made new changes since your last backup, these changes are unrecoverable. You need to make new copies satisfying the 3-2-1 rule.

Where can I backup my files?

There are many options for locations to store copies of files:

  • A personal computer
  • A lab computer
  • An external hard drive
  • Filesystems on the HPCC
  • Your MSU OneDrive account
  • Other options at MSU: see the Data Storage Finder

However, not all locations are appropriate for all data. When assessing locations for storing backup copies of your files, you should consider:

  • Does your data have any restrictions which do not allow it to be stored in certain places? E.g., if your data contains any Personally Identifiable Information (PII), it should never be stored on the HPCC.
  • How much data do you have? Often, large datasets may not fit on a personal computer. You may need to consider purchasing access to additional storage (e.g., MSU's Elastic Storage).
  • How robust are your storage locations? While your personal laptop or an external hard drive may be convenient, they can easily be broken or stolen. Other options like cloud storage can have their own backup solutions, which can give you additional coverage.

No matter what options you choose though, you should always follow the 3-2-1 rule.

Snapshots are not backups

The Home and Research file systems take automated snapshots of the state of files at regular intervals. However, these are not true backups, and cannot be relied on to restore your files. In particular:

  • Snapshots are temporary: This is the most important consideration. Snapshots are taken regularly and only stored for a certain period of time. As new snapshots get taken, old ones get rotated out, and are unavailable.
  • Snapshots are automated outside of your control: Snapshots are taken automatically at set intervals. If you make a change you would like to recover between snapshot intervals and then overwrote the file, you cannot recover that change. You need to be in charge of when your files are backed up.

At the end of the day, snapshots are primarily a disaster recovery mechanism. If there is a disaster which makes a set of files wholly unavailable, snapshots can be used as a temporary solution to keep you working. As a side-effect, you may be able to restore a file from a given snapshot. However, for the reasons above, you cannot rely on snapshots to recover files to your desired state.