This document highlights several simple methods to transfer files to the HPCC home and research directories. There are two main gateway systems for copying files.
hpcc.msu.edu: This is our login gateway. While it can be used for file transfer, it's not intended for high volumes of files. More importantly, the scratch space is not mounted there and so you can't access your files on scratch.
rsync.hpcc.msu.edu: It has access to scratch, and is dedicated to file transfer. Although this gateway is named by the popular Linux "rsync" command, it can be used for "sftp" or "scp" as well. Starting in October 2022, login to the rsync gateway will accept SSH keys as the ONLY authentication method. Username/password won't work. Please refer to the SSH key tutorial for setting up your keypair.
All operating systems
The OnDemand portal is best for transferring files less than ~1 GB in size. For transferring larger files to and from the HPCC, see Large file transfer (Globus)
The most straightforward way to transfer files to and from the HPCC is via our OnDemand web portal. Log in with your NetID at https://ondemand.hpcc.msu.edu and click "Files" to access your different user spaces.
On this page, you can upload, download, rename, and modify most files in storage locations you have access to.
Specific operating systems
Use the tabs below to view the relevant options for your system.
A number of different command-line utilities are available to OS X and Linux users. Each of them has its own advantages.
Using the commandline to connect to the
rsync.hpcc.msu.edu gateway requires SSH key setup. Please refer to the SSH key tutorial for setting up your keypair.
Basic file copy (
A simple command for transferring files between the cluster and another host is
scp. To copy a file from a local directory to file space on the cluster, run a command such as
scp example.txt email@example.com:example_copy.txt
This will copy the file named
example.txtin the local host's current directory to the user's home directory on the cluster, with the copy having the name
example_copy.txt. Leaving the space after the colon blank gives the new file the same name as the original. Note: To transfer a file name with spaces you must put a backslash before each space in your file name, i.e.
scp "My File Name" firstname.lastname@example.org:"My\ File\ Name".
To copy a file from the cluster to your local directory,
scp email@example.com:example.txt ./example_copy.txt
will copy the file named
example.txtfrom the user's home directory on the cluster to the home directory of the local host, naming the new file
example_copy.txt. Leaving the space after the slash blank gives the new file the same name as the original. The
-roption can be used to copy entire directories recursively.
Synchronize directories (
If you are an advanced LINUX/Mac user, there is a useful utility that makes mirroring directories simple. The syntax looks very similar to
<local_dir>on my local computer to
<hpcc_dir>on hpcc, the following command can be run:
rsync -ave ssh <local_dir> firstname.lastname@example.org:<hpcc_dir>
In the above command, rsync will scan through both directories. If any files in the
<local_dir>are newer, they will be uploaded to
<hpcc_dir>. (It is also possible to get rsync to upload ALL different files, regardless of which is newer).
To mirror the HPCC directory to your local system, call
rsync -ave ssh email@example.com:<hpcc_dir> <local_dir>
Please use the
rsynccommand with the option
--chmod=Dg+sto transfer files from a local computer to your research space.
See the following example:
rsync -ave ssh TestDir --chmod=Dg+s <username>@rsync.hpcc.msu.edu:/mnt/research/<GroupName>/
the first time you use
rsync, you might want to add the
-nflag to do a dry run before any files are copied.
Interactive file copy (
When performing several data transfers between hosts, the
sftpcommand may be preferable, as it allows the user to work interactively. Running
from a local host establishes a connection between that host and the cluster. Both hosts can be navigated. For the local file system,
lcdchanges to the specified directory,
lpwdprints the working directory, and
llsprints a list of files in the current directory. For the remote file system, the same three commands are available, minus the leading
l. Also available are commands to change permissions, rename files, and manipulate directories on the remote host. The two key commands are
get <file>, which copies the file in the remote working directory to the local working directory, and
put <file>, which copies the file in the local working directory to the remote working directory. The
quitcommand closes the connection between hosts.
Copy files from Internet (
wgetis a simple command useful for copying files from the Internet to a user's file space on the cluster. Running the line
examplefile.txtto the user's working directory.
An alternate method for transferring files on Windows is MobaXTerm. Installation and setup instructions are available here. Once you are connected to the HPCC, you can use the MobaXterm SCP interface tab to upload and download files, available on the left side of the window.
You can type a path at the top of the SCP interface tab to access anywhere you have permissions on the HPCC e.g. your scratch and research spaces.
MobaXTerm can also be set up for use as a SFTP client to transfer data with the
rsync.hpcc.msu.edu gateway. As with other uses of the
rsync.hpcc.msu.edu gateway, you must have an SSH key pair. Please refer to the SSH key tutorial for setting up your keypair.
The configuration for the SFTP session should look like this:
The settings should be:
- Remote host:
- Username: your HPCC username.
- Select the Advanced Sftp settings tab.
- Use private key: check the box. Click the small file icon to open a file browser and select your private key. Alternatively, type in the path to your private key.
- All other settings can remain default.