Steps to Migrate to the New Home System Manually

If you have not already moved your files to your home directory on the new system, you will need to follow these steps to do so manually.

This is primarily for users that were using more than 100GB in their home directory as of March 28th, 2025, have not transferred their own files, and as of June 10th, 2025, now have an empty, new home directory. Since all quotas on the new system are a maximum of 100GB of space, you will need to choose what to copy over to the new system.

Frequently asked questions

If you are having issues, please see our Frequently Asked Questions below

Moving files

There are two workflows for moving files. Choose the tab below that best matches your situation.

Reduce your current home directory usage and sync all filesChoose files to copy manually

1. First, log into a development node. Check your home space usage with the powertool file-sizes:

   module load powertools
   file-sizes --detail
   

   Take note of how much space you are using over 100GB.

   Important: don't use the quota command or any other tool to check your usage

   There is a difference between the way that the old and new home systems compress files. The output from the quota command and most other tools will therefore be inaccurate on the old home system.

   Please only use the file-sizes command described above to get usage information that will be more accurate for the new home system.

   You can also use this tool to view how the space is being used within a directory, e.g.,

   file-sizes --detail /mnt/ufs18/gpfs-home/$USER/project
   

   to break down the space usage within project directory in your old home directory.

2. Move excess files manually. There are a few choices:

   • Move files to a shared research space. If your PI does not have one that you have access to, please ask them to create one or update your membership using this form. Each PI can request up to 3TB of space for free. See additional information for using research spaces below.
   • Delete files that you are no longer using.
   • Sync files to a non-HPCC location (like a personal computer, OneDrive, etc) and remove them from the HPCC. The recommended tool for this is Globus.

3. After repeating Step 1 and verifying that you are using less than 100GB in your home directory, you can sync the entire contents of your home directory using these steps:

   GraphicallyFrom the command line

   1. Go to Globus
   2. Log in with MSU credentials
   3. Select the File Manager tab
   4. Select the Collection "Michigan State University MSU ICER msu#hpcc msuhpcc" on both sides
   5. On the left, navigate to /mnt/ufs18/gpfs-home/<USER> replacing <USER> with your MSU username; if you enter this directly in the "Path" box, press enter
   6. On the right, navigate to /mnt/ffs24/home/<USER> replacing <USER> with your MSU username; if you enter this directly in the "Path" box, press enter
   7. Click the left-side file browser; in the column of icons between the two file browsers, click the icon that looks like an eye in a folder, the second from the bottom (circled in pink below). You should now see hidden files that start with a .
   8. On the left, check the "select all" box (circled in green below) to select all of your files
   9. On the left, click the Start button (circled in red below)
   10. You will receive an email when your sync is complete

   The File Manager setup is depicted in this screenshot:

   

   Use the command:

   rsync -avP /mnt/ufs18/gpfs-home/$USER/ /mnt/ffs24/home/$USER/
   

   Note that the trailing slashes are necessary. If your sync is interrupted for any reason, you can restart it using the same command.

1. Select directories you wish to copy to the new home system.

2. Copy these directories manually using these steps:

   GraphicallyFrom the command line

   1. Go to Globus
   2. Log in with MSU credentials
   3. Select the File Manager tab
   4. Select the Collection "Michigan State University MSU ICER msu#hpcc msuhpcc" on both sides
   5. On the left, navigate to /mnt/ufs18/gpfs-home/<USER> replacing <USER> with your MSU username
   6. On the right, navigate to /mnt/ffs24/home/<USER> replacing <USER> with your MSU username
   7. On the left, check the boxes next to all files/directories you would like to copy
   8. On the left, click the Start button
   9. You will receive an email when your sync is complete

   The File Manager setup is depicted in this screenshot:

   

   Suppose you wish to only copy the directory project from your old home directory to your new home directory. Use the command:

   rsync -avP /mnt/ufs18/gpfs-home/$USER/project /mnt/ffs24/home/$USER/
   

   Repeat for any other directories or files you would like to keep. Note that if you remove any files in your old home and want to resync this directory, you should use the --delete option, like

   rsync -avP --delete /mnt/ufs18/gpfs-home/$USER/project /mnt/ffs24/home/$USER/
   

   This will ensure that these deletions are replicated in your new home directory.

   Transferring files that are too large

   If you transfer files that total more than 100GB to your new home directory, you will exceed your quota on the new system. In this case, you will have to remove those files from your new home directory before trying again with fewer files.

3. Note that all files that you do not copy will eventually be removed from the system.

Tips

Managing files graphically via OnDemand

You can also use OnDemand's file browser to perform many operations like moving, copying, and deleting files. You can also download files to your personal computer, though this is limited to files that are 1GB or smaller.

However, note that the file sizes reported on OnDemand may be inaccurate on the new home system. Please use the file-sizes powertool, e.g., by starting a new command line via the Development Nodes tab, and running the commands

module purge
module load powertools
file-sizes --detail

Managing files on the command line

• To move files on the command line use the mv command:

  mv ~/project ~/other_project_location
  

  Don't use mv with research spaces

  Using mv with a research space may preserve undesired group ownership. It is better to copy to a research space first (e.g., using cp), then delete the original version (e.g., using rm).

• To copy files on the command line, use the cp command. When you are copying a directory, you need to use the -r option to copy everything inside:

  cp -r ~/project /mnt/research/my_group
  

  You can also use the rsync command as shown above for a more robust option.

• To remove files on the command line, use the rm command. When you are removing a directory, you need to use the -r option to remove everything inside. Be careful with using rm because deletions cannot be undone:

  rm -r ~/project
  

Moving files to research spaces

We recommend checking with your PI to see if there are any conventions for moving files to your shared research space (e.g., putting your files underneath a directory with your username).

The most important considerations when using research spaces are to:

1. Use the newgrp command to change to the corresponding research group before copying any files
2. Copy files and remove the originals rather than move them directly

For example, to move the directory ~/my_project from the home space to the research space /mnt/research/my_group, use the commands:

newgrp - my_group  # Change group to match the research space
cp -r ~/my_project /mnt/research/my_group  # Copy files
ls /mnt/research/my_group  # Check files transferred correctly
rm -r ~/my_project  # Remove the originals

Instead of using the cp command, you can also use the more robust rsync command. However, rather than using the -avP options as described above, use the following options:

newgrp - my_group
rsync -rlptoDvP ~/my_project /mnt/research/my_group

This ensures that the group ownership of the files will change to match the research space group. Not doing so will result in "Disk quota exceeded" errors.

Before moving files to research spaces please see our additional tips for Using Research Spaces.

Frequently asked questions post-migration

The following is a list of common questions or problems for users migrated on or after June 10th.

I can't log in with my SSH keys (especially for rsync.hpcc.msu.edu)

You may not have copied over your .ssh directory that contains your keys to your new home directories. To do so, log in to a development node either by using your password, or the tab in our OnDemand web portal.

Then run the command

cp -r /mnt/ufs18/gpfs-home/$USER/.ssh ~

Alternatively, you can retransfer the existing keys on your computer depending on how you originally set them up:

• Mac/Linux/MobaXTerm terminal
• Using MobaXTerm's graphical user interface

How do I remove the warning on login?

If you were migrated on or after June 10th, you will receive a warning on login directing you to this page. To remove this login, you can copy your old .bashrc file to your new home directory with

cp /mnt/ufs18/gpfs-home/$USER/.bashrc ~

If you would rather use the fresh .bashrc in your home directory, use the command

sed -i '/^#BEGINFFS24MIGRATION/,/^#ENDFFS24MIGRATION/d' ~/.bashrc

to remove the section that adds the warning on login. If you find that you are missing configuration that was set in your old bashrc, please use the cp command above instead.

Note that this can also help with issues logging in via SFTP servers like FileZilla and WinSCP, as well as starting new remote sessions via VS Code.

I cannot log into an SFTP file transfer program (like FileZilla or WinSCP) or VS Code

The warning on login can stop these programs from being able to connect. Please follow the steps above.

After copying over my conda directory, I can no longer activate my conda environments

Suppose your conda environments are stored in ~/.conda/envs (as is the case when you use the system Miniforge3 module) Run the command:

module load Miniforge3
conda config --append envs_dirs ${HOME}/.conda/envs

If you installed conda yourself, your environments may be in a different directory. Common options are ~/miniforge3/envs, and if you are using Anaconda or Miniconda (which ICER no longer recommends or supports), ~/anaconda3/envs or ~/miniconda3/envs, you will need to load the Conda/3 module and substitute the corresponding directory below:

module load Conda/3
conda config --append envs_dirs ${HOME}/miniforge3/envs  # substitute your conda env directory here

Some of my files are no longer executable or have the wrong permissions after transferring with Globus

Some users have reported issues with executables and permissions after using Globus. Having these users resync old files using rsync has helped in some circumstances. Please try resyncing the relevant directories containing those files using the steps above under the "Choose files to copy manually" and "From the command line" tabs.

Even though my file usage reported less than 100GB and 1 million files with the file-sizes tool, I exceed my quota on the new system when transferring files

Some users have reported that their quota usage increases over the 100GB and 1 million file limit on the new system while transferring, especially with directories containing conda environments.

First, you can try transferring all files except your conda environments, then try reinstalling them on the new system using our steps here. We recommend using the system installed Miniforge3 module to access the conda command.

If you are still having trouble, please contact us via our contact form to help get your files transferred over. You will then need to reduce your file usage on the new system based on quota usage after transferring.