(2023-02-09) Lab Notebooks: Rclone --- Example of copying folder from HPCC to MSU OneDrive
This is as a Lab Notebook which describes how to solve a specific problem at a specific time. Please keep this in mind as you read and use the content. Please pay close attention to the date, version information and other details.
HPCC to OneDrive with Rclone (2023-02-09)
The documentation to use rclone to copy data from the HPC to their MSU OneDrive account is very complex. Rclone is a powerful tool with lots of options so it can be frustrating to sift through all of the options to know what you need to do. Since I had never used rclone before I thought I would give it a try and type up what I learned.
Step 1: Configure Rclone to connect with OneDrive
First, we need to log on to an interactive desktop using OnDemand. This is required because OneDrive needs to set up an authentication token and it uses a web browser running on the server to do this step. Just log onto "http://ondemand.hpcc.msu.edu/" using your MSU NetID and password. Then click on "Interactive apps" and then click on "Interactive desktop" use the default settings which should be something like the following:
1 2 3
Then you can just hit launch and wait for the job to start. You will be taken to a screen with a job ID number and the word "Queued" in the upper right corner. It often doesn't take long the word "Queued" will change to "Starting" and then "Running". A button will appear that says "Launch Interactive Desktop". Click that button and a new window will appear in your browser with the desktop running.
Then next step is to run the "rclone config" command from a terminal. To get to a terminal go up to the start menu at the stop of the desktop. Click on "Applications-->System Tools-->Terminal" and a terminal window will appear. Type the following command to load rclone in the terminal:
rclone config command (you only need to do this once). I typed the following:
1 2 3 4 5 6 7 8 9 10 11 12
During this process a window browser window will pop up which may include a button. Just hit accept. You can close the browser when it says "Success". Here is a slightly more detailed breakdown of the command I used when configuring:
1 2 3 4 5 6 7 8 9 10 11
Step 2: Run Rclone commands to create and copy files to OneDrive
Now that the rclone authentication is configured we can move on to actually using the rclone command. As an example I want to create a directory on my OneDrive (called testdir) and then copy the contents of a directory on the HPC to that directory. Although I should be able to just run the rclone commands from the terminal I am going to run the command inside a job using the following job script.
1 2 3 4 5 6
I named this file "rclone.sb" and submitted the job to the cluster using the
sbatch rclone.sb command. Seemed to work for me. Please let us know if you found this helpful.
Director of User Support
Institute for Cyber-Enabled Research