(2023-02-09) Lab Notebooks: Rclone --- Example of copying folder from HPCC to MSU OneDrive

Warning

This is as a Lab Notebook which describes how to solve a specific problem at a specific time. Please keep this in mind as you read and use the content. Please pay close attention to the date, version information and other details.

HPCC to OneDrive with Rclone (2023-02-09)

The documentation to use rclone to copy data from the HPC to their MSU OneDrive account is very complex. Rclone is a powerful tool with lots of options so it can be frustrating to sift through all of the options to know what you need to do. Since I had never used rclone before I thought I would give it a try and type up what I learned.

Step 1: Configure Rclone to connect with OneDrive

First, we need to log on to an interactive desktop using OnDemand. This is required because OneDrive needs to set up an authentication token and it uses a web browser running on the server to do this step. Just log onto "http://ondemand.hpcc.msu.edu/" using your MSU NetID and password. Then click on "Interactive apps" and then click on "Interactive desktop" use the default settings which should be something like the following:

Number of hours 4
Number of cores per task 1
Amount of memory 4gb

Then you can just hit launch and wait for the job to start. You will be taken to a screen with a job ID number and the word "Queued" in the upper right corner. It often doesn't take long the word "Queued" will change to "Starting" and then "Running". A button will appear that says "Launch Interactive Desktop". Click that button and a new window will appear in your browser with the desktop running.

Then next step is to run the "rclone config" command from a terminal. To get to a terminal go up to the start menu at the stop of the desktop. Click on "Applications-->System Tools-->Terminal" and a terminal window will appear. Type the following command to load rclone in the terminal:

module load rclone

Run the rclone config command (you only need to do this once). I typed the following:

n
remote
onedrive
<<enter>>
<<enter>>
n
y
onedrive
0
y
y
q

During this process a window browser window will pop up which may include a button. Just hit accept. You can close the browser when it says "Success". Here is a slightly more detailed breakdown of the command I used when configuring:

n (create new remote)
remote (name of the new remote. We are just calling it remote but you can use whatever name)
onedrive (storage type of remote. In this case onedrive)
<enter> (Just using default ClientID of an empty string)
<enter> (Just using default client_secret of an empty string)
n (No we don't want to use advanced settings)
y (Yes we do want to use automatic configuration, this is when the browser window will pop up)
onedrive (Pick your Onedrive Personal or Bisness account)
0 (Pick the first onedrive account. This assumes you only have one and it is the first in the list)
y (Yes we want to use our MSU onedrive which will have a URL similar to https://michiganstate-my.sharepoint.com/personal/userid_msu_edu/Documents)
q (quit the rclone config)

Step 2: Run Rclone commands to create and copy files to OneDrive

Now that the rclone authentication is configured we can move on to actually using the rclone command. As an example I want to create a directory on my OneDrive (called testdir) and then copy the contents of a directory on the HPC to that directory. Although I should be able to just run the rclone commands from the terminal I am going to run the command inside a job using the following job script.

#!/bin/bash
#SBATCH --mem=2gb
#SBATCH --time=4:00:00
module load Rclone
rclone mkdir remote:testdir
rclone copy ~/UserCode/colbrydi/see-benchmark/see-segment/Image_data/ remote:testdir

I named this file "rclone.sb" and submitted the job to the cluster using the sbatch rclone.sb command. Seemed to work for me. Please let us know if you found this helpful.

Dirk Colbry
Director of User Support
Institute for Cyber-Enabled Research