Skip to content

Running Jupyter Notebooks on the HPCC through VS Code

VS Code supports Jupyter notebooks. This how-to guide will show you how to run these notebooks on the HPCC while using VS Code on your personal/local computer.

Prerequisites

  1. You'll need to connect VS Code to the HPCC over SSH. Pay special note that this requires setting up SSH tunneling.
  2. Building off the SSH tunneling needed for the previous step, you'll also need to set up tunneling to compute nodes.
  3. (Optional) Set up SSH keys to connect to the HPCC without needing to enter a password. For Windows users, use the MobaXterm GUI instructions (or ensure that your private SSH key is saved to C:\Users\<Account_Name>\.ssh\id_rsa).
  4. Install either the jupyter package or (if you don't plan on using Jupyter through OnDemand) the ipykernel package to the Python environment you plan on using. We recommend installing jupyter to avoid future confusion.
  5. Install the Python and Jupyter extensions in VS Code.

Start an Interactive Job

If we want to run Python and/or Jupyter on the HPCC, it's best to do so from an interactive job. This will give us a compute node on which to run our calculations. While a development node may seem more convenient, remember that jobs using more than 2 hours of CPU time are automatically cancelled! For example, we can request a single CPU for four hours: 

1
salloc -t 04:00:00 --mem=4GB
It is strongly suggested that you request a few GB of memory. The default is only 750 MB.

Once your interactive job starts, run the command hostname; e.g. 

1
2
bash-4.2$ hostname
lac-046
to get the name of the compute node you have been allocated. Copy or write down the output of this command for later.

Connect VS Code to the Interactive Host

Inside VS Code, press F1 and select 'Remote-SSH: Connect to Host...' You may need to start typing this option to get it to appear.

Screenshot of the VS Code command entry box with Remote-SSH: Connect to Host... highlighted

Note

If this option is not available, please follow the steps in this how-to guide.

Instead of selecting a host from the drop down list, enter the host of your interactive job into the textbox.

Screenshot of the VS Code SSH host selection box with "lac-046" entered into the textbook

A new VS Code window will pop up. Follow any of the prompts that follow, selecting "Continue" and "Linux" as applicable. When you are successfully connect, the bottom left of the VS Code window will show the hostname you just connected to. You may open files and folders as you normally would when connecting VS Code with SSH.

Set your Python Kernel

Open an existing Jupyter Notebook file or create one by going to the File menu, selecting "New File..", and choosing the "Jupyter Notebook" option. With your Notebook open, click "Select Kernel" in the upper right and then choose "Python Environments." If a Kernel was automatically chosen for you, you'll first need to choose "Select Another Kernel."

Screenshot of the VS Code Kernel Selection menu with "Python Environments..." highlighted

Pay attention to the paths given for each entry. In the screenshot below, for example, the "Recommended" Python interpreter corresponds to the base Conda environment. We can also see our other Conda environments (in this example, there is only one called "astro-analysis") as well as system installations of Python in /bin, /usr/bin, and /opt/software. These system installation are likely not the Python versions you want to use.

Screenshot of the VS Code Kernel Selection menu with a long list of Python environments listed. Some are labeled on the right as "Recommended", "Conda Env", and "Global Env"

You may now execute cells in your Jupyter Notebook while running on the HPCC! VS Code will remember your choice of kernel next time you open that workspace.