Skip to content

(2023-06-30) Lab Notebooks: Jupyter --- Port-Forwarding Servers

Warning

This is as a Lab Notebook which describes how to solve a specific problem at a specific time. Please keep this in mind as you read and use the content. Please pay close attention to the date, version information and other details.

Port-Forwarding for Jupyter Notebooks (updated 2023-06-30)

The OnDemand system provides a wonderful way to run Jupyter notebooks on the HPCC, but sometimes it may not perfectly meet our needs. Perhaps you are having difficulty setting up the environment in which Jupyter should run, or you want to use the newer JupyterLab instead of the old-style Jupyter notebook server.

If you run jupyter notebook from the command line on the HPCC, you may find that a browser window opens just as if you had run the command in a terminal on your own machine. This browser will be very slow to respond, however, as it has launched via the X Windows System.

The solution is to run our Jupyter Notebook (or JupyterLab) server with port-forwarding, such that the server is running on the HPCC but we use the browser on our local machine to connect to it.

The following information is modified from instructions Philipp Grete wrote for Brian O'Shea's research group.

Pre-Requisites

These instructions assume you have access to a terminal on your local machine. Users interested in attempting this with MobaXterm should follow these instruction instead.

As a prerequisite, set up an SSH Key and SSH Tunneling.

Then, add the following to the bottom of your local .ssh/config file (this is the same file edited when setting up SSH tunneling):

1
2
3
    Host lac* vim* skl* nvl* amr* nvf* nif* nal* acm*
    User here_you_put_your_net_id
    ProxyJump intel18

The three-letter names correspond to the kinds of nodes available from each cluster. This will make it possible to SSH to compute nodes from your local machine! This is useful if you would like to run a Jupyter server from an interactive job.

Note

You can only SSH to a compute node if you have job running on that node.

Setting up a Server on the HPCC

After preparing your local machine, open a terminal and connect to a development node or compute node with an interactive job.

Either from the command line or within a batch script, you should launch your Jupyter server on the HPCC with the following command:

jupyter notebook --port=12345 --ip="*" --no-browser

where 12345 can be a 5-digit number between 10000 and 65535. If you and someone else happen to be using the same port at the exact same time, issues may arise. Simply cancel the server and restart with a new port number.

If using JupyterLab, simply swap notebook for lab.

Connecting to the Remote Server from your Local Machine

Having set up the server on the HPCC, we now need to connect to it from our local machine.

In the same terminal that is connected to the HPCC, type ~C (do not press Enter). You should see an ssh> prompt appear.

Warning

These instructions will not work if you use the OnDemand development node terminals to access the HPCC.

You'll type the following information without spaces:

-L<localport>:<remotehost>:<remoteport>

For example, -L12345:lac-250:12345. Then press Enter.

Each piece of information, from right to left, is as follows:

  • <remoteport> is the port you entered when launching your Jupyter server. For this example, it was 12345.
  • <remotehost> is the name of the host your server is running on. This could be a development node (e.g. dev-amd20) or it could be a compute note (e.g. lac-250). For this example, we'll assume a compute node. This is why setting up SSH Tunneling is required: it's the only way to connect directly to development and compute nodes!
  • <localport> is another 5-digit number like <remoteport>. For simplicity, we'll make it the exact same as <remoteport> (but it doesn't have to be).

When you launched the Jupyter server, you should have been given a URL that starts with http://127.0.0.1. This is the URL from which you should access your server; copy and paste it into your browser. It should already have the notebook token as part of the URL, but if you are asked for it, copy the string of numbers and letters that follows ?token=.