Skip to content

(2023-08-11) Lab Notebooks: Jupyter and MobaXterm--- Port-Forwarding Servers

Warning

This is as a Lab Notebook which describes how to solve a specific problem at a specific time. Please keep this in mind as you read and use the content. Please pay close attention to the date, version information and other details.

Port-Forwarding for Jupyter Notebooks with MobaXterm (updated 2023-08-11)

The OnDemand system provides a wonderful way to run Jupyter notebooks on the HPCC, but sometimes it may not perfectly meet our needs. Perhaps you are having difficulty setting up the environment in which Jupyter should run, or you want to use the newer JupyterLab instead of the old-style Jupyter notebook server.

If you run jupyter notebook from the command line on the HPCC, you may find that a browser window opens just as if you had run the command in a terminal on your own machine. This browser will be very slow to respond, however, as it has launched via the X Windows System.

The solution is to run our Jupyter Notebook (or JupyterLab) server with port-forwarding, such that the server is running on the HPCC but we use the browser on our local machine to connect to it.

The following information is modified from instructions Philipp Grete wrote for Brian O'Shea's research group.

Pre-Requisites

These instructions assume you are using Windows and have access to MobaXterm on your local machine. They also assume that you have setup an SSH connection to the HPCC in MobaXterm using these instructions.

Setting up a Server on the HPCC

After preparing your local machine, open a terminal and connect to a development node or compute node with an interactive job.

Either from the command line or within a batch script, you should launch your Jupyter server on the HPCC with the following command:

jupyter notebook --port=12345 --ip="*" --no-browser

where 12345 can be a 5-digit number between 10000 and 65535. If you and someone else happen to be using the same port at the exact same time, issues may arise. Simply cancel the server and restart with a new port number.

If using JupyterLab, simply swap notebook for lab.

Connecting to the Remote Server from your Local Machine

Having set up the server on the HPCC, we now need to connect to it from our local machine. There are two steps to this process.

1. Forwarding the gateway node to the development or compute node

Open MobaXterm and start a new SSH session to the HPCC gateway. Take note of which gateway node is next to your username when you login. For example

1
[<username>@gateway-02 ~]$

means that I will use gateway-02 in the next step.

Instead of logging into a development node directly, you will use an extra command flag to forward the port from where Jupyter is running to the gateway. The template for this is:

1
ssh -L<port>:<jupyter-host>:<port> <dev-node>

In our case, let's say that we started Jupyter in an interactive session on lac-250 with port 12345. We can use

1
ssh -L12345:lac-250:12345 dev-intel18

to forward the port. You can use any development node. If you are running the Jupyter server on a development node, you should use it in between the ports.

2. Forwarding the local port to the gateway node

From the MobaXterm's "Tools" menu, choose "MobaSSHTunnel (port forwarding)". Select "New SSH tunnel" from the window that opens.

In the next Window, you will fill out three sections:

  1. Between "Local clients" and "My computer with MobaXterm"
    • Forwarded port: This can be any port number but to reduce confusion, you can use the same port number that you started Jupyter with, in our example, 12345.
  2. Between "My computer with MobaXterm" and "SSH server"
    • "SSH server": gateway-##.hpcc.msu.edu where gateway-## is the gateway node you logged into in the previous step. In the example above, we used gateway-02.
    • "SSH login": Your HPCC username
    • "SSH port": 22
  3. Between "SSH server" and "Remote server"
    • "Remote server": localhost
    • "Remote port": The port you used to launch the Jupyter server with. In our case, 12345.

Opening the Jupyter notebook on your local computer

When you launched the Jupyter server, you should have been given a URL that starts with http://127.0.0.1. This is the URL from which you should access your server; copy and paste it into the browser on your local computer. It should already have the notebook token as part of the URL, but if you are asked for it, copy the string of numbers and letters that follows ?token=.

Reusing the port forwarding in the future

MobaXterm will save the local to gateway port forwarding configuration that we set up in step 2 above. You can restart it from the "MobaSSHTunnel" menu at any time, however, be aware that you need to match the gateway-## used in this port forwarding with the gateway-## used to forward the port from the compute node to the gateway in step 1. If this gateway node is different than the one saved in MobaXterm, you will need to setup another SSH tunnel in MobaXterm to correspond to that one.

You can also setup MobaXterm tunnels for all four gateway nodes (gateway-00, gateway-01, gateway-02, and gateway-03) using the configuration steps above, and choose the corresponding tunnel depending on where your gateway to compute tunnel starts.