(2023-08-11) Lab Notebooks: Jupyter and MobaXterm--- Port-Forwarding Servers
This is as a Lab Notebook which describes how to solve a specific problem at a specific time. Please keep this in mind as you read and use the content. Please pay close attention to the date, version information and other details.
Port-Forwarding for Jupyter Notebooks with MobaXterm (updated 2023-08-11)
The OnDemand system provides a wonderful way to run Jupyter notebooks on the HPCC, but sometimes it may not perfectly meet our needs. Perhaps you are having difficulty setting up the environment in which Jupyter should run, or you want to use the newer JupyterLab instead of the old-style Jupyter notebook server.
If you run
jupyter notebook from the command line on the HPCC, you may find
that a browser window opens just as if you had run the command in a terminal on
your own machine. This browser will be very slow to respond, however, as it has
launched via the X Windows System.
The solution is to run our Jupyter Notebook (or JupyterLab) server with port-forwarding, such that the server is running on the HPCC but we use the browser on our local machine to connect to it.
The following information is modified from instructions Philipp Grete wrote for Brian O'Shea's research group.
These instructions assume you are using Windows and have access to MobaXterm on your local machine. They also assume that you have setup an SSH connection to the HPCC in MobaXterm using these instructions.
Setting up a Server on the HPCC
Either from the command line or within a batch script, you should launch your Jupyter server on the HPCC with the following command:
jupyter notebook --port=12345 --ip="*" --no-browser
12345 can be a 5-digit number between 10000 and 65535. If you and
someone else happen to be using the same port at the exact same time, issues
may arise. Simply cancel the server and restart with a new port number.
If using JupyterLab, simply swap
Connecting to the Remote Server from your Local Machine
Having set up the server on the HPCC, we now need to connect to it from our local machine. There are two steps to this process.
1. Forwarding the gateway node to the development or compute node
Open MobaXterm and start a new SSH session to the HPCC gateway. Take note of which gateway node is next to your username when you login. For example
means that I will use
gateway-02 in the next step.
Instead of logging into a development node directly, you will use an extra command flag to forward the port from where Jupyter is running to the gateway. The template for this is:
In our case, let's say that we started Jupyter in an interactive session on
lac-250 with port
12345. We can use
to forward the port. You can use any development node. If you are running the Jupyter server on a development node, you should use it in between the ports.
2. Forwarding the local port to the gateway node
From the MobaXterm's "Tools" menu, choose "MobaSSHTunnel (port forwarding)". Select "New SSH tunnel" from the window that opens.
In the next Window, you will fill out three sections:
- Between "Local clients" and "My computer with MobaXterm"
- Forwarded port: This can be any port number but to reduce confusion, you
can use the same port number that you started Jupyter with, in our
- Forwarded port: This can be any port number but to reduce confusion, you can use the same port number that you started Jupyter with, in our example,
- Between "My computer with MobaXterm" and "SSH server"
- "SSH server":
gateway-##is the gateway node you logged into in the previous step. In the example above, we used
- "SSH login": Your HPCC username
- "SSH port":
- "SSH server":
- Between "SSH server" and "Remote server"
- "Remote server":
- "Remote port": The port you used to launch the Jupyter server with. In
- "Remote server":
Opening the Jupyter notebook on your local computer
When you launched the Jupyter server, you should have been given a URL that
http://127.0.0.1. This is the URL from which you should access
your server; copy and paste it into the browser on your local computer. It
should already have the notebook token as part of the URL, but if you are asked
for it, copy the string of numbers and letters that follows
Reusing the port forwarding in the future
MobaXterm will save the local to gateway port forwarding configuration that we
set up in step 2 above. You can restart it from the "MobaSSHTunnel" menu at any
time, however, be aware that you need to match the
gateway-## used in this
port forwarding with the
gateway-## used to forward the port from the compute
node to the gateway in step 1. If this gateway node is different than the one
saved in MobaXterm, you will need to setup another SSH tunnel in MobaXterm to
correspond to that one.
You can also setup MobaXterm tunnels for all four gateway nodes (
gateway-03) using the configuration steps
above, and choose the corresponding tunnel depending on where your gateway to
compute tunnel starts.