Tunneling Through Two SSHs

You disabled JavaScript. Please enable it for syntax-highlighting, or don't complain about unlegible code snippets =) This page doesn't contain any tracking/analytics/ad code.

The situation

If your workplace is any similar to mine, you have to open an SSH connection within another SSH connection to remotely access your workstation: Matryosshka!

lucas@laptop ><((("> ssh beyer@company.com
beyer@company.com ><((("> ssh lb@workstation
lb@workstation ><(((">

There are various reasons for such a setup, which we will let admins discuss; we devs just have to accept it as a fact of life.

I'm mostly working on my notebook (I don't like fixed desks too much) and the Wi-Fi happens to be on a different network than the workstations. This makes (at least) two things more complicated than they'd otherwise be:

Copying files using SCP.
Tunneling things like Jupyter (ex-IPython) notebook through SSH.

The latter is especially awesome, since it'd allow me to use the notebook on my notebook (ok, I'll call the latter laptop from now on) while all code runs on the more powerful workstation.

SSH and netcat, a match made in heaven

Netcat (nc) is a nifty tool which, at its core, creates a connection and lets you send anything to it. A bit like a lower-level telnet on steroids. It can be used for many things, the simplest being exploring a network protocol:

><((("> nc lucasb.eyer.be 80
GET / HTTP/1.0
Host: lucasb.eyer.be

[OH CRAP SO MUCH HTML]

After having pressed enter twice (because the HTTP protocol requires that), you should get my home page sent back to you.

Ad-hoc file transfer

Another use-case, just as useful, is ad-hoc file transfer. On the receiving machine, run

><((("> nc -l -p 1337 >out.file

This command instructs netcat to listen on port 1337 and the >out.file tells your shell not to print stdout to the screen but rather write it to the file out.file. (Note that there are multiple versions of netcat in the wild, some of which need you to specify the port with -p 1337 and some of which forbid it. Read your man nc.) Now that netcat is listening, on the sending machine, run:

><((("> nc receiver 1337 <in.file

This opens a connection to receiver (you might want to put an IP address here) on port 1337 and writes the content of in.file to that connection.

There are 3 caveats here:

The receiver needs to be able to listen on said port, i.e. firewalls, routers and ISPs can cause problems.
The receiving end doesn't know when transfer is over.
There is no progress indicator.

It is easy to work around the latter two by using your brain or google.

Back to SSH

Now that we understood netcat, let's go back to our SSH. There are two features of SSH we'll use. The first one is rather trivial: if called with superfluous arguments, SSH will run these as a command on the remote host and then disconnect. For example,

><((("> ssh beyer@company.com ls -l
beyer@company.com's password:
total 12345
drwxr-xr-x  4 user group       4096 date secret_folder
-rw-r--r--  1 user group    7811485 date trade_document.pdf
[...]
><(((">

The second one is slightly more involved: the ProxyCommand option. From the manual:

Specifies the command to use to connect to the server. The command string extends to the end of the line, and is executed with the user's shell. In the command string, any occurrence of '%h' will be substituted by the host name to connect, '%p' by the port, and '%r' by the remote user name. The command can be basically anything, and should read from its standard input and write to its standard output. [...]

This directive is useful in conjunction with nc(1) and its proxy support. For example, the following directive would connect via an HTTP proxy at 192.0.2.0:

ProxyCommand /usr/bin/nc -X connect -x 192.0.2.0:8080 %h %p

This gives us the needed hint. Basically, what it means is that SSH will first run whatever command you're passing (on laptop) and "communicate" to that command through stdin/stdout. Sounds like a fit made in heaven for netcat.

The final command

><((("> ssh -o "ProxyCommand ssh beyer@company.com nc %h %p" lb@workstation

That's the magic command.

Thanks to ProxyCommand, it will first open an SSH connection to company.com in which it will start netcat to connect to workstation. Whatever laptop's SSH wants to send will use the "inner" SSH to travel between laptop and company.com and then netcat to travel between company.com and workstation. If you understood that, it should be obvious that this use of netcat is no security concern.

Making it more convenient

After using this for some time, you'll be annoyed by having to enter such a long command and your password twice, every time. In this case, use one or two SSH keypairs. Or maybe you're only allowed to use SSH with keypairs. In any case, there are plenty of good tutorials online about just that.

Once you've got the keypairs ready, you can add the following to your ~/.ssh/config to ease the pain:

host company
    HostName company.com
    User beyer
    IdentityFile ~/.ssh/company

host workstation
    HostName workstation
    User lb
    IdentityFile ~/.ssh/workstation
    ProxyCommand ssh company nc %h %p

You can now simply use workstation in any ssh (and scp) commands on laptop:

><((("> scp results.csv workstation:/home/lb/results.csv
><((("> ssh workstation
lb@workstation$

Killed by signal 1

Whenever you leave the double-tunnel (using Ctrl+d of course!), you will be greeted by a weird "Killed by signal 1" message:

><((("> ssh workstation
lb@workstation$ [Ctrl+d]
Connection to workstation closed.
Killed by signal 1.
><(((">

This doesn't mean anything bad, but it can become annoying. POSIX signal 1 is SIGHUP, which stands for "hang up" and dates back to the modem days and means that the controlling terminal has been closed. You might have learned the trick of starting a program with nohup to let it outlive your terminal window; nohup just eats up the SIGHUP message that's being sent to the program when you close the window.

In our case, the message comes from the ssh executed in the ProxyCommand option, which tells you it exits because its controlling terminal, the actual SSH command, has been closed. Of course, that's what we did. You can get rid of this message by telling ssh to be quiet (-q) except for errors. So the fixed line would be:

    ProxyCommand ssh -q company nc %h %p

Jupyter (IPython) notebook

Finally, thanks to having set-up your ~/.ssh/config file as above, you can follow any of the tutorials online on how to run the Jupyter (ex-IPython) notebook server on workstation and access it in your browser on laptop.

Here's a quick reference anyways.

You can either connect to workstation and start the notebook server there, ideally in a tmux session so you can reattach to it later:

lucas@notebook ><((("> ssh workstation
lb@workstation ><((("> tmux
lb@workstation ><((("> ipython notebook --no-browser --port=1337

Now type in Ctrl+b, then :det and then hit the return key. You can now disconnect from that SSH session (Ctrl+d) and later reattach to it by typing tmux ls to see the active sessions and then tmux at -t SESSION_ID.

If you just want a quick session, which will die as soon as your laptop disconnects, you can also just run

lucas@notebook ><((("> ssh workstation ipython notebook --no-browser --port=1337

But I'd rather go with the first way so that IPython survives me walking around with the laptop. Now that the server is running, create a local redirect:

lucas@notebook ><((("> ssh -N -f -L localhost:8080:localhost:1337 workstation

This creates a forwarding from localhost:8080 on notebook to localhost:1337 (and here comes the non-obvious part) on workstation. You can optionally drop the first localhost since it's implied, i.e. just write :8080:localhost:1337.

The mnemonic I use to remember that command is the national football league, where the football itself is small compared to our (European) football, hence lowercase f.

Have fun!

PS: Tasty copy-pasta for self: ssh -N -f -L :8080:localhost:1337 grolsch