Published: 6. 11. 2019   Category: GNU/Linux

SSH reverse tunnel and SSH via bastion

If you need to connect to your box hidden behind firewalls and NATs you may end up with setting SSH reverse tunnel. In this case, you need to have SSH access to a computer with public IP address accessible from the public Internet. Sometimes you also need to access computers in private networks via some host usually called bastion.

Setup permanent reverse tunnel

When using SSH tunnels you are usually connecting to a remote server and define some other remote IP address and port to be accessed by a local port. I made a cheat sheet for Networking covering this and I am also using dynamic port forwarding described here.

Reverse tunnel works in a reverse manner, you will tell the remote host to use a local port to access an IP and port at your place.

I am currently using this script to enable reverse SSH tunnel on port 2222 (remember the only root can use privileged ports below 1024). It is using autossh which provides automatic restart when the connection died.

I am having this script saved as /usr/local/bin/autossh_tunnel:

REMOTE_ADDR=...public server here...
LOG_FILE=/var/log/autossh.log     # when started by root

while true
    autossh -v -M 0 -N -o "ServerAliveInterval 59" -o "ServerAliveCountMax 3" \
    -p 22 -i $SSH_KEY \
    -o ExitOnForwardFailure=yes \
    -o StrictHostKeyChecking=no -o "UserKnownHostsFile=/dev/null" \
    sleep 5
done > $LOG_FILE 2>&1

Update REMOTE* variables and provide correct SSH_KEY. If you care, remove StrictHostKeyChecking line to enable fingerprint checking, but then be sure you have key in known_host file.

I am using Fedora Linux and having the tunnel started automatically I needed to put /usr/local/bin/autossh_tunnel & to /etc/rc.d/rc.local and enable it by SystemD: systemctl enable rc-local.

On the server-side, forwarded remote ports are usually bound to the loopback address if you want to expose those ports update /etc/ssh/sshd_config and restart SSH daemon (systemctl restart sshd.service):

GatewayPorts yes

You will also need to update other parts of your infrastructure like a local firewall or security group to expose the used port.

Setup permanent reverse tunnel with systemd

I was notified in discussion on reddit that script above can be implemented with systemd this way:

REMOTE_ADDR=...public server here...

Description=Reverse Tunnel
ExecStart=/usr/bin/ssh -f -N -T -R 2000:localhost:22

$ systemctl enable --now tunnel.service

Connect via bastion (jump box)

Bastion is a host that will forward your SSH connection (also forward ports, scp/sftp or rsync) with hosts in different network.

OpenSSH since version 7.3 has CLI option -J (config option ProxyJump) to specify one or more hostnames of the server used for forwarding. More hostnames must be separated by a comma and it will use them in the given order:

ssh -J server1,server2 target
ssh -J user1@host1:port1,user2@host2:port2 user3@target -p port3

It can use short names defined in your $HOME/.ssh/config too or specify different users and port of each host. It is also using your private keys stored on your localhost and it makes jumping super easy.

Older versions of OpenSSH can easily jump just over one box with -W option and ProxyCommand. A quite complex, but real example is below. It is connecting to mybastion with reverse SSH tunnel bind on mybastion's localhost port 2222. It will first connect to the bastion as user alice with private key id_rsa2, then it will run another SSH session with -W and provide remote host (%h) and port (%p):

ssh -o ProxyCommand="ssh -i ~/.ssh/id_rsa2 -W %h:%p" bob@localhost -p2222

This can be defined in $HOME/.ssh/config this way:

Host mybastion
    User alice
    IdentityFile ~/.ssh/id_rsa2
    IdentitiesOnly yes

Host work
    Hostname localhost
    Port 2222
    ProxyCommand ssh mybastion -W %h:%p
    DynamicForward 8008

Once defined in config file, I will simply execute ssh work and I will open SSH shell to my work computer behind NAT/firewall. It can be also used with -D (DynamicForward) for dynamic port forwarding. And it, of course, support sftp, scp, and my favorite rsync --rsh="ssh".

Really old versions of OpenSSH may do similar tricks with -t (RequestTTY) when it will immediately after login execute another ssh command:

ssh -i ~/.ssh/id_rsa2 -t ssh bob@localhost -p 2222

In this case, scp or port forwarding does not work, however, it can still be created with explicit defining of ports between mybastion and target. Find more tricks in OpenSSH wikibook.