How do you connect to 8 computers, dispersed around a city, through 3G dongles?
The biggest hurdle we faced with Shadowing was repairing and altering lamp heads while they were in situ.
Yep, try programming up here!
We did have the option of calling out the ever-helpful SSE to have a look inside the lamp heads. However, we had no real way of affecting or altering the software. To complicate matters further, our only potential method of communicating to the lamp heads was through a 3G dongle.
Solution
Normally, if you want to communicate to a remote computer on a standard network, you would first find its IP address and then SSH into the unit:
ssh computerUser@HostOrIPAddress
However, 3G dongles do not allow you to connect to them via their public IP Address as they are hidden behind a firewall, rendering this method useless.
Our solution was to create a Reverse SSH Tunnel system where the lamp heads phoned home to a server on our network, allowing us control over the lamphead’s computer from our office. Crucially, if someone got hold of the device and hacked into the SSH tunnel, they wouldn’t be able to get onto our networks: the tunnel is a dead-end.
This post will provide step-by-step instruction of how to setup your own Reverse SSH Tunnel server using an Amazon AWS Server.
Create yourself an Instance
To setup an Amazon Server, follow the guide below. If you already have a suitable server (with root access), you can skip ahead to the next section.
- Sign-up to Amazon AWS Services
- On the dashboard, click Launch a New Instance
- Choose Ubuntu Server 14.04LTS
- Choose t2.micro for instance type
- Add Storage (Standard storage is fine)
- Add Tag Instance Name: Dead-End-Server
- Configure Security Group:
- Create a new security group
- Security Group Name: MyAccess
- By default there will be an SSH rule added. However, change its source to My IP from the drop down box
- Click Add new rule:
- Choose HTTP; again, change its source to My IP
- Click Review and Launch
- Click Launch
- Create a new key pair
- Give the permissions key a Name: awspemkey then download. Do not lose this key!
- Then click Launch
- Make a note of the public IP address of your instance.
To connect to your instance you will need to configure some SSH settings on your computer. The following is for Mac OS/X, but there are usually plenty of documentation/guides available on the web for you to use your favourite client…
- Open Terminal.app
- Navigate to your SSH folder:
cd ~/.ssh/
- Create a Folder called AWSKEYS:
mkdir AWSKEYS
- Move the .pem key, downloaded above, to this folder:
mv ~/Downloads/awspemkey.pem ~/.ssh/AWSKEYS/
- Change the permissions of the .pem file:
chmod 400 ./AWSKEYS/awspemkey.pem
- Then edit your config file:
nano config
- Type the following in below anything else in the file; don’t forget to substitute IPAddress for your instance’s IP, noted above:
Host dead-end-server Host IPAddress User ubuntu IdentityFile ./AWSKEYS/awspemkey.pem
- Now SSH to the server:
ssh dead-end-server
- You are now connected to your Amazon EC2 Server!
Server Setup
To setup the Tunnel system you’ll need to run all of the following as the root user. For the Amazon Instance created above, this should simply be a case of running the following having SSH’d in:
sudo -i
- Install SSHGuard to help protect your instance:
apt-get install sshguard
- Add your remote IP address to the SSHGuard’s whitelist. This ensures that you will not be locked out of the system:
nano /etc/sshguard/whitelist
- Alter the SSH daemon’s config file to harden it a little:
nano /etc/ssh/sshd_config
- Add the the following lines to the file:
PermitRootLogin no TCPKeepAlive yes ClientAliveInterval 30 ClientAliveCountMax 99999 GatewayPorts yes AllowTcpForwarding yes ChallengeResponseAuthentication no PasswordAuthentication no
- Create a new group to collect all the VPN-tunnels and permissions:
sudo groupadd -g 2000 ssh-vpn-tunnels
- Alter the ownership and permissions of root’s directories:
chown root:ssh-vpn-tunnels /root/.ssh chmod 0751 /root /root/.ssh/
- If there isn’t one already, create a blank SSH config file (used later) and alter its permissions:
touch /root/.ssh/config chown root:ssh-vpn-tunnels /root/.ssh/config chmod 0640 /root/.ssh/config
Connection management
We had eight lamp posts within the project and, at the time our server was setup, we were also considering additional deployments; because we wanted to keep them separated, we came up with the following convention:
- Each deployment (henceforth referred to as “projectname”) has a user-account with a specific user-ID (e.g. 2220)
- Each of up to nine lamp posts within a deployment had an ID (e.g. 1)
- Each lamp post would setup a reverse-tunnel on a port of [projectname’s UID] + [LampPostID] (e.g. 2220+1 = 2221)
The following script allowed us to see, at a glance, the status of each connection within a project.
- Create a new file:
nano /root/list-connections.awk
- Then paste in the following:
#!/usr/bin/awk -f BEGIN { # Extract the processes EUID, then create a prefix-version iEUID = PROCINFO["euid"]; iEUIDPrefix = substr(iEUID, 0, 3); # Go find all VPN ports sCommand = "netstat -4lnt"; while (( sCommand | getline) > 0) { # Found a port? if(match($4, /:(2[0-9]{3,})$/)) { iCurrentPort = substr($4, RSTART + 1); # If root, or port-prefix matches, note the port if(iEUID == 0 || substr(iCurrentPort, 0, 3) == iEUIDPrefix) { abPortActive[ iCurrentPort ] = 1; } } } close(sCommand); # Now find all the known hosts sCurrentHost = "UNKNOWN"; iCurrentPort = -1; while ((getline < "/root/.ssh/config" ) > 0) { # Look for Host entries if(match($1, /^Host$/)) { # Emit any exiting blocks if(iCurrentPort != -1) { printf("%s\t%4d\t%s\n", abPortActive[ iCurrentPort ] ? " UP " : "down", iCurrentPort, sCurrentHost); abPortActive[ iCurrentPort ] = "DISPLAYED"; iCurrentPort = -1; } # Note the new current host sCurrentHost = $2; # Look for Port entries } else if(match($1, /^Port$/)) { # If root, or port-prefix matches, note the port if(iEUID == 0 || substr($2, 0, 3) == iEUIDPrefix) { iCurrentPort = $2; } } } close(sFile); # Emit any trailing blocks if(iCurrentPort != -1) { printf("%s\t%4d\t%s\n", abPortActive[ iCurrentPort ] ? " UP " : "down", iCurrentPort, sCurrentHost); abPortActive[ iCurrentPort ] = "DISPLAYED"; } # Emit any unknown ports for(iCurrentPort in abPortActive) { if(abPortActive[ iCurrentPort ] != "DISPLAYED") { printf(" UP \t%4d\tUNKNOWN\n", iCurrentPort); } } }
- Change the ownership and permissions of the script:
chmod 0751 list-connections.awk chown root:ssh-vpn-tunnels list-connections.awk
User Settings
- Create the user for the deployment/project:
useradd -u 2220 -g 2000 -M -N projectname -c 'brief description of your project' (-u 2220: Set the user ID for the project -g 2000: Set the group ID; this is what we used above for ssh-vpn-tunnels -M: Do not create the home-directory -N: Do not create a user-group projectname: The username -c: The GECOS/description field )
- Create the home and SSH folders for your user:
mkdir -p ~projectname/.ssh
- Change the permissions:
chown -R projectname:ssh-vpn-tunnels ~projectname chmod 751 ~projectname chmod 700 ~projectname/.ssh chown projectname:ssh-vpn-tunnels ~/projectname/.ssh/authorized_keys chmod 600 ~projectname/.ssh/authorized_keys
- Link the SSH config file from root:
ln -s /root/.ssh/config ~projectname/.ssh/config
- Link the list-connection.awk file:
ln -s /root/list-connections.awk ~projectname/
- To make connecting to the Remote Units easier, edit root’s SSH config file:
nano /root/.ssh/config
- Add an entry for each Remote Unit you will want to connect to. To work with the AWK script above, use a port number based on user-ID plus 1 to 9; i.e. 2221 through 2229:
Host <Remote-Unit-Name> Hostname localhost User username Port <port> ServerAliveInterval 7
For example:
Host Lamppost1 Hostname localhost User ubuntu Port 2221 ServerAliveInterval 7
Remote Unit Setup
This is the computer you put out in the wild. You’ll also need to run all of the following as the root user.
- Generate an SSH key:
ssh-keygen
- Follow the on-screen instructions, do not enter a password, leave it blank.
- Now you need to transfer the public part of the key (contained within ~/.ssh/id_rsa.pub) from the Remote Unit to the Tunnel Server – for example copy+paste it between two SSH-terminals. Add it to the end of the Tunnel Server’s ~projectname/.ssh/authorized_keys file.
- Each unit will have its own ID within the project stored within a file. For example, for Lamppost1, we use ID 1:
echo 1 > /root/id.txt
- Now create a script that will be used to connect into the server.
- Create a new shell script:
nano /root/create_ssh_tunnel.sh
- Then paste in the following, not forgetting to swap out ServerIPAddress:
#!/bin/bash /bin/pidof ssh > /dev/null if [[ $? -ne 0 ]]; then echo Creating new tunnel connection id=$( cat /root/id.txt ); /usr/bin/ssh -i /root/.ssh/id_rsa -N -R 222${id}:localhost:22 projectname@ServerIPAddress if [[ $? -eq 0 ]]; then echo "Tunnel to the server created successfully" else echo "An error occurred creating a tunnel to the server. RC was $?" fi fi
- Change the permissions:
chmod 700 /root/create_ssh_tunnel.sh
- Then run the script once from the command-line to allow you to accept the server’s SSH key:
/root/create_ssh_tunnel.sh
- We want this script to run all the time so that we don’t lose connection to the server. So set a cron task to run. Simply:
crontab -e
- And add the following lines:
# Keep SSH-VPN going * * * * * /root/create_ssh_tunnels.sh
All thats left is to logon to your server:
ssh projectname@dead-end-server
Check which of your project’s remote units are active:
./list-connections.awk
Then SSH into the relevant unit:
ssh Lamppost1