HOWTO: SSH to your IoT device when it’s behind a firewall, or on a 3G dongle.

How do you connect to 8 computers, dispersed around a city, through 3G dongles?

The biggest hurdle we faced with Shadowing was repairing and altering lamp heads while they were in situ.

10691927_535396549895974_983641021_n

Yep, try programming up here!

We did have the option of calling out the ever-helpful SSE to have a look inside the lamp heads. However, we had no real way of affecting or altering the software. To complicate matters further, our only potential method of communicating to the lamp heads was through a 3G dongle.

Solution

Normally, if you want to communicate to a remote computer on a standard network, you would first find its IP address and then SSH into the unit:

ssh computerUser@HostOrIPAddress

However, 3G dongles do not allow you to connect to them via their public IP Address as they are hidden behind a firewall, rendering this method useless.

Our solution was to create a Reverse SSH Tunnel system where the lamp heads phoned home to a server on our network, allowing us control over the lamphead’s computer from our office. Crucially, if someone got hold of the device and hacked into the SSH tunnel, they wouldn’t be able to get onto our networks: the tunnel is a dead-end.

This post will provide step-by-step instruction of how to setup your own Reverse SSH Tunnel server using an Amazon AWS Server.

Create yourself an Instance

To setup an Amazon Server, follow the guide below. If you already have a suitable server (with root access), you can skip ahead to the next section.

  • Sign-up to Amazon AWS Services
  • On the dashboard, click Launch a New Instance
  • Choose Ubuntu Server 14.04LTS
  • Choose t2.micro for instance type
  • Add Storage (Standard storage is fine)
  • Add Tag Instance Name: Dead-End-Server
  • Configure Security Group:
    • Create a new security group
    • Security Group Name: MyAccess
    • By default there will be an SSH rule added. However, change its source to My IP from the drop down box
    • Click Add new rule:
    • Choose HTTP; again, change its source to My IP
  • Click Review and Launch
  • Click Launch
  • Create a new key pair
  • Give the permissions key a Name: awspemkey then download. Do not lose this key!
  • Then click Launch
  • Make a note of the public IP address of your instance.

To connect to your instance you will need to configure some SSH settings on your computer. The following is for Mac OS/X, but there are usually plenty of documentation/guides available on the web for you to use your favourite client…

  • Open Terminal.app
  • Navigate to your SSH folder:
    cd ~/.ssh/
  • Create a Folder called AWSKEYS:
    mkdir AWSKEYS
  • Move the .pem key, downloaded above, to this folder:
    mv ~/Downloads/awspemkey.pem ~/.ssh/AWSKEYS/
  • Change the permissions of the .pem file:
    chmod 400 ./AWSKEYS/awspemkey.pem
  • Then edit your config file:
    nano config
  • Type the following in below anything else in the file; don’t forget to substitute IPAddress for your instance’s IP, noted above:
    Host dead-end-server
        Host IPAddress
        User ubuntu
        IdentityFile ./AWSKEYS/awspemkey.pem
    
  • Now SSH to the server:
    ssh dead-end-server
  • You are now connected to your Amazon EC2 Server!

Server Setup

To setup the Tunnel system you’ll need to run all of the following as the root user. For the Amazon Instance created above, this should simply be a case of running the following having SSH’d in:

sudo -i
  1. Install SSHGuard to help protect your instance:
    apt-get install sshguard
  2. Add your remote IP address to the SSHGuard’s whitelist. This ensures that you will not be locked out of the system:
    nano /etc/sshguard/whitelist
  3. Alter the SSH daemon’s config file to harden it a little:
    nano /etc/ssh/sshd_config
  4. Add the the following lines to the file:
    PermitRootLogin no
    TCPKeepAlive yes
    ClientAliveInterval 30
    ClientAliveCountMax 99999
    GatewayPorts yes
    AllowTcpForwarding yes
    ChallengeResponseAuthentication no
    PasswordAuthentication no
  5. Create a new group to collect all the VPN-tunnels and permissions:
    sudo groupadd -g 2000 ssh-vpn-tunnels
  6. Alter the ownership and permissions of root’s directories:
    chown root:ssh-vpn-tunnels /root/.ssh
    chmod 0751 /root /root/.ssh/
  7. If there isn’t one already, create a blank SSH config file (used later) and alter its permissions:
    touch /root/.ssh/config
    chown root:ssh-vpn-tunnels /root/.ssh/config
    chmod 0640 /root/.ssh/config
    

Connection management

We had eight lamp posts within the project and, at the time our server was setup, we were also considering additional deployments; because we wanted to keep them separated, we came up with the following convention:

  • Each deployment (henceforth referred to as “projectname”) has a user-account with a specific user-ID (e.g. 2220)
  • Each of up to nine lamp posts within a deployment had an ID (e.g. 1)
  • Each lamp post would setup a reverse-tunnel on a port of [projectname’s UID] + [LampPostID] (e.g. 2220+1 = 2221)

The following script allowed us to see, at a glance, the status of each connection within a project.

  1. Create a new file:
    nano /root/list-connections.awk
  2. Then paste in the following:
    #!/usr/bin/awk -f
    BEGIN {
    	# Extract the processes EUID, then create a prefix-version
    	iEUID = PROCINFO["euid"];
    	iEUIDPrefix = substr(iEUID, 0, 3);
    
    	# Go find all VPN ports
    	sCommand = "netstat -4lnt";
    	while (( sCommand | getline) > 0) {
    
    		# Found a port?
    		if(match($4, /:(2[0-9]{3,})$/)) {
    			iCurrentPort = substr($4, RSTART + 1);
    
    			# If root, or port-prefix matches, note the port
    			if(iEUID == 0 || substr(iCurrentPort, 0, 3) == iEUIDPrefix) {
    				abPortActive[ iCurrentPort ] = 1;
    			}
    		}
    	}
    	close(sCommand);
    
    	# Now find all the known hosts
    	sCurrentHost = "UNKNOWN";
    	iCurrentPort = -1;
    	while ((getline < "/root/.ssh/config" ) > 0) {
    		# Look for Host entries
    		if(match($1, /^Host$/)) {
    			# Emit any exiting blocks
    			if(iCurrentPort != -1) {
    				printf("%s\t%4d\t%s\n", abPortActive[ iCurrentPort ] ? " UP " : "down", iCurrentPort, sCurrentHost);
    				abPortActive[ iCurrentPort ] = "DISPLAYED";
    				iCurrentPort = -1;
    			}
    
    			# Note the new current host
    			sCurrentHost = $2;
    
    		# Look for Port entries
    		} else if(match($1, /^Port$/)) {
    
    			# If root, or port-prefix matches, note the port
    			if(iEUID == 0 || substr($2, 0, 3) == iEUIDPrefix) {
    				iCurrentPort = $2;
    			}
    		}
    	}
    	close(sFile);
    
    	# Emit any trailing blocks
    	if(iCurrentPort != -1) {
    		printf("%s\t%4d\t%s\n", abPortActive[ iCurrentPort ] ? " UP " : "down", iCurrentPort, sCurrentHost);
    		abPortActive[ iCurrentPort ] = "DISPLAYED";
    	}
    
    	# Emit any unknown ports
    	for(iCurrentPort in abPortActive) {
    		if(abPortActive[ iCurrentPort ] != "DISPLAYED") {
    			printf(" UP \t%4d\tUNKNOWN\n", iCurrentPort);
    		}
    	}
    }
    
  3. Change the ownership and permissions of the script:
    chmod 0751 list-connections.awk
    chown root:ssh-vpn-tunnels list-connections.awk
    

User Settings

  1. Create the user for the deployment/project:
    useradd -u 2220 -g 2000 -M -N projectname -c 'brief description of your project'
            (-u 2220: Set the user ID for the project
    	 -g 2000: Set the group ID; this is what we used above for ssh-vpn-tunnels
             -M: Do not create the home-directory
    	 -N: Do not create a user-group
    	 projectname: The username
    	 -c: The GECOS/description field
             )
  2. Create the home and SSH folders for your user:
    mkdir -p ~projectname/.ssh
  3. Change the permissions:
    chown -R projectname:ssh-vpn-tunnels ~projectname
    chmod 751 ~projectname
    chmod 700 ~projectname/.ssh
    chown projectname:ssh-vpn-tunnels ~/projectname/.ssh/authorized_keys 
    chmod 600 ~projectname/.ssh/authorized_keys
  4. Link the SSH config file from root:
    ln -s /root/.ssh/config ~projectname/.ssh/config
  5. Link the list-connection.awk file:
    ln -s /root/list-connections.awk ~projectname/
  6. To make connecting to the Remote Units easier, edit root’s SSH config file:
    nano /root/.ssh/config
  7. Add an entry for each Remote Unit you will want to connect to. To work with the AWK script above, use a port number based on user-ID plus 1 to 9; i.e. 2221 through 2229:
    Host <Remote-Unit-Name>
            Hostname                        localhost
            User                            username
            Port                            <port>
            ServerAliveInterval             7

    For example:

    Host Lamppost1
            Hostname                        localhost
            User                            ubuntu
            Port                            2221
            ServerAliveInterval             7

Remote Unit Setup

This is the computer you put out in the wild. You’ll also need to run all of the following as the root user.

  1. Generate an SSH key:
    ssh-keygen
  2. Follow the on-screen instructions, do not enter a password, leave it blank.
  3. Now you need to transfer the public part of the key (contained within ~/.ssh/id_rsa.pub) from the Remote Unit to the Tunnel Server – for example copy+paste it between two SSH-terminals. Add it to the end of the Tunnel Server’s ~projectname/.ssh/authorized_keys file.
  4. Each unit will have its own ID within the project stored within a file. For example, for Lamppost1, we use ID 1:
    echo 1 > /root/id.txt
  5. Now create a script that will be used to connect into the server.
  6. Create a new shell script:
    nano /root/create_ssh_tunnel.sh
  7. Then paste in the following, not forgetting to swap out ServerIPAddress:
    #!/bin/bash
    /bin/pidof ssh > /dev/null
    if [[ $? -ne 0 ]]; then
            echo Creating new tunnel connection
            id=$( cat /root/id.txt );
            /usr/bin/ssh -i /root/.ssh/id_rsa -N -R 222${id}:localhost:22 projectname@ServerIPAddress
            if [[ $? -eq 0 ]]; then
                    echo "Tunnel to the server created successfully"
            else
                    echo "An error occurred creating a tunnel to the server. RC was $?"
            fi
    fi
  8. Change the permissions:
     chmod 700 /root/create_ssh_tunnel.sh
  9. Then run the script once from the command-line to allow you to accept the server’s SSH key:
    /root/create_ssh_tunnel.sh
  10. We want this script to run all the time so that we don’t lose connection to the server. So set a cron task to run. Simply:
     crontab -e
  11. And add the following lines:
    # Keep SSH-VPN going
    *  *    *     *     *   /root/create_ssh_tunnels.sh

All thats left is to logon to your server:

ssh projectname@dead-end-server

Check which of your project’s remote units are active:

./list-connections.awk

Then SSH into the relevant unit:

ssh Lamppost1