Footfall: A Camera Based People Counting System for under £60


FootfallInitialtest

In Watershed, we have numerous systems that tell us statistics such as how many cinema seats have been booked, how much food is left in stock and so on. But, we have no real method of capturing one of the most important metric for an arts organisation, how many people visit or are currently in the building.

This post shows how we developed a lightweight, un-intrusive tracking system with an accuracy of 90-95% costing under £60.

Important!

This application does not have the ability to save live images. The images used in this post were for debugging purposes only. We are not interesting in the who, we are only interested in the how many!

Existing Systems

breakbeam counter
Now, we’ve all seen these devices sulking about in the entrances to buildings. The trusty break beam. They are able to record how many people have entered an area at any given time. These systems operate on a very simple logic: a laser is fired across a span at a retroreflective marker which reflects the beam to a reader. If the beam is broken at any point, a counter is increased.

In fairness this is a reliable method of capturing data. However, it has its problems. For instance, it is unable to distinguish how many people go through the beam at once, if two people walk side by side then they are counted as one event. More importantly they are unable to distinguish whether people are going in or out of an area (what tends to happen is a final count is taken and is divided by 2). The resulting data would then be inputted into a spreadsheet manually.

What we did?

We wanted the ability to accurately plot whether people were coming into or going out of Watershed, as well as the time they entered or left. We also wanted to see the counter statistics in real time. However, the emphasis for this project was to ensure the unit had as small a footprint as possible so as not to be physically imposing while being cost effective. Our hardware setup was a Raspberry Pi (£23), a Standard PiCamera (£20) and a PoE Unit (£10) (Power over Ethernet)*.

IMG_0760

Fully Constructed Footfall

IMG_0763

Footfall Mounted (yep, it’s the little black box in the celling).

In terms of software there were two main programs.

The first was a openFrameworks C++ application which ran on the Raspberry Pi handling the camera input, image abstraction, tracking and people counting. In basic terms the application did the following processes.

  1. Capture images from the PiCamera
  2. Pass them through a series of openCV Algorithms
  3. Compile blobs and track their movements
  4. If blob adheres to specific conditions
  5. Fire event to footfall server

The second program sat on a server and was used to contain and visualise the data from the Raspberry Pi. In this we used a combination of PHP, MySQL and Javascript.
To display the result we used the fantastic ChartJs library, which uses HTML5 <canvas> to generate graphs,charts and diagrams.

The main processes in this instance were as follows.

  1. Get data from the database (MySQL, PHP and Javascript)
  2. Generate Charts (Javascript)
  3. Display the Charts (Javascript)
  4. Update the Charts (PHP and Javascript)

In the next couple of sections we will explain some of the processes we implemented, specifically the Computer Vision algorithms and how they helped us extract people from the camera’s view. We will also document how we displayed the counter data in real time using ChartJS.

OpenCV

In an ideal scenario we would have used a HoG and SVN detector to compute the tracking. For those unfamiliar with HoG (History of Orientated Gaussians) or SVM (Support Vector Machines) detectors.

HoG is a method of object detection that counts the occurrences of gradient orientation in localised portions of an image. So HoG segments a source image into smaller blocks, calculates a histogram of each block, then calculates the orientation of each block. This source image is checked against a positively trained descriptor image. Below we see the results of applying a HoG descriptor.

These images are taken from the fantastic resource provided by Carl Vondrick and his contributors. Here is their paper Visualizing Object Detection Features

The far right image is the HoG image, this is the positive image.
If we trained the positive image to be that of a person, the HoG Descriptor might look something like this.

Trained HoG Descriptor

And if you implement the HoG to find people in a standard setting and overlay the tracker. This would be the result.

HoG Detector

However, we do not have any suitable areas in Watershed to successfully implement this system.

Instead, our approach was to use a form of blob tracking whereby the IDs of the blobs are tracked until they reach condition then they are erased and ignored in future processes. But before any tracking could occur we needed to clean our camera image.

Mask

We decided to place the counter above the main stairwell, enabling us to capture nearly all customers that entered the building (2). However, it did mean that our source image was too wide. We needed to select a specific section of the image, to do this we generated a Mask Matrix:

//--------------------------------------------------------------
void ofApp::makeMask()
{
    // Coordinates of mask polyline
    vector <cv::Point> maskright;
    maskright.push_back(cv::Point(95,0));
    maskright.push_back(cv::Point(123,211));
    maskright.push_back(cv::Point(228,220));
    maskright.push_back(cv::Point(270,0));
    
    // Create new Matrix, Remember the size is flipped
    mask = cvCreateMat(240, 320, CV_8UC1);
    // Loop through the matrix and turn it black
    for(int i=0; i < mask.cols; i++)
        for(int j=0; j < mask.rows; j++)
            mask.at<uchar>(cv::Point(i,j)) = 0;
    
    // Convert the points to polylines
    vector<cv::Point> polyright;
    approxPolyDP(maskright, polyright, 1.0, true);
    
    // Add a white polyline shape to the Mask Matrix
    fillConvexPoly(mask,&polyright[0],polyright.size(),255,8,0);
}

After connecting to the PiCamera, images are piped into the application, these images are transferred/copied into another Matrix along with the mask Matrix; which combines both images.

lightenMat.copyTo(maskOutput,mask);

Finding People

As we were using blob tracking/contour finding we needed to provide a background image to compare the live image against. This would highlight any changes to live view. But, our stairwell comes out on to an area flooded with daylight, which varies enormously. So standard background subtraction was out of the question. We therefore used a variant of the background subtraction known as MOG or Mixture of Gaussians, which essentially is like a running total of background images:

// In setup
pMOG2 = new BackgroundSubtractorMOG2(_history,_MOGThreshold,false);

In the setup procedure you define the history length or the number of backgrounds to average. Then the threshold of the background and whether or not to find shadows. Then simply pass the masked image into the MOG, threshold the matrix, slightly blur, dilate and perform contour finding:

// Activate the background subtraction
pMOG2->operator()(maskOutput, fgMaskMOG2);

// Threshold the image
threshold(fgMaskMOG2, output, _threshold);

// Blur
blur(output, _blur);

// Dilate
dilate(output);

// Pass through the Contour Finder
if (ofGetFrameNum() > 200) {
    contourFinder.findContours(output);
    // Do tracking
    tracker.track(contourFinder.getBoundingRects());
}

We placed a small sanity check on the tracker that only enabled it once the application had elapsed 200 frames, this stopped any false positives being generated on the startup of the application. The image below show each process results.

footfallgif

  • Top Right: Live Image with Contour Finder
  • Top Right: Processed MOG Image (thresholded,blurred and dilated)
  • Bottom Left: Unprocessed MOG
  • Bottom Right: Combination of the Mask and Camera image

Counting

Once we have our blobs, the next phase is counting them.
We created a condition checker which evaluated the following parameters.

  • Initial position of the Blob
  • Current position
  • Size of Blob
  • How many people within Blob

Then when a new blob is detected, its initial position is logged, once it reaches the centre of the camera view. The checker then evaluates which direction the blob is travelling in as well as the width of the blob which determines how many people are in the blob:

currentPos = toOf(track).getCenter();
area = toOf(track).getArea();
previousPos.interpolate(currentPos, .3);
width = toOf(track).width;
height = toOf(track).height;

if (!inLineLatch && previousPos.y >= currentPos.y && (currentPos.y >= _lineStartPoint.y-20 && currentPos.y <= _lineStartPoint.y+20)) {     bIn = true;     bOut = false;          if (width >= _three) {
        numberOfPeople = 3;
    }
    else if(width >= _two) {
        numberOfPeople = 2;
    }
    else if(width >= _one) {
        numberOfPeople = 1;
    }
    else {
        cout << "Not big enough for a person" << width << endl;
    }
    
    inLineLatch = true;
}

if (!outLineLatch && previousPos.y <= currentPos.y && (currentPos.y >= _lineStartPoint.y-20 && currentPos.y <= _lineStartPoint.y+20)) {     bIn = false;     bOut = true;     if (width >= _three) {
        numberOfPeople = 3;
    }
    else if(width >= _two) {
        numberOfPeople = 2;
    }
    else if(width >= _one) {
        numberOfPeople = 1;
    }
    else {
        cout << "Not big enough for a person" << width << endl;
    }
    outLineLatch = true;
}

If someone enters or leaves, a simple HTTP Post event is fired to our server. These posts contain the timestamp from the Raspberry Pi, where the unit is located and the number of people who have entered or left. You’ll see that in the out post form the count file is prepended with a minus symbol this will be explained in the next section:

vector<Blob>& people = tracker.getFollowers();
    for(int i = 0; i < people.size(); i++) {
        //If the tracker returns true in open latch increment, then close latch. Then kill the tracker element.
        if (people[i].bIn) {
            if (counterLatches[i]) {
                string httpString = ofToString(people[i].howManyIn());
                ofxHttpForm formIn;
                formIn.action = _uploadurl;
                formIn.method = OFX_HTTP_POST;
                formIn.addFormField("secret", _secretKey);
                formIn.addFormField("location", _locationID);
                formIn.addFormField("count", httpString);
                formIn.addFormField("rawtimestamp", ofGetTimestampString("%Y-%m-%d %H:%M:%s"));
                formIn.addFormField("submit","1");
                httpUtils.addForm(formIn);
                counterLatches[i] = false;
            }
            people[i].kill();
            counterLatches[i] = true;
        }
        //If the tracker returns true out open latch increment, then close latch. Then kill the tracker element.
        if (people[i].bOut) {
            if (counterLatches[i]) {
                string httpString ="-"+ofToString(people[i].howManyOut());
                ofxHttpForm formOut;
                formOut.action = _uploadurl;
                formOut.method = OFX_HTTP_POST;
                formOut.addFormField("secret", _secretKey);
                formOut.addFormField("location", _locationID);
                formOut.addFormField("count", httpString);
                formOut.addFormField("rawtimestamp", ofGetTimestampString("%Y-%m-%d %H:%M:%s"));
                formOut.addFormField("submit","1");
                httpUtils.addForm(formOut);
                counterLatches[i] = false;
            }
            people[i].kill();
            counterLatches[i] = true;
        }
    }
    
    if (people.empty()) {
        // Counter Latch used to increment the counter and stop duplicate counters
        for (int i = 0; i < 30; i++) {
            counterLatches[i] = true;
        }
    }

ChartJS

ChartJS is a lightweight javascript library that uses HTML5 and Canvas to generate and render charts. Users can specify a number of different chart types such as pie, bar, line and polar. This section demonstrates how we used ChartJS to visualise our footfall data

Generating the Charts

We created two charts, one for displaying the total number of people in Watershed with information about the screenings and events occurring in Watershed for that day and another chart displaying the flow of traffic.

To generate the charts we first define some global variables:

var trafficData;
var totalsData;
var trafficChart;
var totalChart;
var interval = 300;

Then in a new function we create the Totals Chart. First create a variable called canvas and give it the context of the html element with an id of ‘totalChart’. This looks for a space in our webpage to put the chart. Next we create a datasets object, this is the line element of our chart where we define colour, labels and data points. For our system we pre populated the data with 2 zero values and 2 timestamps 03:00 and 04:00 so that both axis have some initial data. Next we configured the chart options, by default ChartJs sets a global chart configuration, to override this simply pass your specific options into the chart object:

function createTotalChart()
{
	var canvas = document.getElementById('totalChart').getContext('2d');
	var datasets = [{
		label: "People In Watershed",
		fillColor: "rgba(0,77,255,.01)",
		strokeColor: "rgba(0,77,255,1)",
		pointColor: "rgba(0,77,255,1)",
		pointStrokeColor: "#fff",
		data: [0,0]
	}];

	totalsData = {
		labels: ['3:00','04:00'],
		datasets: datasets
	};

	// Chart Options
	var options = {
		// Do not animate
		animation: false,
		responsive:true,
		scaleShowLabels: 20,
		scaleBeginAtZero : false,
		skipXLabels:true,
		bezierCurveTension : 0.3,
		pointHitDetectionRadius : 10,
		scaleGridLineColor : "rgba(0,0,0,.1)",
		scaleGridLineWidth : 2,
		bezierCurve : true
	};

	// Make Chart, Pass options
	totalChart = new Chart(canvas).Line(totalsData, options);
}

Again for the traffic chart we create a function. Then find the canvas element with the id of ‘trafficChart’. Like the totals chart we generate the dataset, but give it two data points with different colours and labels. More importantly, when passing options to the chart we define that the bars must begin at origin which is zero and that the scale does not begin at zero. This allows us to plot positive and negative bars side by side in the same chart.

function createTrafficChart()
{
	var canvas = document.getElementById('trafficChart').getContext('2d');

	trafficData = {
		labels: ['03:00','4:00'],
		datasets:
	[
		{
			label: "Traffic In",
			fillColor: "rgba(65,190,242,0.5)",
			strokeColor: "rgba(65,190,242,0.5)",
			pointColor: "rgba(65,190,242,0.5)",
			pointStrokeColor: "#fff",
			data: [0,0],
			chartType: 'Bar'
		},
		{
			label: "Traffic Out",
			fillColor: "rgba(242,65,65,0.5)",
			strokeColor: "rgba(242,65,65,0.5)",
			pointColor: "rgba(242,65,65,0.5)",
			pointStrokeColor: "#fff",
			data: [0,0],
			chartType: 'Bar'
		}
	]
	};

	// Chart Options
	var options = {
		animation: false,
		responsive:true,
		scaleGridLineColor : "rgba(0,0,0,.1)",
		// Important: allows the chart to draw both scales + / -
		barBeginAtOrigin: true,
		scaleBeginAtZero : false
	};
	// Make Chart
	trafficChart = new Chart(canvas).Bar(trafficData,options);
}

All that needs to be done is assign the correct element ids to the appropriate HTML element.

<canvas style='margin-left: 5px;' id="totalChart" width='900' height='300'>
<canvas style='margin-left: 5px;' id="trafficChart" width='900' height='300'>

Populating the Charts

When our tracker detects a person it fires an event value with a timestamp from the RPi into our database, when we select some data from the database we return the following.
Screen Shot 2015-04-07 at 11.32.25
On its own this data is fairly meaningless. Therefore, we process these events into something more tangible. We perform the following MySQL query which aggregates events into three new fields.

SELECT FROM_UNIXTIME( FLOOR( UNIX_TIMESTAMP( TIMESTAMP ) / 600 ) *600 ) AS `timekey`, 
SUM( event ) AS `movement`, 
SUM( IF( event > 0, event, 0 ) ) AS `peoplein`, 
SUM( IF( event < 0, ABS( event ) , 0 ) ) AS `peopleout`
FROM `data`
WHERE DATE( TIMESTAMP ) = DATE( NOW( ) ) 
GROUP BY `timekey`
ORDER BY `timekey` ASC 	

The first field is movement this is the running total event of people in the building, so positive numbers will increment and negative number will decrease the overall count. The other two fields peoplein and peopleout evaluate whether event is positive or negative, then aggregates the result into the the corresponding field (Calculating the absolute value for the negative field). The data is then split into chunks or ranges, this can be put into a variable and changed on the fly but, for this example we have used an interval value of 600 seconds (10 minutes).

Screen Shot 2015-04-07 at 11.38.46

$get = $DBH->prepare($query);
$get->execute();

if (!$get) {
	echo "Error: couldn't execute query. ".$get->errorCode();
	exit;
}

if ($get->rowCount() == 0) {
	echo "[]";
	exit;
}

$rows = array();
$runningtotal = 0;
$runningtotalin = 0;
$rowCount = 0;
$totalin = 0;
$starttime = strtotime("$endtimedate 8am");
$endtime = 0;
while ($row = $get->fetch(PDO::FETCH_ASSOC)) {
	$runningtotal += $row['movement'];
	$row['total'] = $runningtotal;
	$totalin += $row['peoplein'];
	$row['totalin'] = $totalin;
	$rows[$row['timekey']] = $row;
	$endtime = $row['timekey'];
	$rowCount++;
}
$endtime = strtotime($endtime);
if (!$endtimedate || $endtimedate == date("Y-m-d")) {
	$endtime = time() - (time() % $interval);
}
$runningtotal = 0;
$runningtotalin = 0;
for ($t = $starttime; $t <= $endtime; $t += $interval) {
	$dt = date("Y-m-d H:i:s",$t);
	if (isset($rows[$dt])) {
		$runningtotal = $rows[$dt]['total'];
		$runningtotalin = $rows[$dt]['totalin'];
	}
	else {
	$rows[$dt] = array("timekey" => $dt, "movement" => 0, "peoplein" => 0, "peopleout" => 0, "total" => $runningtotal, "totalin" => $runningtotalin);
	}
}
ksort($rows); // sort
$rows = array_values($rows); // change back into indexed

echo json_encode($rows);

This data is then encoded into json.

Screen Shot 2015-04-07 at 13.49.24

To input this data into the chart we make an ajax request to a PHP script on our server. The results are then parsed and assigned to the relevant chart.

function preLoadData()
{
	// Get the First Load of Data, this will update throughout the day
	$.ajax({ url:"http://someGetRequest.co.uk/", async: true, dataType: 'json', type:'get',
		}).done(function(data){
			// If we already have data update the labels on the web page
			updateLabels(data);
			for (var i in data) {
				labelLength++;
				var label = data[i].timekey.substring(11,16);
				var eventsDatasets = [];
				eventsDatasets.push(data[i].total);
				eventCount = 0;
				for (var title in events) {
					eventAtCurrentLabel = null;
					var ev = events[title];
					for (var t in ev) {
						if (ev[t].start <= label && ev[t].end >= label) {
							eventAtCurrentLabel = -10-(eventCount*10);
						}
					}

					eventsDatasets.push(eventAtCurrentLabel);
					eventCount++;
				}
				// Best way of getting the data into the graphs | substring cuts the timestamp down
				totalChart.addData(eventsDatasets,label);
				trafficChart.addData([data[i].peoplein, -data[i].peopleout],data[i].timekey.substring(11,16));
			}
	});
}
Screen Shot 2015-04-07 at 13.51.01

Totals Graph Pre-populated, including film screenings that are shown as lines on the graph to help us correlate screenings with footfall

Updating the Information

To add data to the charts after the initial load, the following function is called every 5 seconds.

function updateValues()
{
	$.ajax({
		url:"http://someGetRequest.co.uk/",
		async:true,
		dataType: 'json',
		type:'get',
	}).done(function(data)
	{
		updateLabels(data);
		updateTotals(data);
		updatePeople(data);
		addDataToCharts(data);
	});
}

This updates the HTML and values on the page but, more importantly it adds new data to the charts. Now I stress new data because without the code below the chart would constantly add the same value until the timecode changed. So the function below evaluates the current returned timestamp against the last inputted timestamp in the charts data. If the timestamp changes then a new timestamp is added to the chart and the final value in the previous timestamp is added to that timestamp.

function addDataToCharts(data)
{
  	if (data[data.length-1].timekey.substring(11,16) == trafficData.labels[trafficData.labels.length-1]) {	}
	else {
		trafficChart.addData([data[data.length-1].peoplein,-data[data.length-1].peopleout],data[data.length-1].timekey.substring(11,16));
		var trafficInDataLength = trafficChart.datasets[0].bars.length-2;
		var trafficOutDataLength = trafficChart.datasets[1].bars.length-2;

		trafficChart.datasets[0].bars[trafficInDataLength].value = data[data.length-2].peoplein;
		trafficChart.datasets[1].bars[trafficOutDataLength].value = -data[data.length-2].peopleout;
		trafficChart.update();
	}
	if (data[data.length-1].timekey.substring(11,16) == totalsData.labels[totalsData.labels.length-1]) {	}
	else {
		totalChart.addData([data[data.length-1].total],data[data.length-1].timekey.substring(11,16));
		var totalDataLength = totalChart.datasets[0].points.length-2;
		totalChart.datasets[0].points[totalDataLength].value = data[data.length-2].total;
		totalChart.update();
	}
}

And here are the resulting charts.
Screen Shot 2015-04-07 at 13.51.01

Screen Shot 2015-04-07 at 14.40.42

Alongside putting the data into MySQL, we also made the PHP code put footfall data into our elasticsearch cluster, and experiment with graphing it in Kibana.

Although it's hard to graph the 'total people in watershed' graph, we hope it'll be a lot easier to integrate with other business intelligence metrics, such as film screenings, bar takings, conference bookings, etc.

kibana

We have made the source code from this project available on GitHub.

Disclaimer

The original software was intended for sole use within Watershed, therefore some of the source code has been altered for public use and differs slightly to our systems. For example our system generated event tags showing screening in conjunction with the total number of people in Watershed, to do this we had to pre-populate some timestamps and may cause an issue if the system is ran past a certain time.

4 thoughts on “Footfall: A Camera Based People Counting System for under £60

  1. Great project, I see alot of use for this in the whole, especially retail operation that depends on foot traffic.

  2. HI, compliments for your idea….should be possible to have the source video file in order to test your code in your same condition ? Thanks a lot.

    Pippo

  3. Hi, you made a great job. I need a little help building the project. It shays that i am missing some references. Can you tell me how can i add additional references and should i do that?
    I get things like this at the end of compilation:
    (.text.cvFitLine+0x980): undefined reference to `__acos_finite’
    ../../../addons/ofxOpenCv/libs/opencv/lib/linuxarmv7l/libopencv_imgproc.a(linefit.o): In function `cvFitLine’:
    (.text.cvFitLine+0x1436): undefined reference to `__acos_finite’
    collect2: ld returned 1 exit status

    I would appreciate your help.

    P.S. I am compiling only RPi part of the project.

    BR

Comments are closed.