There’s always been a great divide between displays that can receive user feedback (e.g. touchscreens) and the displays that most people use. Touch screens are nifty, but there are usually a number of drawbacks to them. Smudges from fingerprints, less than optimal brightness and contrast, and most importantly, price.
Leap Motion aims to bridge the gap and allow users to communicate more naturally with their hands. They’ve done an outstanding job of communicating with developers and aggregating documentation and resources. They’ve even gone so far as to include support for Linux users, although the requirement of libc6 2.17 may mean an upgrade is in order for some.
When I first heard of the Leap I pictured Tom Cruise in Minority Report sorting pictures that made him angry from pictures that excited him. Okay, so it’s been a while, and I don’t remember it that well. The point was, Tom’s fluid hand movements had a direct result on the screens in front of him, the computer was obeying his physical commands, tracking his movements with excellent precision.
With support for most languages included out of the box, and the company targeting a variety of devices, and not just PCs, the possibilities are almost endless. I remember seeing one developer wanting to use the Leap to control his TV, practical limitations (most notably distance) aside. Many others have already developed methods of generating music. Sign language, orchestra conduction, interactive anatomy lessons, electronics engineering, the list of possible products goes on and on. Though it hasn’t received near as much press, Leap can be used on today’s websites, either in the form of Greasemonkey extensions in the user’s own browser or server-side by site authors using the Leap Javascript Library.
Leap currently comes enabled with four standard gestures: Circle, Swipe, Screen Tap, and Key Tap. The last two are just how they sound, one being horizontal and other vertical. More gestures will continue to be added, but you can also create your own with a small amount of experimentation.
In this example, I’m going to demonstrate listening for “thumbs up” and “thumbs down” gestures and triggering custom jQuery events when found.
First, we create a tiny HTML page that loads minified versions of jQuery and Leap, as well as our own tiny JavaScript. The page will appear black until one of our custom gestures are detected.
<html>
<head>
<style type="text/css">
body {
overflow: hidden;
margin: 0;
padding: 0;
}
#pad {
width: 100%;
height: 100%;
background-color: #000;
text-align: center;
}
#pad img {
width: 30%;
height: 50%;
margin: 100px auto;
}
</style>
</head>
<body>
<div id="pad">
<img src="transparent.gif" alt="image">
</div>
<div id="log"></div>
<script src="./leap.min.js"></script>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"></script>
<script src="pad.js"></script>
</body>
</html>
UI is pretty much nonexistent here. There are no errors displayed when Leap isn’t connected, and nothing in the way of feedback except when gestures are recognized correctly.
The bulk of this sample is LeapHelper, which you’ll see below. Instead of it directly modifying the DOM, I’m just having it send event signals to a container element based on the name of the gesture detected.
var LeapHelper = {
isListening: false,
$container: null,
timeout: null,
TriggerGesture: function (gesture)
{
LeapHelper.StopListening();
LeapHelper.$container.trigger('gesture:'+gesture);
if (null !== LeapHelper.timeout)
{
clearTimeout(LeapHelper.timeout);
}
},
TriggerReset: function()
{
LeapHelper.$container.trigger('gestureReset');
},
StartListening: function()
{
LeapHelper.isListening = true;
},
StopListening: function()
{
LeapHelper.isListening = false;
},
ResumeListening: function()
{
setTimeout(LeapHelper.StartListening, 500);
},
init: function (containerId)
{
LeapHelper.$container = $(containerId);
var handY = null, handCenter = null, handRadius = null,
fingerY = null, fingerX = null;
LeapHelper.StartListening();
Leap.loop(function(frame)
{
if (!LeapHelper.isListening) return;
if (frame.hands.length == 1)
{
var hand = frame.hands[0];
handY = hand.palmPosition[1];
handCenter = hand.sphereCenter[0];
handRadius = hand.sphereRadius;
if (frame.pointables.length == 0)
{
if (null !== LeapHelper.timeout)
{
clearTimeout(LeapHelper.timeout);
}
LeapHelper.timeout = setTimeout(LeapHelper.TriggerReset, 500);
}
else if (frame.pointables.length == 1)
{
var finger = frame.pointables[0];
fingerX = finger.tipPosition[0];
fingerY = finger.tipPosition[1];
var upright = handY < fingerY,
fingerExtended = Math.abs(handY - fingerY) > 75,
fingerVertical = Math.abs(finger.direction[1]) > 0.65;
var isTheBird = upright && fingerVertical && finger.length > handRadius * 0.75 &&
(fingerX > handCenter - handRadius / 5 && fingerX < handCenter + handRadius / 5);
var eventName = '';
if (upright && fingerExtended && fingerVertical)
{
eventName = 'thumbsUp';
}
else if (!upright && fingerVertical)
{
eventName = 'thumbsDown';
}
if (eventName)
{
LeapHelper.TriggerGesture(eventName);
}
}
else
{
LeapHelper.TriggerReset();
}
}
else
{
LeapHelper.TriggerReset();
}
});
}
};
(function($){
$('#pad').bind('gesture:thumbsUp', function() {
$('#pad').find('img').attr('src', 'thumbsUp.png');
LeapHelper.ResumeListening();
}).bind('gesture:thumbsDown', function() {
$('#pad img').attr('src', 'thumbsDown.png');
LeapHelper.ResumeListening();
}).bind('gestureReset', function() {
$('#pad img').attr('src', 'transparent.gif');
});
LeapHelper.init('#pad');
})(jQuery);
Timeouts are used to handle clearing of the display and any custom application logic when an appropriate gesture is no longer detected. We stop listening to new events while running callbacks, and we only care about those frames that include one hand, and one finger. Other frames are considered invalid and will trigger a reset operation.
Video demo (sorry for the horribly shaky video):
[youtube http://www.youtube.com/watch?v=Red16njrDBc]
Source HTML:
Leap Motion thumbs up/down demo (requires Leap Motion)
In practice, it works fairly well, though an outstretched hand with five extended fingertips can occasionally trigger a thumbs up/down gesture. To really make it solid, we would need to consider previous frames in addition to the current frame. Leap has been somewhat frustrating to work with, as the fifth-generation developer unit I’m using will occasionally detect a monitor corner or something else as a pointer. If the detection environment isn’t pristine, the screen calibration can be a real annoyance. That said, I expect these issues to be nil, or at least significantly improved, come release. I’m looking forward to great things with Leap.
