FlashSimpleGesture: example applicaton for using the gesture API

→Return to Examples

Introduction

The FlashSimpleGesture application shows how you can create applications using gesture tracking. The gesture functionality is provided by two classes contained in the metaio.tracking.gesture package of the UnifeyeViewerToolkit: Hotspot, HotspotTracking.

Requirements

For the example below we will use

• the Adobes Flash Builder 4 and the Flex framework. Generally it shouldn't be a problem to use alternative Flash IDEs, e.g. FlashDevelop (http://www.flashdevelop.org),
• the UnifeyeViewer installation (see the installation guide).

Setup of the project

• Create new Flex project called e.g. SimpleGesture.
• Copy the UnifeyeViewerToolkit.swc from package\Flash\examples\UnifeyeViewerToolkit\bin to the project's libraries directory.
• Instead of creating new project you can also import the FlashSimpleGesture example project (located at package\Flash\examples\ into your workspace) in order to follow this tutorial.

Implementation

Application setup

The created project contains the SimpleGesture.mxml file in the default package of the project. Like in the FlashSimpleAR application we will use the applicationComplete event in order to be informed when the application is completely loaded. The registration of the event handler can be done either by using the design mode / category view or by adding it manually as shown in the code snipped below:

<?xml version="1.0" encoding="utf-8"?>
<s:Application xmlns:fx="http://ns.adobe.com/mxml/2009" 
            xmlns:s="library://ns.adobe.com/flex/spark" 
            xmlns:mx="library://ns.adobe.com/flex/mx"
            minWidth="800" minHeight="600" 
            applicationComplete="applicationComplete()">
   <fx:Script>
      <![CDATA[
         import mx.events.FlexEvent;

         // some definitions goes here
         ... 
         
         protected function applicationComplete():void
         {
            // TODO Auto-generated method stub
            
            // init camera
            initCamera();
            
            // init gesture tracking
            initGestureTracking();
         }

Camera initialization

As in the FlashSimpleAR application the applicationComplete handler will initialize the camera and the gesture tracker.

The code snipped bolow shows the initialization of the camera. The constructor of ARCamera needs two resolutions. The first resolution is the tracking resolution. This resolution is used to perform the tracking. The second resolution (the video resolution) is the resolution which will be used for the initialization of the webcam and this resolution will be used for displaying the camera stream on the screen. The fifth parameter is the desired framerate of the webcam.

In order to initialize the camera we first create a camera object and register the event handlers for ARCameraEvent.INITIALIZED, ARCameraEvent.ERROR, ARCameraEvent.DECLINED events which will or can be raised during the camera initialization phase. In the next step we activate the first camera in the list of available cameras ( by m_camera.activateCameraByIndex(0) ) and add the camera object to the stage.

private function initCamera():void
{
   m_camera = new ARCamera( m_trackingWidth, m_trackingHeight, 
                            m_videoWidth, m_vidoHeight,  m_cameraFps);

   m_camera.addEventListener( ARCameraEvent.INITIALIZED, onCameraInitilized );
   m_camera.addEventListener( ARCameraEvent.ERROR, onCameraError );         
   m_camera.addEventListener( ARCameraEvent.DECLINED, onCameraError );
   
   ///! Use the first camera found. 
   m_camera.activateCameraByIndex(0);
   
   ///! add the camera to the stage, so that it is visible.
   stage.addChild(m_camera);   
}

The initialization of the camera is completed when ARCameraEvent.INITILIZED is raised. On errors during initialization or when the user declines the usage of the camera, the events ARCameraEvent.ERROR respectively ARCameraEvenet.DECLINED are raised.

private function onCameraError( e:ARCameraEvent ):void
{
   trace("Camera Error: " + e.toString() );
   stage.removeEventListener(Event.ENTER_FRAME, onEnterFrame);   
}

Once the camera is initialized we can register the onEnterFrame() function as event handler for the onEnterFrame event, which will be raised each time the application starts with a new frame.

private function onCameraInitilized( e:ARCameraEvent ):void
{
   trace("Camera started!");
   m_camera.setMirrorMode(true);
   stage.addEventListener(Event.ENTER_FRAME, onEnterFrame);
}

Tracking initialization

The function initGestureTracking() initializes the gesture tracking. The initialization is done by creating an object of class HotspotTracking. The HotspotTracking class implements functionality for movement detection in particular regions (so called "Hotspots") in the frame buffer. These regions can be created by calling the createCamHotspot() member of the HotspotTracking class, which returns an object of class Hotspot. A Hotspot object represents a region of the frame buffer which should be observed for motion. In order to inform the application about changes inside the observed regions, the Hotspot objects raises the HotspotEvents:

• HotspotEvent.HAND_ENTER ( raised when something enters the region, like the user's hand entering the hotspot region )
• HotspotEvent.HAND_OVER ( raised when the region differs (from origin) like a hand being located over the region )
• HtospotEvent.HAND_LEAVE ( raised when something leaves the region, e.g. a hand leaving the region )

The code snipped below shows the implementation of the initGestureTracking() function. As you can see the constructor of HotspotTracking class needs the resolution of the frame buffer, which will be passed to tracking. In our case it is the tracking resolution defined by the members m_trackingWidth/Height. For creation the according construction of a hotspot we need the position and size. Note: the position and size of a hotspot are defined with respect to the tracking resolution.

private function initGestureTracking():void
{
   ///! init gesture tracking
   m_gestureTracking = new HotspotTracking(m_trackingWidth, m_trackingHeight);
   
   ///! setup a hotspot in the top-left corner. 
   var myHotspot:Hotspot =  m_gestureTracking.createCamHotspot( 0, 0, 20, 20 );
   
   myHotspot.addEventListener( HotspotEvent.HAND_ENTER, onHotspotEnter );
   myHotspot.addEventListener( HotspotEvent.HAND_OVER, onHotspotOver );            
   myHotspot.addEventListener( HotspotEvent.HAND_LEAVE, onHotspotLeave );
}

In order to visualize hotspot states, we added the Flex UI element of type BorderContainer with the id m_hotspot by adding:

<s:BorderContainer x="0" y="0" width="20" height="20" depth="20" 
   contentBackgroundAlpha="0.5" id="m_hotspot" borderVisible="false" 
   contentBackgroundColor="#FF9898" backgroundAlpha="0.5" 
   backgroundColor="#FF0000"></s:BorderContainer>

to the applications mxml file, and change the color of it on different events:

// called when a hand entered the hotspot
private function onHotspotEnter( e:HotspotEvent ):void
{
   ///! change the color of hotspot to green
   m_hotspot.setStyle("backgroundColor", 0x00ff00);
}

// called when a hand is over the hotspot
private function onHotspotOver( e:HotspotEvent ):void
{
   ///! change color of hotspot to blue
   m_hotspot.setStyle("backgroudColor", 0x0000ff00);
}

// Called when a hand left the hotspot.
private function onHotspotLeave( e:HotspotEvent ):void
{
   ///! change the color of hotspot to red
   m_hotspot.setStyle("backgroundColor", 0xff0000);
}

Main loop

Finally we have to implement the handler of the enterFrame event. A possible implementation is shown below:

private function onEnterFrame( e:Event ):void
{
   ///! grab an image
   var cameraImage:ByteArray = m_camera.getCurrentFrame(true);
   
   cameraImage.position = 0;
   
   if ( cameraImage.bytesAvailable == 0 ) 
   {
      trace("Error: image is empty!");
      return;
   }
   
   ///! trackFrame fires the respective events, if a hotspot was activated.
   m_gestureTracking.trackFrame( cameraImage );
   
   ///! bring m_hotspot into the front
   stage.addChild(m_hotspot);
}

For more information regarding the gesture API please refer to the UnifeyeViewerToolkit reference or FlashSimpleGesture example application.

→Return to Examples

-- SupportMetaio - 2011-02-21