HOW TO CREATE YOUR FIRST AR Application WITH ADOBE FLASH

→Return to Examples

Introduction

This tutorial shows how to create a simple Augmented Reality application using Adobe Flash (with Flash Builder as development environment) and the Unifeye Viewer for Flash SWC library. The application will have simple AR functionality like the FlashSimpleAR application shipped with the Unifeye Viewer package. FlashSimpleAR can be found in the examples directory of the UnifeyeViewer installation.

Requirements

• In order to create the sample application we will use Adobes Flash Builder 4 (http://www.adobe.com/products/flashbuilder/) and the Adobe Flex framework. Generally it should be no problem to use alternative Flash IDEs, e.g. FlashDevelop (http://www.flashdevelop.org/)
• UnifeyeViewer installation (see the installation guide).

Setup the Flash Builder Project

1. Start Adobe Flash Builder and create a new Flex project, which will be added to your workspace.
2. Next we will add the libraries needed for tracking and rendering. All required libraries are located in libs directory of the FlashSimpleAR folder included in the UnifeyeViewer package.
   
   • First add metaioUnifeyeViewer.swc package to the project's libraries.
   • In this tutorial we use the papervision3d rendering engine, so we also have to add the papervision3d package to the project's libraries.
3. In the next step we have to copy TrackingData files from the FlashSimpleAR bin-debug folder to the current project bin-dir folder. In this example we use the file car.xml_enc.
4. To make the life easier we will reuse two classes ( Tracking and ARCamera) from the FlashSimpleAR application. Therefore we copy the metaio folder located in the src directory of the FlashSimpleAR, to the src folder of our project. In the next sections we will get into more details and explain the functionality of the classes defined by the files.

Implementation

Application Setup

After the Flash project was created we will found a .mxml file in the src directory of the project. MXML is a XML based user interface markup language which was introduced by Macromedia. Actually .mxml defines the main container of a Flash application and describes the layout of the user interface. The file should look like this:

<?xml version="1.0" encoding="utf-8"?>
<s:Application xmlns:fx="http://ns.adobe.com/mxml/2009" 
            xmlns:s="library://ns.adobe.com/flex/spark" 
            xmlns:mx="library://ns.adobe.com/flex/mx" minWidth="955" minHeight="600">
   <fx:Declarations>
      <!-- Place non-visual elements (e.g., services, value objects) here -->
   </fx:Declarations>
</s:Application>

Furthermore it can contain also ActionScript code, which implements the logic of a Flash application. In order to add ActionScript code to the application, a <mx:Script> section has to be defined. This sections can contain declarations of variabes and functions as well as module imports.

<?xml version="1.0" encoding="utf-8"?>
<s:Application xmlns:fx="http://ns.adobe.com/mxml/2009" 
            xmlns:s="library://ns.adobe.com/flex/spark" 
            xmlns:mx="library://ns.adobe.com/flex/mx" minWidth="955" minHeight="600">
   <fx:Declarations>
      <!-- Place non-visual elements (e.g., services, value objects) here -->
   </fx:Declarations>
        <fx:Script>
        <![CDATA[
        
        ... imports ...
        ... variables ...
        ... functions  ...       

        ]]>
        </fx:Script>
</s:Application>

Variables

Before we start implementing some functionality, we should define some "global" variables which will be defined for all functions:

   // BasicView object, Papervision3D viewport, responsible for rendering the scene
   private var m_renderView:BasicView = null;
         
   // Camera object, responsible for controlling the camera 
   private var m_camera:ARCamera = null;

   // Tracking object, responsible for tracking         
   private var m_tracking:Tracking = null;
                  
   // flag which will be set to true when camera is initialized and ready
   private var m_cameraReady:Boolean = false;

   // flag which will set to true when tracking is initialised and ready
   private var m_trackingReady:Boolean = false;

   // cube geometry 
   private var m_geometry:DisplayObject3D = null;

Application initialization

As already mentioned the main application logic can be defined directly within the .mxml file. In order to do some initialization we first register the applicationComplete() method as handler for the applicationComplete event which will be raised once the application is loaded. It can be done as showed code snipped below:

<?xml version="1.0" encoding="utf-8"?>
<s:Application xmlns:fx="http://ns.adobe.com/mxml/2009" 
  xmlns:s="library://ns.adobe.com/flex/spark" 
  xmlns:mx="library://ns.adobe.com/flex/mx" minWidth="955" minHeight="600" pageTitle="SimpleAR"
  applicationComplete="applicationComplete()"
  width="640" height="480">
  ...

In the applicationComplete method we create some objects for tracking, camera video stream and rendering.

private function applicationComplete():void
{
    // create papervision3d object 
    m_renderView = new BasicView(this.stage.width, this.stage.height, true, false, CameraType.FREE); 
    // add render view to the stage
    stage.addChild(m_renderView);
    // create camera sprite
    m_camera = new ARCamera(320, 240, stage.width, stage.height, 15, onCameraReady, onCameraError);
    // add camera sprite to the render view
    m_renderView.addChildAt(m_camera, 0);
    // craete tracking
    m_tracking = new Tracking(320, 240, "car.xml_enc", onTrackingInitialized, onTrackingError);
    // init ar scene
    initARScene();
}

In the first step we create an Papervision3D renderer object BasicView and add it to the stage (for more details see the Papervision3D documentation). Next a ARCamera object is created and added to the children list of the Papervision3D renderer object, such that the camera video stream can be displayed in tje background of the AR scene. As next step we create an ARCamera object. The !ARCamera class encapsulates some logic for displaying the camera's video stream as well as for grabbing frames for tracking. Furthermore by creating an ARCamera object the first camera available in the system will automatically be used. As shown above the constructor of ARCamera requires a few parameters wich are listed below:

• desired camera resolution (width, height),
• resolution of the stage (width, height),
• desired framerate of the video stream (fps),
• callback function for the cameraReady event,
• callback function for the cameraError event.

Next the Tracking object is created. The Tracking class is a wrapper for the metaioUnifeyeViewer.swc package and implements some additional logic for controlling the tracking. The constructor of the Tracking class takes the following parameters:

• The resolution of the frames which will be used for tracking (typically it is the resolution of video stream respectively the camera resolution, 320x240).
• The path to the TrackingData file.
• A callback function for the trackingInitialized event.
• A callback function for the trackingError event.

In the next step we will call initARScene which will initialze the AR scene.

Scene Initialization

public function initARScene():void
{                           
   // init field of view of the camera
   m_renderView.cameraAsCamera3D.fov = 44.5;

   // create meterial list for each face of a cube
  var ml:MaterialsList = new MaterialsList();                        
  ml.addMaterial(new ColorMaterial(0xFF0000), "back");
  ml.addMaterial(new ColorMaterial(0xFFFF00), "front");
  ml.addMaterial(new ColorMaterial(0x00FF00), "top");
  ml.addMaterial(new ColorMaterial(0x00FFFF), "bottom");
  ml.addMaterial(new ColorMaterial(0x0000FF), "left");
  ml.addMaterial(new ColorMaterial(0xFF00FF), "right");
  
  // create a cube of side-size 50            
  m_geometry = new Cube(ml, 50, 50, 50, 8,8,8);

  // add cube to the scene
  m_renderView.scene.addChild(m_geometry, "DemoCube");
}

In the initARScene() function we initialize the scene and create some geometry which will used later for the augmentation. First we adjust the Field-Of-View of the camera. In the next step we create a simple geometry. Papervision3D provides a number of geometrical primitives. We will use a simple one: the Cube class (For more information on creating geometry please look at the Papervision3D documentation). As last step we add the created cube to the scene.

Initialization complete

As described in the section Application initialization we provide the constructors of ARCamera and Tracking with some callback functions which will be called as soon as the initialization of these objects is completed or some errors during initialization occured. We will only have a close look at the callback functions which are called when initialization is completed. In our case the functions are defined as follows:

private function onCameraReady(evt:Event):void
{
   m_cameraReady = true;
}
         
private function onTrackingInitialized():void
{
   m_trackingReady = true;
            
   // activate update frame when initialization is ready
   addEventListener(Event.ENTER_FRAME, onEnterFrame);
}

private function onCameraError(type:int, description:String):void
{
   trace("Failure: Cann not initialise camera.");   
}

private function onTrackingError(description:String):void
{
  trace("Error on tracking initialization: " + description);
}

In both cases we use global variables which indicate the initialization state of the camera and tracking. As soon as an object is initialized the appropriated flag will be set to true. In this case the system is ready for tracking. Tracking is proceeded in the main loop of the application. Therefore we register an event listener or handler function onEnterFrame() for the ENTER_FRAME event of the application.

Tracking

As mentioned in section Initialization complete, the onEnterFrame() function will be called each time the application receives a new frame. In this function the tracking of the pattern on the current camera frame will be performed and the scenegraph with respect to the new pose will be updated. The function is defined as follows:

private function onEnterFrame(evt:Event):void
{
   // check whether camera and tracking are initialized
   if (m_cameraReady & m_trackingReady)
   {
      // track current frame
      m_tracking.trackFrame(m_camera.getCurrentFrame(), m_camera.getCameraWidth(), m_camera.getCameraHeight(), m_camera.getChannels());
      
      // get current pose for cos whith the id 1
      var trackingPose:Array = m_tracking.getTrackingValues(1);
   
      // create a Papervision3D matrix which will be store the actual pose
      var matrix:org.papervision3d.core.math.Matrix3D = new org.papervision3d.core.math.Matrix3D(); 
      
      // set the values of the matrix
      matrix.n11 = trackingPose[0];
      matrix.n12 = trackingPose[1];
      ...
      matrix.n44 = trackingPose[15];

      // assign new pose to geometry after conversion of the pose
      m_geometry.transform = convertToTrackingPose(matrix);
   }

   // render the scene
   m_renderView.singleRender();
}

First we check whether the camera and the tracking is successfully initialized. This is done by testing the member variables m_cameraReady and m_trackingReady. Once they are both set to true (which can only be done inside the onCameraReady(), onTrackingInitialized() functions) tracking can be performed by calling the member function trackFrame() of the Tracking object. The trackFrame() function receives the current frame as a ByteArray, the frame's resolution and the number of color channels as parameter. Corresponding functions are provided by the Camera class.

After tracking the frame, the current pose for a particular coordinate system can be retrieved by calling the getTrackingValues() member function of the Tracking class. The pose is returned as an array of 16 Number values which define a row driven/major 4x4 matrix. In our example we retrieve the pose for the fist coordinate system (with the ID 1), and assign the values to a Matrix3D. Note: if the tracking is not successfull a zero matrix is returned (all 16 values of the matrix are set to zero).

In the last step, before assigning the pose to a object, we convert the pose to the left-hand coordinate system of the papervision renderer. Therefore we define a separate function convertToTrackingPose():

private static function convertToTrackingPose( poseMat:org.papervision3d.core.math.Matrix3D ):org.papervision3d.core.math.Matrix3D
{
   var quat:Quaternion = Quaternion.createFromMatrix( poseMat );
   
   // flip rotation around x, y, axis.
   var finalMat:org.papervision3d.core.math.Matrix3D = org.papervision3d.core.math.Matrix3D.quaternion2matrix(-quat.x, -quat.y, quat.z, quat.w);

   // mirror the translation part on z-axis 
   finalMat.n14 = poseMat.n14;
   finalMat.n24 = poseMat.n24;
   finalMat.n34 = -poseMat.n34;
   
   return finalMat;
}

Finally we make the renderer render the scene to the viewport. This is done by calling the singleRender() function.

Congratulations. You just finished the creation of your first AR application using Adobe Flash and the Unifeye Viewer for Flash SWC library. For a more complex example please refer to the FlashARFrame example application.

The application can now be started by pressing the button.

→Return to Examples

-- SupportMetaio - 2010-08-13