FlashARFrame: A complex AR application

→Return to Examples

Introduction

The Unifeye Viewer Flash ships with two different example applications. The most basic application is the FlashSimpleAR application and the advanced application is the FlashARFrame. In this section, the FlashARFrame is explained in detail. The FlashARFrame reads in a configuration file (see section "configuration without programming"), can handle multiple 3D model loading, animation playback of 3D models, sound playback, error checks and GUI components. You can use this application as template for your own application development. It is very extensible and using it as template might speed up your development process. The FlashARFrame application can be configured via an application configuration file (see ApplicationConfiguration.scef) for greater flexibility. It is the same file we use in the ShockwaveARFrame. Most of the features of a ShockwaveARFrame specific ApplicationConfiguration are also available in the FlashARFrame and vice versa. The ApplicationConfiguration is a human readable XML file that defines the 3D models you want to use, the sound you want to play and so on. For the full documentation of the configuration file please refer to the ApplicationConfiguration.scef file and the section "Configuration without programming".

Setup the FlashARFrame in Flashbuilder

1. Open Flash Builder
2. Click on File > Import > Flash Builder Project
3. In the Dialog select "Project Folder", select the "FlashARFrame" root folder and click on "Finish".
   The Project should appear in your workspace.
4. Click the "Run" Button to start the Application.

You can add the UnifeyeViewerToolkit library as a Project as well when you want to have access to it.

Software Architecture

The main class is the FlashARFrame mxml application class. It initializes, aggregates and controls all classes that are necessary to run an AR based application. All these classes are packaged into a flash library called "UnifeyeViewerToolkit", the toolkit itself includes the Papervision library for Rendering and the MetaioUnifeyeViewer library for markerless 2D tracking. The class diagramm above hides several interface implementations and internal classes which can be viewed when you open the UnifeyeViewerToolkit library. These classes are not relevant for application development and therefore not available over the library boundaries. However, when you want to extend the Toolkit, for example adding a new resource types, you will have to open the library and have to have a look into the implementations.

The Papervision Rendersystem is abstracted through IRenderSystem and IModel and therefore exchangeable. Adobe stated it will present a new hardware driven Renderer in October 2010. Our abstraction layer will make the future integration a feasible task.

The MetaioUnifeyeViewer library is wrapped directly by the Tracking class. You can send an image to the Tracking instance and the Tracking instance will forward it to metaios UnifeyeViewer tracking library. Then the image is searched for markerless 2D tracking patterns which had been defined in the Tracking data before. All tracking patterns that were found in the image are used to calculate their pose in 3D space. The 3D pose for every tracking pattern is then stored and can be queried from the Tracking class.

The most important packages and minimal required to build an AR based application with the UnifeyeViewerToolkit are tracking, camera, render and resources. The tracking package simplifys the tracking process and offers some optional tools like the TrackingObserver which can keep the tracking alive to remove model flickering. The camera package gives you control over the webcam and the video feed it delivers to the rendering and the tracking. Optionally you could also track still images without the need of a webcamera. Then you could, of course, omit the usage of the camera package. The next usefull package is the render package. It manages the scene graph setup and structure and renders 3D models onto the screen and on top of the video image.The last important package is the resource package. It reduces the dynamic file loading complexity and centralizes the resource storing. If you want to completely build up the render system or the resource management yourself, you will have to skip using both packages because of the resource loading of models depends on the renderer. Another possibility is modifying the UnifeyeViewerToolkit implementation and adding your own renderer or resource management.

Other available packages are tools, loader and XML. These packages are mostly optional and are not needed for your AR application. The tools package gives you mostly static functions for different purposes like string functions and math functions. The loader package contains a customized preloader which replaces the standard preloader of FLEX in the FlashARFrame. Lastly, the XML package is used to parse the ApplicationConfiguration.scef file which is used to configure the FlashARFrame. The ApplicationConfiguration class can be passed directly to the ResourceManager to load all the resources defined there.

Initialization Process

The image above shows the initialization process of the FlashARFrame which is controled by the FlashARFrame mxml main class. The main class calls load() on the ApplicationConfiguration class. Now the XML file is loaded, put into a class structure and parsed for correctness. If no errors occured, the ResourceManager is initialized and all resources in the ApplicationConfiguration are loaded. During the load process, a progress bar will be displayed by the main class. After the resources had been downloaded successfully, the Camera gets activated ARCamera() and the tracking is finally started Tracking(). Now the FlashARFrame enters the main loop and the EnterFrame flash event is used to control the Application logic (see Application Flow).

The two load() calls and the camera activation ARCamera() are asynchronous calls. That means, that you could execute some logic in the meantime until they are finished. The FlashARFrame waits for each step to complete, but of course, this is not necessary and you don't have to wait for each step to complete in your application, but its recommended because of the steps depend on each other.

Application Flow

The image above shows you the main application logic. This logic can be replaced completely by your application logic, it is just the default application logic which is defined through the ApplicationConfiguration.scef file.

The Application Flow:

1. We check if the next frame should be skipped or not. This happens with every N'th frame when the Unifeye Viewer searches for tracking patterns (we call this: initialization phase), and does not track any pattern. The initialization phase somtimes leads to heavy CPU load and therefore we highly recommend to skip every third or even second frame. This will improve the performance drastically on initialization and lead to a smoother video image display without any tracking pattern detection degradation.
2. We update the tracking observer. It tells us which Coses are tracked and which Coses are not tracked.
3. Check if the tracking pattern which was tracked last has still a valid tracking pose. If it's still valid, we update the pose of the Cos for the renderer and all models which are bound to theCos of the tracking pattern are repositioned in space, then we wait for the next loop and return to 1. If the pose not valid anymore, we jump to 4.
4. Check if any of the other Coses which has been defined in the tracking data is now tracked. If another Cos is tracked, we hide all models bound to the last tracked Cos, stop their animations, movie textures and sound file playbacks. In return, the models which are bound to the new tracked Cos are displayed and their animations, movie textures and Cos Sound files are started and we jump to 1. If no other Cos is tracked, we jump to 5.
5. The tracking of the last tracked pattern interupted. Check if the interuption duration is still smaller than the threshold given by <trackingTolerance> tag in the ApplicationConfiguration file. If the interuption duration is smaller than the threshold, we jump to 1. If it's bigger, we hide all models bound to the last tracked Cos, stop their animations etc. and jump to 1.

Configuration without programming

The FlashARFrame example application ships as Flash Builder project and as a published .swf movie version. You can integrate the published version directly into your HTML site and configure it via the ApplicationConfiguration.scef file without programming. Customizations can be done by exchanging or modifying the tracking configuration file which is referenced in the ApplicationConfiguration and/or the ApplicationConfiguration.scef file itself. You can edit both readable XML files by using a text editor.

Section ApplicationConfiguration will explain how to configure the application and how to exchange and modify media content. Section "Content preparation" will explain how to prepare such content. More details about content generation in general can be found in section ContentPreparation.

Content preparation

You can load different types of content, for example 3d models, sound files and movie textures into your application via the ApplicationConfiguration. This section explains how to prepare and create such contents.

1. 3D models
   1. The FlashARFrame can load Collada and MD2 models. You can include geometries into the application via the "geometry" tag. Please refer to section ContentPreparation for detailed information.
2. Sounds
   1. Create sound files with your favourite sound tool and save them in the mp3 format. Reference the sound files through “audio” tags in the ApplicationConfiguration.scef file.
3. Movie Textures
   1. You can use “flash video files” (.flv) or any “shockwave movie file” (.swf) for movie textures. You can create your own “swf” movie files using the Adobe Flash Authoring Tool. Flash video files can be created with an accordant video tool. Make sure that the video resolution isn't higher than 320x240 or the performance will decrease noticeable.
   2. If you want to place a movie texture on a 3D model, you have to specify the movie texture parts of the 3D model accordingly. This is done by assigning a special material to the movie texture parts. Inside your 3D modelling tool, create a standard material and include the substring “metaioMovieTexture” in the name of the material. Assign this material to all 3D model parts that should contain the same movie texture. Afterwards connect a movie (.swf or .flv) to your geometry (Collada) using the configuration options inside the ApplicationConfiguration.scef file.

→Return to Examples

-- SupportMetaio - 2010-08-13