Tracking is the process of finding the pose (rotation and translation) of the camera towards the real world and is an essential requirement for Augmented Reality. Only when having a fast, robust and accurate tracking system 3D content can be overlayed/superimposed properly (perspectivly correct) into the real world and in real time. Unifeye Design allows to use and configure the following systems:
The reference pattern/image is identified in the image input (e.g. coming from a webcam) using a built-in image recognition software component and is compared with the patterns stored/configured in the according configuration files. Once detected the rotation and translation of the camera in reference to the pattern/image can be calculated to allow correct superimposition of 3D content into the input image. Thus e.g. putting the reference pattern on a street the information of its perspective in the image will be calculated and for example a virtual vehicle can be rendered into the image in the correct perspective.
Which tracking system you want to use is configured in so called "Tracking Configuration" files, sometimes also refered to as the "TrackingData" file. The tracking configuration file is a XML file. For every tracking system there are specific parameters and options available which are reflected in the configuration file. Fortunately you won't have to edit the XML file manually but there are tools available that allow a step by step graphical user interface based configuration (see sections below). After the configuration file was created you can use it directly with the Unifeye Design GUI or the Workflow Authoring tool/engine by simply loading it.
There is a multitude of different tracking systems and approaches available. If you want to use those, please refer to the Unifeye SDK
or contact firstname.lastname@example.org
A marker is a 2-dimensional optical reference pattern similar to a 2D barcode:
The Marker tracking is based on coded markers which can be configured in arbitrary number and size (but you can't change the inside pattern). These markers were developed by metaio and provide a robust tracking due to a maximum contrast and an integrated error correction mechanism which allows the detection even in relativly low quality images and from flat angles. The system determines the identity of the marker through the inner pattern of the dark squares. Up to 512 different markers can be created using the Marker Generator
tool. In order to allow tracking, always the full marker must be visible.
When you start the Unifeye Design GUI it automatically loads a default marker tracking configuration file with 3 defined markers (ID 1 - 3) of size 140mm. If you print out the provided pdf file
(available via the Windows start menu in the submenu
), activate your camera, load a geometry and put the marker with ID 1 in front of the camera, it will be rendered on top of the marker with ID 1. Please note, that every loaded 3D model will be placed/assigned to the first configured tracking coordinate system (in the default case, the marker with ID 1) per default. You can change that by the according dropdown box within the Unifeye GUI:
You can find more information about coordinate systems here
If you want to use a different marker tracking configuration (more marker, less marker, different size), you need to create an according configuration. Please note that you should not just configure 512 markers and use this configuration always as this heavily impacts performance. It is recommended to always create the configuration file according to your needs.
The creation is done by the "Marker Tracking Configuration" tool within the Unifeye Design GUI. The tool can be found in the tool category "Configuration".
- The first step is to create a markerset which defines the markers and the size of the markers you want to use. Usually you select "Add multiple markers of the same size", then define which markers to use and the according size. After that you will have to save this initial marker set.
- This marker set can now be printed, using the PDF button in the tool. This automatically opens the metaio "Marker Generator" tool:
Leave everything unchanged and simply press "OK". Select a folder where the output PDF file should be save to. Now the "PDF" file with your markers will be created and opened afterwards. Print out the PDF file. In the print dialog make sure to NOT apply any kind of scaling (e.g. "fit to print area") is applied as this will change size and potentially the aspect ratio of the markers (which has to be square).
- Now use your marker set to create the final tracking configuration file. Usually you will create a coordinate system for every defined marker. Do this and select all the defined markers:
- Now save the marker tracking configuration file to disk.
- The final step is to apply the just created marker tracking configuration file. This is achieved by clicking the "Apply" button.
Now your just created configuration is active within the Unifeye Design GUI. If you now save the scene, the saved scene file will contain a link to your just created marker tracking configuration file and will be loaded automatically once you load the scene. Please note that this only works if you do not move the created tracking configuration file to a different folder.
If you only want to load (any) tracking configuration file, e.g. when you try different configurations, you can do so by using the Unifeye Design tool "Tracking Configuration". It provides a file dialog that allows you to load an arbitray tracking configuration file (marker tracking, markerless tracking):
- You should not create one configuration file with 512 markers and use it all the time as this drastically impacts the overall system performance. Better create tracking configuration files according to the really user markers.
- You should provide the correct dimension in millimeter inside the configuration file and not up-/downscale the marker PDF file during/after printing.
- If you need high precision you should provide an appropriate camera calibration.
- You should always leave enough white space around the black border of the print out of the marker:
There are some more advanced configuration options available (e.g. the difference between the fast and robust mode) which are covered here
2D/Planar markerless tracking
Planar markerless tracking uses arbitrary images or so called reference images/patterns/patches as reference to the real world. Reference images can be photographs, illustrations, catalogue pages, print advertisements and so on. In order to be suitable for the system as tracking reference they must be sufficiently well textured and contain enough features for the internal image processing and tracking algorithms. Good reference image usually contain a lot of contrasts, various colors and shapes and/or color gradients. Large, monotonous surfaces and constantly repeating patterns should be avoided. Also only printed text is not well suited for tracking purposes.
The clear advantage of planar markerless tracking is, that in general arbitrary images can be used. Also once initialized not the whole image has to be visible to the camera all the time and can be occluded (e.g. by a person's hand holding the reference image). The disadvantage is a lower accuracy. Also the hardware requirements are higher (a dual core CPU is strongly recommended) and you usually can't use more than about 20 patterns at the same time (this value depends very much on the system performance). Planar markerless tracking is suited for live camera situations and does not work well with still images as image source. Also the tracking pattern should not be too far away from the camera.
There are several example tracking configurations for planar markerless tracking provided within the Unifeye Design installation package which can be found in the subfolders of
using the filenames
If you want to use your own tracking pattern you have to create an according tracking configuration file. Fortunately there is a graphical user interface based configuration tool available that helps you creating a suitable tracking configuration file for Unifeye Design: "Planar markerless configuration". It can be started either directly from within the according menu entry of the Unifeye Design GUI or by using the link in the Windows start menu: "Configuration" -> "Planar markerless tracking configuration".
- The tool is available in a standard (default) and expert mode. Usually the standard mode is sufficient.
- After startup click on the "Create patch" icon, select a well suited reference image and give the dimensions in millimeters of the real (print out) reference. Please keep the aspect ratio and make sure that the aspect ratio in pixels of the image matches those provided in millimeters. Also the image used as tracking reference/tracking patch should be rather low resolution (usually around 400x300 pixel):
- After adding the reference image/patch, click "OK". You can now check how well your choosen image is suited for tracking by clicking on the "Check quality" icon. This will start a simulation process that might take up to one minute. The output is a percentage value. Usually values above 70% indicate a well suited image. Everything below 30% is bad.
- You can now save your planar markerless tracking configuration file and exit the planar markerless configuration tool.
For using the just created configuration file, simply load it into e.g. into Unifeye Design using the "Tracking Configuration" tool.
After starting the camera, loading a 3D model and putting the print out of the reference image in front of the camera, the 3D model will appear on the reference image. Please note that by default a loaded 3D model will appear on the first defined coordinate system (see also coordinate systems
). So if you configured more tracking patches you might have to change the model to coordinate system (COS) binding (as also described above) to see the model on your reference image. When saving the scene now, a reference link to the currently used tracking configuration file will be stored in the scene file itself. When loading a scene again, the according tracking configuration file then be loaded automatically (if the file was not moved or renamed).
- For a more detailed explanation of the planar markerless tracking configuration process please go here.
- If you encouter an unstable tracking the following means can be taken:
- You might define tracking regions within the planar markerless configuration tool. This is also described here: here
- Make sure that the reference image has is suitable with respect to its texture and content.
- Depending on how close it will be looked at the printed image with a camera, the resolution of the reference image used in the tracking configuration should be chosen accordingly. As a rule of thumb, the resolution of the reference image should be similar as it is seen from a "default position" by the camera.
For example: If you want to track a DIN A4 (210mmx297mm) sized print-out and the default resolution of the camera will be 640x480 pixel, the reference image should be scaled to approx. 350x495 pixel. Looking at the whole page with the camera, the printed image seen in the current camera view will have approximately the same size as the reference image. The reference image should not be high-resolution unless you want to look very close at the printed image.
- The digital version should have the same aspect ratio than the printed version (e.g. no severe stretching of the image):
In this example the pixel aspect ratio is the same in the reference image and the printed version: Rx/Ry = Px/Py. When measuring the dimension of the printed version, make sure that it is not rotated 90°
- Tracking Performance: to get a good tracking performance, choose a reference template size verifying:
- (A x B) such that min(A,B) = pow(2,n)*25, where n is a positive integer. “Good sizes” would be: (A x B) where min(A,B) is equal to 25, 50, 100, 200, 400, 800, 1600, 3200,... or slightly above, i.e. (A x B) where min(A,B) is equal to 26, 52, 105, 1606,...Avoid using sizes in-between or sizes that are slightly below these values. “Bad sizes” would then be: (A x B) where min(A,B) is, for example, equal to: 47, 98, 300, 1500, …
- For optimal performance, additionally, choose min(A,B) close to max(A,B). Example of optimal sizes: (100 x 100), (200 x 200), … Condition 2. only makes sense when condition 1. is fulfilled.
Face Tracking (Unifeye Design 2.5)
Unifeye face tracking enables you to detect a face in the input image/video stream, follow its movements and track it in six degrees of freedom.
For the face tracking to work properly, lighting conditions are important. The face should be well lit with the lighting coming from the front and being homogeneous. If the use case allows, the autofocus of the camera should be turned off.
You can test the face tracking by loading the file AS_Tracking_Data_FaceTracking.xml from the config folder in the Tracking Configuration tool. Make sure to use a camera calibration to get the best result.
For a more detailed description please refer to http://docs.metaio.com/bin/view/Main/UnifeyeSDKTrackingConfigurationFaceTracking
Extensible Tracking (Unifeye Design 2.5)
The 3D Extensible Tracking in Unifeye allows the generation of a 3D map of the environment "on the fly". This means, that a 3D map of the environment is created based on the pose provided by a starting tracking system and then constantly extended while tracking. The starting tracking system is used for as long as it is available. If the starting tracking system is lost, the 3D map is used for tracking until the starting tracking system is available again. The 3D map can also be used for relocalisation after the tracking has been lost entirely. This means that if you loose the tracking while the 3D map is being used, you can but you don't necessarily have to go back to the starting tracking system. The tracking will start again as soon as enough features stored in the 3D map are located again or the starting tracking system delivers valid tracking values again.
The 3D Extensible Tracking is well suited if you want to use a certain tracking system (e.g. Marker Tracking or 3D Markerless Tracking) but you need to move around a bit and don't want to pay too much attention to not loosing the tracking target but instead concentrate on your main task. This is very useful in scenarios such as AR-supported maintenance. The Extensible Tracking does not only provide you with some extra moving space but also adapts the 3D map to changing lighting situations. Make sure to use a camera calibration to improve the tracking quality.
For a more detailed description please refer to http://docs.metaio.com/bin/view/Main/UnifeyeSDKTrackingConfigurationExtensible
When working with Augmented Reality you have to deal with coordinate systems (see e.g. http://en.wikipedia.org/wiki/Coordinate_system
), usually (right-handed) cartesian and 3D (see e.g. http://en.wikipedia.org/wiki/Cartesian_coordinate_system
). In Unifeye Design there are several coordinate systems you should know about.
In the most common cases you define one coordinate system per tracking reference (marker or reference image). Per default this is done automatically and the center of this coordinate system is placed in the middle of the according tracking reference with the z-axis pointing out of the tracking reference, the y-axis pointing upwards (north) and the x-axis pointing to the right (east). Every tracking reference has it's own coordinate system and these location and rotation of those coordinate systems are determined during image processing/tracking and are refered to as tracking values:
If you want to determine the location of the coordinate systems in your current configuration (e.g. Unifeye scene) you can activate the visualization of the tracking indicator by using the button in the Unifeye GUI menu called "Show COS indicator and COS labels". This will give you a COS 3D geometry per defined tracking coordinate system (usually one per tracking reference) in the center of the according coordinate system (red bar x-axis, green bar y-axis, blue bar z-axis):
When loading a 3D geometry you bind/assign it to the according (tracking) coordinate system which establishes a connection between the 3D model and the tracking reference and thus the real world. Please note that also a 3D model has a coordinate system whose origin might not be in the center of the model itself or rotated thus causing the model to be rotated or translated when placing on a tracking reference coordinate system.
Inside the Unifeye Design GUI and the Workflow Authoring GUI you can also move, rotate and scale models (in respect to the tracking coordinate system they are bound to). This is done using the "Translation", "Rotation" and "Scale" elements. In the Unifeye GUI those are located in the "Geometries/Planes" tool of the "Objects" category and become available once a model is loaded and selected.
For more advanced configurations you can also define tracking coordinate system offsets and even combine several tracking references (e.g. markers) to create only one coordinate system. This enables you to cover larger areas because the coordinate system will track as soon as one of the according tracking references (e.g. markers) is visible in the camera image: