3D Markerless tracking

→Return to Tracking Configuration

Introduction

3D markerless tracking can be used to track 3D objects and environments. For using 3D markerless tracking a 3D map of the object/environment has to be created first.
The Markerless Tracking Map Generator allows a 3D reconstruction of the environment based on a set of images acquired by one camera. Using those images, it computes a (colored) point cloud representation of the environment. This point cloud is saved to a so called 3D map. The Markerless Tracking Map Generator also creates a tracking configuration file refering to the 3D map. If this tracking configuration file is loaded into Unifeye, the 3D object represented by the point cloud is tracked.

Data preparation

Calibrating the camera

The first thing to do is to calibrate the camera that will be used to acquire the images of the environment. For this the Unifeye Sextant tool or the Unifeye Extended Sextant tool can be used. It is possible to use other tools as well, but the calibration result needs to be stored in one of these two formats: the Sextant format (.xml file) or the Extended Sextant format (.bin file). The camera calibration file needs to be saved and given as input to the Markerless Tracking Map Generator tool.

Preparing the environment that shall be reconstructed

Some markers have to be placed in the environment that needs to be reconstructed. The markers allow the reconstruction tool to generate a map at the correct scale and make it possible to recover the camera position with respect to the environment in the different pictures. Once the set of markers to be used has been chosen, the user needs to configure the markers, i.e. create a Tracking Configuration file that contains the different markers IDs and sizes.

First, in order to create the markers, the Marker Generator tool of the Unifeye software can be used. This tool creates a .pdf file that contains square markers with the desired IDs and sizes. The .pdf file should then be printed and the markers can be attached in the environment on planar surfaces.

Second, the Marker Tracking Configuration tool of the Unifeye software can be used to create the Tracking Configuration file. The user needs to create one coordinate system to which the desired markers are attached. The resulting (.xml) file needs to be saved and given as input to the Markerless Tracking Map Generator tool.

Acquiring images

Using the calibrated camera, the user would need to acquire images of the environment to be reconstructed. Some images need to have at least one marker completely visible. Few images need to contain several markers visible in order to make the inter-marker calibration possible. The images should be acquired with the same value of the focal length as the one used during the camera calibration. The images need be acquired with (or converted to) the .bmp or .jpg formats. The best results are obtained with images acquired from several different viewpoints and containing many markers. The user needs to put all images in the same folder. The folder path will be given as input to the Markerless Tracking Map Generator tool.

Performing the reconstruction

The Markerless Tracking Map Generator can be started using the windows start menu or from within Unifeye using the Tools menu:

Input data

The following input data is needed for the tool to be able to perform the 3D reconstruction:

• Image Folder: The image folder corresponds to the path of the folder in which all the images that are going to be used for the reconstruction are located (described here).
• Tracking Data: In this field, the path of the tracking configuration file (described here) should be given.
• Camera Calibration: In this field, the path of the camera calibration file (described here) should be given.

The needed files/folders can be selected by either typing the path into the textfields or by clicking on the button on the right side of the text field. This will open a file/folder selector dialog.

If the Markerless Tracking Map Generator has been started via the windows start menu, the textboxes will be empty upon startup. If the tool has been started from within Unifeye, the textboxes will contain pathes if an Image Folder, a Tracking Data or a Camera Calibration has already been loaded in Unifeye. If this is not the case, the textboxes will be set to "undefined".

Settings

Several settings can be altered to adapt the 3D reconstruction to your specific needs. The default values have been carefully chosen by us to deliver good results in most cases. You can always return to those settings by pressing the "Reset all settings to default values" button.

• Max. Features: In this field, the user provides the maximum number of the feature points that are going to be extracted in every image. This parameter generally depends on the environment nature, on the image resolution and on the processing speed. If the user prefers having only strong features to be detected, this number should be low (<1000). If the user prefers having all possible feature extracted, this number should be high (>2000). In this case, the reconstruction process may take several minutes. For a standard usage of the application, let us say using an image resolution 640x480 and in a 5x5 meter fairly textured environment, one can choose a value ranging between 500 and 1500.
• Matching Strength: This parameter is slightly more technical. It influences the number and the quality of the reconstructed 3D points. This parameter should vary between 0 and 1. By default, it is set to 0.8. This default value showed the best compromise between quantity and quality. This default value needs only to be changed in case the resulting reconstruction is not satisfactory, e.g. in case the user is reconstructing an environment with many repetitive structure the default value should be decreased, and in case the image quality is very low (e.g. noisy) the default value should be increased.
• Max. Epipolar Dist.: This parameter is also quite technical. It influences the reconstruction of the extracted feature points in the images. A low value of this parameter (<1.0) is quite restrictive and generates a map containing a small number of 3D points that are quite precisely reconstructed. A high value (>5.0) will result in a higher number of 3D points that are less precise. The value 3.0 showed to be a good compromise between quantity and quality.
• Num. Observations: This parameter determines the minimum number of images in which the same physical point should be detected in order to be reconstructed. The higher this number is, the better the 3D points are reconstructed. However, if it is too high, the number of the reconstructed 3D points will be very low. A good compromise is to set the value of this parameter to 3.
• Min. Track. Quality: If the markers in your image set are not tracked, you can decrease this value. Be aware that you will be able to use pictures with quite distorted markers if you decrease this value but the tracking accuracy also decreases. You should not choose values below 0.7.
• Min. Baseline [mm]: Increase this value if you get unsatisfactory results when reconstructing objects that are far away from the camera.
• Do Intermarker Calibration: When this check box is checked, the inter-marker calibration is performed before the reconstruction. The user simply needs to configure the markers attached to the environment by specifying the sizes and the IDs when generating the tracking configuration file (explained here). In case the user has placed the markers precisely and knows the transformation between them using some extra measurement tools, he can specify the inter-marker calibration directly when generating the configuration file. In this case, the check box should be unchecked and no inter-marker calibration needs to be performed.
• Reference COS: In case the “Do Intermarker Calibration” check box is checked, the user needs to specify which of the configured markers should be considered as the reference marker. All other marker transformations will be computed with respect to this reference marker. The position of all the 3D reconstructed points will be computed in the coordinate system defined by this marker.
• Generate only points for tracking: This checkbox is used to decide between the map generation for only tracking and only reconstruction purposed. For the creation of a 3D markerless tracking configuration you should always leave this box checked! In case that this option is checked only points which can be used for tracking are generated and displayed in the 3D preview window. Otherwise a denser map is generated which can be used for reconstruction tasks. The reconstruction map will be only exported as an XYZ map.

Output data

Before being able to perform the 3D reconstruction a few pathes have to be selected to save the results. The needed files can be selected by either typing the path into the textfields or by clicking on the button on the right side of the text field. This will open a file selector dialog.

• Tracking Config: In this field, a path for saving the resulting tracking configuration file must be given. The resulting tracking configuration can then be loaded in the Unifeye software in order to perform the markerless tracking. If the Markerless Tracking Map Generator has been started from within Unifeye, you will be asked if you want to load the computed tracking configuration when you close the tool and return to Unifeye.
• Map MUTF Data: In this field, the path for saving the resulting MUTF map must be given. MUTF is a data format created by metaio that consists in the reconstructed 3D point cloud and the mapping information between the 3D points and the corresponding 2D points in the image set used for the computation. The MUTF map file should be located in the same folder as the resulting tracking configuration file when the resulting tracking configuration file is loaded in the Unifeye software.
• Map XYZ: In this optional field, the path of the resulting XYZ map that consists in the reconstructed 3D point cloud can be given. The XYZ map contains the 3D position of the points reconstructed in a readable file. It also contains the RGB color for every point as it was in the input images. Every line of this file contains the X Y Z R G B values of one reconstructed 3D point. In case this field is left empty, the XYZ map is not created.

Computing the 3D map

Clicking on the "Generate 3D Tracking Map" button will activate a check if all needed input and output information is given. If that's the case, the computation of the 3D map is started. Some feedback about the computation being performed is displayed in the status textbox and the progress bar is activated. Please note that the computation might take some time. The computation time depends on the hardware used, the number of images provided as well as their solution and the settings chosen.

Inspecting and editing the reconstruction

In the center part of the Markerless Tracking Map Generator you will find a renderer window, where the 3D points of a generated (or loaded) map are displayed:

You can navigate through this 3D point cloud using the mouse and the keyboard:

• press left mouse button + move mouse: rotate scene around its center
• press middle mouse button + move mouse up (or scroll mouse wheel up): zoom in
• press middle mouse button + move mouse down (or scroll mouse wheel down): zoom out
• press shift key + press left mouse button + move mouse (or press arrow keys on keyboard): pan in scene

If after some navigating you wish to return to the original view, just click on the "Reset view" button in the upper right corner.

The navigation as described above enables you to move to specific points and select them:

• Imagine a frame around the points you want to select --> right-click on one corner of that imaginary frame --> right click on the corner of the imaginary frame located diagonal to the first corner --> all points within the imaginary frame will now be selected and are displayed in red.
• To add points to the selection, repeat the above but press the shift key while right-clicking.
• To remove points from the selection, repeat the above but press the ctrl-key while right-clicking.
• Do deselect all points, click on the "Clear selection" button.

Displaying images corresponding to selected points

To decide on whether or not you want to keep 3D points for your tracking configuration, you can inspect the images the points have been extracted from by clicking on the "Show corresponding images" button. The corresponding 2D position of the 3D points is marked with a red circle in the images. This way it is easy to decide whether the 3D point is part of the object that you want to track or if it is part of the surrounding and should therefore be ignored. The images are displayed pagewise. There display width and height as well as the number of images per column and row can be manually set and thus be adapted to your preferences and to the size of your display. To navigate to the next page of images click on the ">>>" button. To go back one page click on the "<<<" button.

If you have selected several 3D points, one image will be displayed for each selected point. In the example below, you can see that all images on the displayed page contain features belonging to the object that is supposed to be tracked.

If you have selected exactly one 3D point, all images used for extracting this point will be displayed. In the example below you can see that the selected point is not located on the object that is supposed to be tracked.

When you are done inspecting the images, just click on the "Close all image windows" button.

Deleting points from map

To remove points from the map that don't belong to the 3D object that is supposed to be tracked, there are several options:

• Delete selected points
• Delete all but selected points
• Delete points with opened image

The first option is pretty obvious. The second option can be used for cropping the point cloud. As you can see below, features belonging to the 3D object that shall be tracked are grouped locally. Features outside of this accumulation are likely not to belong to the object. To remove all of those outside features, the easiest way is to select all points belonging to the object and then remove the rest.

The third option should be used together with the inspection of corresponding images as described above. You should select all points that you suspect of not belonging to the object you want to track. Then click on the "Show corresponding images" button. If the images displayed show that in fact some of those points do belong to the tracking object, close those image windows. When only images are still displayed that show features you don't want to use for tracking, click on the "Delete points with opened image" button.

You can recover deleted points by clicking on the "Undo delete points" button. This does not work anymore after having saved the map !

Loading and saving maps

After generating a 3D Tracking Map, it will automatically be saved to the path you have defined in the "Output Data" section.

Maps can also be saved manually, e.g. after removing some points. To save a map, click on the "Map Menu".

You will be displayed three options:

• Load Map: Loads any 3D map that has been created with the Markerless Tracking Map Generator

• Save map as RAW: This saves a version of the 3D map containing the 3D position of the points only. The color information and the information of the corresponding 2D points in the provided image set will not be saved. This means that when loading this map, you will not be able to display the images corresponding to selected points anymore!

• Save map as MUTF: This saves a full version of the 3D map containing all its information.

Merging maps

The Markerless Tracking Map Generator comes with a small tool allowing to merge several input maps to one output map. The Map Merger Tool can be started from the "Tools" menu.

First, you have to add all 3D maps that are supposed to be included into the result map to the Input list. This is done by clicking on the "Add input file(s)" button which opens a file select dialog, which is multiselect enabled so several maps that are located in the same folder can be selected together.

CAUTION: At the moment we assume that all maps have the same origin and orientation. They also have to be of the same type: You can only merge RAW maps with RAW maps and MUTF maps with MUTF maps. If you mix RAW and MUTF maps you will get an error message.

You also have to define result file for the merged map. This can be done by either typing the path into the Output textfield or by clicking on the button on the right side of the text field. This will open a file selector dialog.

The result file will be computed after a click on the "Merge input files and save resulting map to output file" button. Please note that the computation might take some time and the window might freeze during that computation process.

Using 3D markerless tracking configuration files in Unifeye

To activate 3D markerless tracking within the Unifeye SDK you simply have to load an according tracking configuration file and use the API function setTrackingData() (see http://doxygen.metaio.com/UnifeyeSDK/UnifeyeSDK_doxygen/group__Configuration.html). You can also load the according tracking configuration file within the Unifeye GUI using the Tracking Configuration Tool.

Further notes

The reconstruction done by the Markerless Tracking Map Generator is originally intended to be used for markerless tracking. However, it can also be used by any other application that requires a (colored) point cloud representation of the environment.

The output of this tool has successfully been tested in the following application fields:

• markerless tracking

• discrepancy check

• creation of 3D mesh of the environment.

→Return to Tracking Configuration

-- SupportMetaio - 2011-01-21