Tracking Plugin Interface

→Return to Development Section

Introduction

Starting with Unifeye 3.5 a flexible plugin mechanism was introduced that allows the integration of custom tracking modules. Unifeye 3.5 and later provides a header file that has to be implemented by a module which has to be built as a DLL. It is recommended to built the according DLL as Multithreaded(-Debug)-DLL (Visual Studio compiler switch /MDd or /MD).

Once the DLL is created, an according tracking configuration file has to be created. The Sensor type provided in the according tracking configuration file has to match the name of the DLL without the .dll extension. The created DLL file itself has to be placed into the directory next to the Unifeye main module ( AS_AR_Browser.ocx) which is usually located inside the metaio/Unifeye/bin program folder (e.g. C:\Program Files\metaio\Unifeye\bin) and will be loaded automatically once the according tracking configuration file is loaded.

Tracking Configuration

An example tracking configuration file is given in the following section for a plugin tracking module called MySensorSource which would match a DLL called MySensorSource.dll.

The code below specifies the content of the tracking configuration file.

• In the Sensor section a Sensor of type "MySensorSource" is defined with a unique SensorID.
• The Parameters section contains an enclosing pair of XML tags called <MySensorSourceParameters> and one parameter called <myParameter1>. Note that your custom tracking/sensor source plugin DLL will receive the XML string below the tag <Parameters> in the init() method and you can specify any parameters in XML format you need. In general it is a good approach to add another pair of enclosing XML tags as done here by <MySensorSourceParameters>.
• Then a SensorCOS is added, configuring one tracked COS for your tracking plugin. A SensorCOS consists of:
  • A unique SensorCosID
  • parameters specific for this SensorCOS. In this case, only one parameter is specified ("<mySensorCOSParameter1>"). Note that your custom tracking/sensor source plugin DLL will receive the XML string below the tag <Parameters> of the SensorCOS section for every configured SensorCOS via the addSensorCOS() method. For every configured SensorCOS this function will be called with the according parameters. In general it is a good approach to add another pair of enclosing XML tags as done here by <MySensorSourceCOSParameters>.
• Then the Connections are defined, which link a SensorID with a SensorCosID in terms of COSes.
• In our case one COS is defined which
  • has a name ( Cos1),
  • uses a Fuser of type BestQualityFuser,
  • links the SensorID "MySensorSource1" with the SensorCosID "sensorCOS1",
  • holds a HandEyeCalibration transform which is the identity (no translation and no rotation),
  • holds a COSOffset transform which is also the identity.

Example Tracking configuration

 
<?xml version="1.0"?>

<TrackingData>
<Sensors>

   <Sensor type="MySensorSource"><!-- Name of the according sensor source -->
      <SensorID>MySensorSource1</SensorID><!-- give it an ID, as there might be several sensors of the same type -->
      <Parameters><!-- sensor specific parameters -->
         <MySensorSourceParameters>
            <myParameter1>100</myParameter1>
         </MySensorSourceParameters>
      </Parameters>
      
      <SensorCOS>
         <SensorCosID>sensorCOS1</SensorCosID>
         <Parameters><!-- sensor cos specific parameters -->
            <MySensorSourceCOSParameters>
               <mySensorCOSParameter1>dummyValue</mySensorCOSParameter1>
            </MySensorSourceCOSParameters>
         </Parameters>
      </SensorCOS>
   </Sensor>
</Sensors>   
<Connections>
   <COS>
      <Name>Cos1</Name>
      <Fuser type="BestQualityFuser">
         <Parameters></Parameters>
      </Fuser>
      <!-- There are 1-n SensorCoses -->
      <SensorSource trigger="1">
         <SensorID>MySensorSource1</SensorID>   <!-- Who tracks... -->
         <SensorCosID>sensorCOS1</SensorCosID>      <!-- what? -->
         <HandEyeCalibration>
            <TranslationOffset>
               <x>0.0</x>
               <y>0.0</y>
               <z>0.0</z>
            </TranslationOffset>
            <RotationOffset>
               <x>0.0</x>
               <y>0.0</y>
               <z>0.0</z>
               <w>1.0</w>
            </RotationOffset> <!-- e.g. Transform between camera and ART body -->
         </HandEyeCalibration>
         <COSOffset>
            <TranslationOffset>
               <x>0.0</x>
               <y>0.0</y>
               <z>0.0</z>
            </TranslationOffset>
            <RotationOffset>
               <x>0.0</x>
               <y>0.0</y>
               <z>0.0</z>
               <w>1.0</w>
            </RotationOffset><!-- offset like before -->
         </COSOffset>
      </SensorSource>
   </COS>
</Connections>
</TrackingData>

Implementation of a tracking/sensor source plugin

For implementing a custom tracking DLL you have to create a DLL that implements the header file metaio_IDLLSensorSource.h to be found in the /config/plugin subfolder of your Unifeye installation. An example can be found in the /examples/plugin directory. It contains a Visual Studio project file ( DynamicallyLoadedSensor.vcproj) file which you can open and contains two example implementations:

• metaio_FramesTriggeredSensorExample.h/cpp will just output tracking values upon every new camera frame it receives and runs out of the box,
• metaio_OpenThreadsSensorExample.h/cpp contains a threaded example that outputs tracking values independant from the camera image, i.e. threaded by interpolating the translation for x,y,z between given start and end values. To keep the example simple, rotation was not taken into account. Please note that you will need the OpenThreads library in order to use this project, but even without it it should give you a clear idea how things work.

For choosing the example you want to use, adjust the file metaio_DLLProcedures.cpp accordingly (see the first #define and #include in the file).

In general you can choose if you want to receive images from the currently active visualization camera or not. The first case makes sense for implementing an optical tracking algorithm, the second if you e.g. want to add a hardware tracking device that doesn't need the visualization image to work with. Pose information is to be returned to the system via the function pointer given by the init() method. The following functions have to be implemented:

• virtual bool init(const std::string& xmlParams, const std::string& trackingDataPath, void (*pCallback)( const std::vector &,void * ), void *pContext, bool *pIsFramesSourceTriggered); This function is called by the system while loading the tracking configuration file and is used to setup your custom tracking module/sensor source.
  • xmlParams contains the sensor source specific XML parameters contained in the loaded tracking configuration file. In the example above everything within the tag <Parameters> which is:

    <MySensorSourceParameters>
        <myParameter1>100</myParameter1>
    </MySensorSourceParameters>

  • trackingDataPath contains the absolute path to the loaded tracking data file. This information is helpful if you e.g. specify files in your sensor specific parameters that should be resolved relative to the loaded tracking configuration file.
  • ( *pCallback)( const std::vector &,void * ) contains a pointer to a function with two parameters that has to be called by your sensor source if new tracking information is available and should be forwared to the system. The first parameter contains a vector of timestamps and poses (tracking information). The timestamps must be in milliseconds and contain the value of a high resolution performance timer of your system (see Windows API function: QueryPerformanceCounter, e.g. at http://msdn.microsoft.com/en-us/library/ms644904(VS.85).aspx). Note that correct timestamps are important if you want to receive a properly synchronized augmentation with a video image. The COS id is the name of the SensorCOS as specified in the tracking configuration file. The pose itself is given as translation (in milimeters) and rotation as Quaternion. The second parameter is a pointer and MUST contain the pointer given in pContext when you call the callback function to signal the system new tracking information.
  • pContext contains a pointer needed in the callback function to let the callback work properly. This pointer should be stored within the implementation of your class and MUST be given when calling the callback function for signalling new tracking data.
  • pIsFramesSourceTriggered is an output parameter that has to be set to true if you want to receive images from the currently set visualization camera (see function onNewImage()).
• virtual bool addSensorCOS(const std::string& cosID, const std::string& cosParameters, const std::string& trackingDataPath, bool *pIsOnlyOneCOSAllowed); This function is called by the system during loading the tracking configuration file for every configured SensorCOS and should be used to perform the per SensorCOS configuration of your tracking module/sensor source.
  • cosID contains the ID of the SensorCOS that is to be configured by this function call.
  • cosParameters contains the SensorCOS specific XML parameters for the SensorCOS to be configured. This is everything within the tag <Parameters> of the according SensorCOS. In the example above this is:

    <MySensorSourceCOSParameters>
        <mySensorCOSParameter1> dummyValue </mySensorCOSParameter1>
    </MySensorSourceCOSParameters>

  • trackingDataPath contains the absolute path to the loaded tracking data file. This information is helpful if you e.g. specify files in your SensorCOS specific parameters that should be resolved relative to the loaded tracking configuration file.
  • pIsOnlyOneCOSAllowed is an output parameter that has to be set to true if your custom tracking module/sensor source is capable of tracking only one SensorCOS. This is for example the case if you add support for a hardware tracking device that has only one sensor to track, like a compass.
• virtual bool start(); This function is called by the system once per tracking module after the complete tracking configuration has been loaded. If your custom tracking module/sensor source requires a separate thread to be started, this should be done within this method. The function has to return true if it contains a separate thread and false otherwise.
• virtual bool onNewImage(const metaio::ImageHeader& imageHeader, const metaio::Camera& camDist& camDist, double timestamp)); This function is called by the system if you have opted for receiving images of the visualization camera (which is the currently loaded image/video or camera) in the call to the init() function.
  • imageHeader contains the image as a const pointer and information about the given image (like resolution, number of channels). The image will already be undistorted if an according camera calibration (Sextant or Extended Sextant) is loaded into Unifeye. You should NEVER try to alter the given image. Otherwise a system malfunction might occur. Also note that you should omit every needless copy operation of the given image as this introduces performance penalties to the whole system especially in HD image scenarios.
  • camDist contains parameters describing the intrinsics of the camera for the given image (focal length, principal point, distortion parameters).
  • timestamp contains the timestamp of the image in milliseconds according to a high resolution performance timer of your system (see Windows API function: QueryPerformanceCounter, e.g. at http://msdn.microsoft.com/en-us/library/ms644904(VS.85).aspx).

Besides the functions mentioned above you have to implement a create and destroy function which are called by Unifeye to create and destroy the according custom tracking module/sensor source:

• extern "C" __declspec(dllexport) metaio::IDLLSensorSource* Create(); This function is called by the system after loading the DLL for your custom tracking module/sensor source and should return a pointer to a class that implements the functions listed above. A simple example implementation would look like the following and return a pointer to a new instance of the class called MySensorSource.

  extern "C" __declspec(dllexport) metaio::IDLLSensorSource* Create()
  { 
          return new metaio::MySensorSource();
   }

• extern "C" __declspec(dllexport) void Destroy(metaio::IDLLSensorSource* p); This function is called by the system when it has to destroy your custom tracking module/sensor source. This happens e.g. when loading a new tracking configuration file or at system shutdown. It should destroy the given instance of a class created by the Create() function. A simple implementation would look like this:

  extern "C" __declspec(dllexport) void Destroy(metaio::IDLLSensorSource* p) 
  {
          metaio::DummySelfTriggeredSS *pDerived = dynamic_cast(p);
          if (pDerived != NULL) 
          {
                  delete pDerived;
           }
  }

  • p contains the pointer returned by the Create() function and can be used to destroy the according object.

→Return to Development Section

-- SupportMetaio - 2011-01-28