WORKFLOW AUTHORING GUI

The Workflow Authoring GUI is the main application for the creation of individual AR scenarios. It offers complete freedom in the creation and implementation of ideas. The user has the option to group and connect virtually all functions and settings of an AR application in a so called "Workflow". With a simple button click such a "Workflow" can be started and runs automatically within the Workflow Engine. This allows the creation of complete scenarios, which operate according to a well-defined operation chart. For example, a live show with augmented reality, in which at the push of a button a virtual 3D car model appears on a real stage and when re-pushing the button a virtual hood opens up. Through the use of so-called "Actions", which are the "building blocks" of each Workflow, extensive configuration possibilities arise with relatively low complexity. Through the use of the workflow engine, the full scope of possibilities of Unifeye Design is opening up

GUI Overview

This is the main window of the Authoring GUI. On Top you find the menu bar which contains several buttons. To the left you can find all available Actions. The big workspace to the right is used to create your Workflow.

Menu bar

The menu bar consitsts of several buttons (numbered from 1 to 10). Amongst others, they allow the user to save, load and run the workflow. Each button is explained in detail below.

1. Recent Workflows

Select an entry from the list to quickly load a recent workflow.

Every Workflow you save / load appears in the list of Recent Workflows. The Workflow you did save / load last, will be the first in the list.

2. New (Strg + N)

Resets the current Workflow and creates a new one.

3. Open (Strg + O)

Opens an existing Workflow file.

4. Save (Strg + S)

Saves the current Workflow.

5. Save As (Strg + S)

Saves the current Workflow to a specific file.

6. System Connections

Opens the System Connections window.

System Connections provide 3 different types of global functionality that are executed when you press the key which is assigned to the connection. The following logics are available:

A. Restart Workflow: The execution of the Workflow is stopped and starts again from the beginning.
B. Stop Workflow: The Workflow Engine will not react to any user inputs until you press the assigned key again.
C. Pause Workflow: The running Workflow is paused and any Action logic is stopped until you press the assigned key again. Additionally the Engine will not react to any user input.

7. Resource Editor

Opens the Resouce Editor window.

Model resources are handled globally through the Resource Manager. You can add, remove or modify models by clicking on the respective buttons. If you click on the "Add" button, a file browser will pop up and you can locate a model an give it a name. It is not possible to have two models with the same name because of the name is an ID that is used to identify models throughout the workflow. If you want to use the same VRML file twice, add a second model with a unique name that is pointing to the same VRML file as your first model.

Hint: When the Workflow is running, you need to load each model before an Action can use it. Loading a model is done via an Action that loads models. This might be the "Load geometry", "Load multiple geometries" or "Load occlusion geometry" Action for example.

8. Action Paths

Opens the Action Paths window.

Each Action Path defines a location on your hard disk where custom Actions are stored. Custom Actions, so called Ufa-Actions are XML-files containing C# code logic.

You can add new Action directories by clicking the "+" button in the lower left corner or by right clicking the Action list window and selecting "Add new directoy".

You can delete Action directories again by right clicking an existing directory and selecting "Delete".

9. Quit

Exits the Workflow Authoring GUI application.

10. Run (Strg + F5)

Runs the current Workflow in a new window.

Action Tree

The Action tree view (2) displays all Actions that are currently available. Each Action is placed in a group that can be expanded or collapsed by clicking on the small "+" or "-" sign. In order to expand or collapse the whole tree, right click the Action tree view and select "Expand Action tree view" or "Collapse Action tree view"

On top you can find the Action search box (1). Enter a string and the system will look for Actions containing the string. If the system finds an Action that matches the string, it will highlight the Action in light gray. It is possible that more than one Action contains the entered string. Press F3 in order to cycle through all found Actions. If you press ENTER, the highlighted Action is created at the workspace.

A description of each Action is shown in the lower text box (3) after you selected an Action from the Action tree view.

Action Creation

There are 4 different ways you can create an Action from the Action Tree:

1. Double click a listed Action.
2. Hover over an Action with the mouse and press "Enter".
3. Click and hold an Action and drag it onto the workspace with the mouse.
4. Search for an Action and press "Enter" and the highlighted Action is created.

Workspace

Each workflow contains of Actions which define the application logic. All Actions are placed, edited, connected and selected on the workspace.

Workspace panning

Click and drag the workspace with the middle mouse button to pan the workspace view.

Expand / Collapse Actions and Connections

You can expand and collapse each Action and Connection panel by clicking on the corresponding collapse or expand-triangle button. Doing so will give you a better overview of the Workflow. Additionally you can expand or collapse all Actions at once. Just right click the Workspace and press on "Expand all Actions" or "Collapse all Actions".

Delete Actions and Connections

Actions and Connections are deleted after you clicked on the red "X" button which is placed in the top right corner.

Position Actions

You can position Actions within the workspace area by clicking and dragging their header. It is also possible to position more than one Action at once. This is done by selecting multiple Actions before dragging. Just left-click on the workspace and drag the mouse to span a selection frame. Release the mouse and all framed Actions get selected. All selected Actions are marked with a blue dotted border afterwards. If you position one of the selected Actions through clicking and dragging its header, subsequently all selected Actions will move too.

Action Info Field

Each Action has an additional Info Field you can use for custom notes.

Connecting Actions

Every Action has a number of inputs and outputs which are graphically represented through blue and green boxes. Blue outputs of an Action can be connected to blue inputs of another Action and define the step-by-step workflow processing. The connection rule applies to green boxes as well but they are used to transport data from one Action to the next. In addition to the blue boxes, green boxes can only be connected if the displayed datatype (marked with "<" and ">") is the same for input and output. For further information on this topic see Connections.

If you want to connect two Actions, click on an output of Action 1 and afterwards on an suitable input of Action 2 or vice versa. If the connection was succesful, a blue or green line will appear which connects both fields.

Actions

Actions are the basic elements and the main "building blocks" of every Workflow. Each Action defines a small application logic part which is written in C# code. For example the Action logic of the "Activate Camera" Action activates the camera when executed and the "Animated Translation" Action logic makes a geometry translate over time. Every Action logic can be configured through parameters which need to be specified according to the functionality of the Action. The parameters are adjusted through different GUI elements like number boxes or textfields.

The application logic of an Action isn't executed immediatelly after the Workflow has been started. First, the Action has to be activated by triggering one of its inputs. During activation the Action can subsequently execute its initialization logic. Sometimes, especially in case of simple Actions, the initialization logic executes all necessery operations and no further processing is required. Such Actions deactivate right after their activation. If the Action requires further processing and didn't deactivate itself after its activation, it can execute further logic in cycles. Such Actions are in the "running" state. Running Actions can be identified in the Authoring GUI through their red border. After all processing is done by the Action, it finally deactivates itself and is put into an idle state, waiting for reactivation,

In summary, Actions make use of the following states:

1. Activated:

How does an Action receive this state?: Action gets triggered through one of its input fields and isn't in the Running state.
What does an Action do in this state?: executes initialization logic.

2. Running:

How does an Action receive this state?: Action continues after being activated and did not deactivate itself already
What does an Action do in this state?: Execution of logic in cycles.
Hint: All Actions being in this state are marked with a red border in the Authoring GUI.

3. Deactivated:

How does an Action receive this state?: Action deactivates itself after the activation process or at some point in time during the running state.
What does an Action do in this state?: Execution of finalization logic.
Hint: Action waits for getting activated again.

The Workflow Engine ships with several build-in Actions. However, the available Action pool is variable and can be extended:

1. Record new Actions using the Unifeye GUI tool Action Recorder
2. Manually written Actions (for developers).

Connections

Connecting Actions enables the possibiliy to create more complex applications scenarios. Simple Action logics can be linked together and used to form powerful Workflows.

Every Action has a set of inputs and outputs that you can connect. They are graphically represented through blue and green boxes. The inputs are listed on the left side and the outputs are listed on the right side of every Action. Inputs as well as outputs are called "fields". The figure above shows an Action with three fields: one input and two outputs.

General connection rule

Inputs can get connected to outputs and vice versa as long as the field type matches. This means that only an input/output pair of state fields (the blue ones) or else wise data fields (the green ones) can get connected. In case of data fields, also the data type (like integer, double, string…) has to match.

State field connections

Blue line connections indicate a route, meaning a Workflow that is processed step by step, always in direction from output to input, after the “Run Script” button was pressed. You could say that state field connections are used jump from Action to Action so that they can execute their application logic. An action starts to execute its logic after it was activated through one of its state inputs. After the Action is done, it deactivates itself through one of its state outputs. This will trigger all connections which are linked to it. The connections will respond and finally trigger the inputs of other Actions. This process will repeat as long as the Workflow Engine is running the Workflow. In order to get this process started, there is the special "Start Action". In opposite to all other Actions, this Action will get activated immediatelly after the Workflow bas been started.

State field connection settings and rules

Any state field connection between two actions can be configured via its "Connection panel". The following rules and settings apply for each "Connection panel":

1. If the connection panel settings are left blank, the connection will continue automatically.
2. If you enter a time value (in milliseconds) into the spin box, the connection will delay the automatic continue by the entered duration.
3. If you enter a mouse / keyboard input into the textbox, the connection will only continue after you press that key.
4. If you enter a time value and a mouse / keyboard input, the connection will first wait for a key press and then automatically continue after the entered duration.

If you connect one state output to multiple state inputs, the simple connection rules modify a bit.

1. If you have setuped two or more connections with keyboard / mouse input, only the connection whose key which pressed first, continues.
2. If you have setuped one connection with delay and another with keyboard / mouse input, the connection with the delay will trigger automaticaly if the keyboard / mouse input connection didn't continue before.
3. See "parallel workflows"

Parallel workflows

If you connect one state output to multiple state inputs and all connections have been setuped with the same delay or left blank, the connected workflows will start to run parallel.

The parallel execution is done in the order, the connections were established. Parallel running Actions are useful for example if you want to translate and rotate a geometry at the same time while the translate and rotate motion is executed by two different Actions. After a parallel execution is done, it is recommended to fuse the workflow execution again. This is done by a special Action called "Synchronize Action". Just connect the outputs of all branches you want to fuse with a seperate input of the Synchronize Action. During the workflow execution, the Synchronize Action will start to trigger its single output state field only when all of its inputs were triggered before.

For an example on "parallel workflows" see the following image:

This subworkflow will rotate and translate the geometry called "projSphere" over time. The rotation Action lasts 1 second; the translation Action lasts 5 seconds. By the help of a Synchronize Action, we can make sure that the Workflow will run parallalel again after 5 seconds.

Data field connections

Green line connections indicate data connections and are used to pass data from one Action to another. This way Actions can receive variable data input during runtime. Everytime the value of a data output is changed, all connected data inputs are set to the value of the data output. The manipulation of data output fields is up to the Action which maintains the outputs.

Data field connection settings

A data field connection cannot be setuped any further. The connection passes the data directly from data output to data input.

User Scripted Actions (Ufa-Actions)

The Authoring GUI gives you the possibility to extend the action pool through user scripted actions, so called Ufa-Actions. Each Ufa-Action is based on a readable XML-format. Look into the subfolder "WorkflowAuthoringGUI/actions" of your Unifeye installation for several examples. Each .ufa is readable and can be opened in a text editor.

You can load custom Actions into the Authoring GUI by adding the Directory of your custom Actions to the Action Paths.

Ufa-Action frame

Each Ufa-Action bases on the XML structure which is shown in the following. Additionally you will see commented section numbers () which indicate that there is additional information available for this tag. You can find the tag description in the table below.

<?xml version="1.0"?> <!-- 1.0 -->
<unifeyeaction version="2.0"> <!-- 2.0 -->
      <name>First Ufa Action</name> <!-- 2.1 -->
      <category>Special</category> <!-- 2.1.1 -->
      <uid>TheFirstUfaAction</uid> <!-- 2.1.2 -->
      <description lang="en">Test</description> <!-- 2.1.3 -->
      <description lang="de">TEST</description>
      <fields> <!-- 2.2 -->
            <activator name="startAction"> <!-- 2.2.1 -->
                  <description lang="en">Starts the action</description> <!-- 2.2.1.1 -->
                  <description lang="de">Startet die Action</description>
            </activator>
            <deactivator name="testActionCompleted"> <!-- 2.2.3 -->
                   <description lang="en">Deactivates the action</description>
                   <description lang="de">Deaktiviert die Action</description>
             </deactivator>
             <dataoutput name="sceneID"/> <!-- 2.2.4 -->
             <datainput name="woot" gui="NUMBERBOX" /> <!-- 2.2.5 -->
             <variable type="AgInteger" gui="NUMBERBOX" defaultvalue="1"/> <!-- 2.2.6 -->
      </fields>
            <code> <!-- 2.3 -->

                  <activation> <!-- 2.3.1 -->
                        <![CDATA[
                              //code here
                        !>
                  </activation>
                  <loop> <!-- 2.3.2 -->
                        <![CDATA[
                              //code here
                        !>
                  </loop>
                  <deactivation> <!-- 2.3.3 -->
                        <![CDATA[
                              //code here
                        !>
                  </deactivation>
                  <libraries> 
                  </libraries>
            </code>
</unifeyeaction>

<?xml version="1.0"?>

<unifeyeaction>

<name>

<category>

<uid>

<description>

<fields>

<activator>

<description>

<deactivator>

<dataoutput>

<datainput>

<variable>

<code>

<activation>

<loop>

<deactivation>

<libraries>

Implementation

Data fields and data types

There is different kind of data you can work with and pass form one action to another. One of these data types has to be used for each <datainput>, <dataoutput> or <variable> definition.

Note on "basic data": If you want to assign a value to data fields marked with "basic data", use the following syntax in your code segment:

<variablename>.Value

Where <variablename> represents the "name" attribute of the <datainput>, <dataouput> or <variable> field.

Data fields and GUI elements

There are different kinds of GUI elements you can bind to data fields. This can be done via the "gui" attribute of data fields. In following you can see a list that shows which data fields can be attached to what type of GUI elements.

Code

With the implementation hints in mind you can start coding your own Actions using C#. The system will dynamically compile this code at runtime and execute it.

HINT1: every word between the percent signs "%%", references a variable, you defined in <activator>, <deactivator>, <dataoutput>, <datainput> or <variable> as tag-attribute "name".
HINT2: built in variables you can use in your code are marked bolted
HINT3: built in methods of variables are marked Italian and bolted

All code segments

Within every code segment (activation, running and deactivation) you can call commands of the Unifeye core compoenent (the Unifeye ActiveX) interface by writing

unifeye.methodName();

where "methodName()" is one of the Unifeye SDK API functions. Please refer to the Unifeye SDK API function reference for more details. Please be aware that when you use a Unifeye API function the function name will be slightly different as mentionend within the API documentation as there is an intermediate layer between the Workflow Engine and the Unifeye core. That's why the function names will be prefixed by "cmd". There are also a couple of functions that have a postfix "noUndo":

void cmdcreateCADGroupNoUndo(string groupName);
void cmddeleteCADGroupNoUndo(string groupName);
void cmdaddNodeToCADGroupNoUndo(string groupName, int sceneID, int nodeID);
void cmdremoveNodeFromCADGroupNoUndo(string groupName, int sceneID, int nodeID);
void cmdloadCADNodeNoUndo(int sceneID, int nodeID);
void cmdunloadCADNodeNoUndo(int sceneID, int nodeID);
void cmdsetCADNodeOcclusionNoUndo(int sceneID, int nodeID, int occluding);
void cmdsetCADNodeRotationNoUndo(int sceneID, int nodeID, float x, float y, float z, float rad);
void cmdsetCADNodeScaleNoUndo(int sceneID, int nodeID, float x, float y, float z);
void cmdsetCADNodeTranslationNoUndo(int sceneID, int nodeID, float x, float y, float z);
void cmdsetCADNodeVisibleNoUndo(int sceneID, int nodeID, int visible);
void cmdsetCADNodeWireframeNoUndo(int sceneID, int nodeID, int isWireframe);
void cmdsetClippingPlaneDirectionNoUndo(int planeID, string direction);
void cmdsetClippingPlaneCosIdNoUndo(int planeID, int cosID);
void cmdsetWireframeNoUndo(int sceneID, bool wireframe);
void cmdsetVisibleNoUndo(int sceneID, bool visible);
void cmdsetDragableNoUndo(int sceneID, bool dragable);
int cmdloadGeometryNoUndo(string geoUrl);
int cmdloadDragableGeometryNoUndo(string geoUrl);
void cmdunloadGeometryNoUndo(int sceneID);
void cmdsetShadowModeNoUndo(int sceneID, int shadowMode);
void cmdsetDragTranslationNoUndo(int sceneID, float x, float y, float z);
void cmdsetMoveTranslationNoUndo(int sceneID, float x, float y, float z);
void cmdsetMoveScaleNoUndo(int sceneID, float x, float y, float z);
void cmdsetMoveRotationNoUndo(int sceneID, float x, float y, float z, float rad);
void cmdremoveClippingPlaneNoUndo(int cpID);
void cmdsetCOSClippingPlaneNoUndo(int planeID, int cosID, string direction);
void cmdsetGlobalClippingPlaneNoUndo(int planeID, string direction);
void cmdsetClippingPlaneRotationNoUndo(int sceneID, float x, float y, float z, float rad);
void cmdsetClippingPlaneDragTranslationNoUndo(int sceneID, float x, float y, float z);
void cmdsetClippingPlaneTranslationNoUndo(int sceneID, float x, float y, float z);
void cmdsetCameraTimestampAdjustmentNoUndo(double timestampAdjustment);

Example:

If you want to use the Unifeye SDK API function for activating a camera (http://doxygen.metaio.com/UnifeyeSDK/UnifeyeSDK_doxygen/group__SourceCapture.html#gae2430406215fc1272c4ee88881428fd0) the Unifeye SDK API function name is: activateCamera(int index) which translates into cmdactivateCamera(int index) when used inside a custom action.

<activation> code segment

The code segment that is defined within the <activation> tag is called first. If you want to know which state input activated this action, use a code snippet similar to this:

If( %yourStateInput_1% == input ) 
{
      // state field 1 triggered this action
} 
else if( %yourStateInput_2% == input)
{
      // state field 2 triggered this action
}

If your action needs no running state and the activation part is sufficient (like in Workflow Authoring GUI 1.0), call

%yourStateOutput%.trigger();

at the end of your code segment. This will activate all Actions connected to the output %yourStateOutput% and deactivate this Ufa-Action.

<loop> code segment

The code segment that is defined within the <loop>-tag is called only when .trigger() wasn’t called during activation code.
In the <loop> code segment you have different possibilities and variables available for use.

• tickDelta: (double) this variable gives you the time difference beginning from the last time the loop method was called.
• tickDeltaFromStart: (double) sum of all tickDelta’s so far beginning from the activation of the Action.
• keyPresses: (List<KeyPair>) A list of all keys, the user has pressed. To use key presses in your action, read the following:
  • The number of all keys that have been pressed can be get by calling
    keyPresses.Count
  • To read a key (A,B,1,2,3…) of the list use:
    keyPresses[i].key
    where i represents the index of the key.
  • To read the type of key (Mouse, Keyboard or Wii) of the list use:
    keyPresses[i].keyType
    where i represents the index of the key type.
  • To know if the key was pressed down or up use:
    keyPresses[i].state
    in order to check if key was up or down use
    

    if(keyPresses[i].state == ButtonState2.DOWN)

    or
    

    if(keyPresses[i].state == ButtonState2.UP)

    if you handled a keyinput its necessary you call:
    keyPresses.Remove(i)

<libraries> code segment

You can use third party libraries in your Actions when you include them using the <library> tag. Each external library has to be nested within in the <libraries> tag as <library> . The <name> tag defines the name of a single DLL which should be included. Additional numerous <namespace> tags which are nested in <namespaces>, reference the imports. For example

<libraries>
    <library>
        <name>mydll.dll</name>
        <namespaces>
            <namespace>myNameSpace.MyClass</namespace>
            <!-- ... more namespace imports here -->
        </namespaces>
        
     </library>
     <!-- ... more libraries here -->
</libraries>

includes the library "mydll.dll" and imports the class "MyClass".

In order to use the library and action do not forget to copy your DLL (and potential dependencies) to the folder of your Action which is using the DLL. If the execution of the workflow fails you might also have to put the DLLs into the installation directory of the Workflow Authoring GUI (e.g. C:\Program Files (x86)\metaio\Unifeye Design\WorkflowAuthoringGUI\).