Example 4: Using a custom renderer - OpenGL ES 2.0

If you like to use your own rendering system the architecture has to be set up a litte different as in the other examples.

This example is related to the general OpenGL-API example of the Android plattform.

Contents

Requirements

• A running development enviroment as described in the Getting-Started-Guide.
• A device running Android 2.2 or newer. You cannot use the emulator as the camera feed is required.
• Recommended: A print-out of the MetaioMan tracking pattern. You can also display the image on your screen and track that.

Please note that this example targets users who are already familiar with augmented reality, programming and OpenGL in general.

Package Overview

Classes Overview

In the package com.metaio.MobileSDKExample.opengl20 you will find serveral classes. Their functionallity will be explained in this chapter. Please also refer to the UML class diagram below.

• GLES20CameraImageRenderer: The purpose of this class is to render the image of the camera on a plane and to tell the attached geometries to render in the current context. In this case it is only a cube.
• Cube: Class that draws a cube using VertexBuffers and shaders.
• ARController: The ARController provides an easy to use interface for programming. It encapsulates most of the relevant features of the MobileSDK. If a pose is detected, a method on the renderer is called. You can use this class together with any View or Application you like to.
• GLES20Utils: Convenience methods like loading a shader programm can be found here.
• GLES20Activity: This class is a simple Android Activity. It sets up the ARController as well as the Renderer.

In the examples you might also stumble upon some helper classes, which are in other packages of the example application. These are:

• Configuration: Camera configuration and the signature of you application can be configured in this class.
• AssetController: Copies assets from the application folder onto the SD-card of the device. It also provides the name of the files.
• Logger: Provides some methods for easy logging of messages and exceptions.

How does this work?

The GLES20Activity has two tasks: Initializing the MobileSDK and the renderer and providing a valid context both for a camera and a GLSurfaceView.

The main communication takes place between ARController and GLES20CameraImageRenderer.

Step 1: Get the MobileSDK running

The GLES20Activity class creates an instance of the ARController which starts everything needed for the MobileSDK. This is done inside the initializeMobileSDK() method:

    /**
     * Creates a new instance of the MobileSDK.
     */
    private void initializeMobileSDK() {
        /*
         * Create a new ARController with the application context and without
         * using the MobileSDK renderer.
         */
        mARController = new ARController(this);
        /*
         * The AssetController is necessary to access the assets in the asses
         * folder of the project.
         * 
         * Instead of creating an AssetController in the Application you can
         * also use mARController.createAssetController() to create one here. It
         * will be assigned to the ARController automatically.
         */
        mARController.setAssetController(mApplication.mAssetExtractor);
        /* 
         * Create an instance of the MobileSDK The required signature is stored
         * in the Configuration class. If you change the package name the
         * signatures as to be changed as well.
         */
        mARController.initializeMobileSDK();
        /*
         * Load the tracking data
         */
        mARController.setTrackingData(trackingDataFileInAssets);
    }

This code starts the MobileSDK. The private member trackingDataFileInAssets holds the name of the tracking data file you like to use. By default the tracking configuration for the MetaioMan will be loaded.

Step 2: Create a custom renderer

In fact, what is coming next is up to you since you like to use your own code now. For this example we create a very simple renderer, which can be extended or reused. It should only do two things:

1. Display the image of the camera and
2. display an AR object which is aligned to the tracking pattern in the camera image.

Therefore we start by creating a new GLSurfaceView. A GLSurfaceView is an Android-specific class which provides an OpenGL-rendering-context for us from the system.

This is done in the initOpenGLRenderer() method in the GLES20Activity.

    private void initOpenGLRenderer() {
        /*
         * Get a GL-context from the system.
         */
        mGLSurfaceView = new GLSurfaceView(this);
        if (detectOpenGLES20()) {
            /*
             * Tell the surface view we want to create an OpenGL ES
             * 2.0-compatible context, and set an OpenGL ES 2.0-compatible
             * renderer.
             */
            mGLSurfaceView.setEGLContextClientVersion(2);
            mRenderer = new GLES20CameraImageRenderer(this);</strong>
        } else {
            throw new RuntimeException("OpenGL ES 2.0 must be supported!");
        }
        /*
         * Link the different elements of this Activity
         */
        mRenderer.setARController(mARController);
        mARController.setRenderer(mRenderer.getARControllerCallback());
        mGLSurfaceView.setRenderer(mRenderer);
        /*
         * Attach the view to the content view, so that it is visible for the
         * user.
         */
        DisplayMetrics displayMetrics = new DisplayMetrics();
        getWindowManager().getDefaultDisplay().getMetrics(displayMetrics);
        addContentView(mGLSurfaceView, new FrameLayout.LayoutParams(
                displayMetrics.widthPixels, displayMetrics.heightPixels));

    }

This method also creates an instance of our actual rendering class. It is the GLES20CameraImageRenderer.

Step 3: Receive events from the MobileSDK

The rendering class GLES20CameraImageRenderer must be provided with a camera image, a projection matrix and tracking values. So the next step is to get these values from the ARController.

Events that can be triggerd by the ARController are:

• Tracking related events:
  • onPatternDetected: Triggered if a pattern was invisible and gets visible again.
  • onTrackingLost: Triggered if a pattern was visible and can't be found (i.e. tracked) anymore.
  • onTrackingSuccessfull: Triggered if there is a valid pose for the target coordinate system.
• Camera related events:
  • setCameraTextureSource: Provides the camera image as byte-array when a new image is available.
  • setGeometryProjectionMatrix: Provides a projection matrix that includes the camera intrinsics and a correct perspective setup, so that the pose-matrix corresponds to the camera image. The matrix can be directly used in OpenGL . It is triggered after onSurfaceChange() was invoked on the ARController and upon start-up.

The ARController class provides an nested interface, called ICallback, that can be implemented by the class that should receive the events from the ARController. To have a better overview of the methods, we recommend using a nested class for doing this job. The nested class is inside of the class that actually shall reveicve the events. In this example, the renderer ( GLES20CameraImageRenderer) has a subclass called ARCallback. The implementation looks like this:

protected class ARCallback implements ARController.ICallback {
        /**
         * Triggered if the pattern can't be detected anymore. 
         */
        @Override
        public void onTrackingLost(int cosId) {
            mCube.setVisibile(false);
        }

        /**
         * Triggered if the pattern has been detected.
         */
        @Override
        public void onPatternDetected(int cosId) {
            mCube.setVisibile(true);
        }

        /**
         * Sets the pose of the attached geometry. This method gets called by the
         * ARController, when there is a valid tracking Pose.
         * 
         * If you insert own geometry, make sure that it gets this matrix at this
         * point.
         */
        @Override
        public void onTrackingSuccessfull(int cosId, float[] matrix) {
            mCube.setMMatrix(matrix);
        }

        /**
         * Updates the texture used for the camera-image Implements IRenderer.
         * 
         * @param bitmap
         *            Array of bytes that contains the image.
         * @param width
         *            Width in pixels of the image. Must correspond to the
         *            byte-data.
         * @param height
         *            Height in pixels of the image. Must correspond to the
         *            byte-data.
         */
        @Override
        public void setCameraTextureSource(final byte[] bitmap,
                final int width, final int height) {
            ByteBuffer buffer = ByteBuffer.wrap(bitmap);
            try {
                GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
                GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureID);

                GLES20.glTexSubImage2D(GLES20.GL_TEXTURE_2D, // textureId
                        0, // level
                        0, // offset x
                        0, // offset y
                        width, // width of buffer
                        height, // height of buffer
                        GLES20.GL_RGBA, // format
                        GLES20.GL_UNSIGNED_BYTE, // type
                        buffer); // data
                GLES20Utils.checkGlError("update texture");
            } catch (Exception e) {
                Logger.logException(e);
            }
        }

        /**
         * Sets the projection matrix for the attached geometry.
         * In this case it is only our cube. 
         */
        @Override
        public void setGeometryProjectionMatrix(float[] projectionMatrix) {
            mCube.mProjMatrix = projectionMatrix;
        }
    }

Note that the member mCube does not belong to the class itself but to the GLES20CameraImageRenderer class, in which this class is nested in.

Rendering the camera image

The camera plane is rendered by the GLES20CameraImageRenderer class by

1. creating a callback-object inside the ARController that will receive events from the MobileSDK, when there is a new frame avaiable,
2. registering this callback-object,
3. creating a texture with the dimensions of the camera image,
4. creating an orthographic projection matrix and a proper view matrix,
5. creating a VertexBuffer with four vertices (XYZ) and texture coordinates (UV),
6. creating simple vertex and fragment shaders for drawing textured polygons,
7. activating the shaders,
8. linking the shaders with the matrices and the vertex array and
9. pushing the vertices into the OpenGL pipeline.

To update the camera image the renderer must be connected to the ARController.

If everything is setup properly, the callback-object (see ARController.MobileSDKCallback) will receive new frames from the MobileSDK. The SDK will call onNewCameraFrame on that object. The frame gets converted to a ByteArray and passed to the renderer using setCameraTextureSource().

The ByteArray gets then converted into a ByteBuffer and gets passed to the texture memory of the device by using GLES20.glTexSubImage2D like this:

       /**
         * Updates the texture used for the camera-image Implements IRenderer.
         * 
         * @param bitmap
         *            Array of bytes that contains the image.
         * @param width
         *            Width in pixels of the image. Must correspond to the
         *            byte-data.
         * @param height
         *            Height in pixels of the image. Must correspond to the
         *            byte-data.
         */
        @Override
        public void setCameraTextureSource(final byte[] bitmap,
                final int width, final int height) {
            ByteBuffer buffer = ByteBuffer.wrap(bitmap);
            try {
                GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
                GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureID);
                GLES20.glTexSubImage2D(GLES20.GL_TEXTURE_2D, // textureId
                        0, // level
                        0, // offset x
                        0, // offset y
                        width, // width of buffer
                        height, // height of buffer
                        GLES20.GL_RGBA, // format
                        GLES20.GL_UNSIGNED_BYTE, // type
                        buffer); // data
                GLES20Utils.checkGlError("update texture");
            } catch (Exception e) {
                Logger.logException(e);
            }
        }

Troubleshooting

There are several reasons why the application could fail or crash:

• The signature used for the MobileSDK is invalid. This normally happens after changing the package name. In that case, just generate a new signature as documented in the FAQs.
• The tracking data is invalid. Please try to use a tracking data provided in the bundle to check if that's the reason. If it depends on that, please make sure that the file name is spelled correctly. If that is correct too, your tracking data might be corrupted or malformed. Please check our section on creating tracking data.
• Your device does not support OpenGL ES 2.0. For this example this is however mandatory.
• If you can't solve your problem here, you might find the Support Forums helpful.

-- SupportMetaio - 2011-11-29