Monthly Archives: February 2015

Sensor Fusion: Rolling your own

Last time, I wrote about sensors and sensor fusion. Over the last couple of years at Linknode, we’ve gained considerable practical experience with the sensors that are built into modern tablets. This blog post is a fairly technical explanation of some of that knowledge and understanding, and why we ended up doing what we did – if you don’t like maths, I suggest you skip this blog post.

I mentioned last time that most sensor fusion algorithms have been written with gaming in mind. Whatever anyone says, games are the things that push the boundaries of the hardware on mobile devices. There are two needs for speed on mobile devices these days – animating the transitions within app and providing detailed 3D experiences within games. Mobile devices are not (yet) used for number crunching or other processor intensive operations.


We have seen some specific problems with manufactures implementation of sensor fusion. A lot of the algorithms are about rate of change and responsiveness over absolute accuracy. With VentusAR, we require accuracy foremost as the visualisations we produce could end up under expert scrutiny. One of our early clients said that they would accept a tolerance of +/- 2° from the compass (and much less in the other sensors). We provide calibration tools to allow sensors to get within a tolerance of 0.1°. We found two main problems with default sensor fusion algorithms:

  • Jitter – the compass / fusion on some android tablets would jitter unacceptably. The terrain model would jump by +/- 10° while the device was sitting on the table.
  • Inaccuracy – the wireline could be a few degrees out when trying to align it to real world terrain. This could sometimes be corrected by rotating the device in 3D then pointing at the view again

Rolling your own

Rolling your own sensor fusion isn’t too hard, we started with some basic requirements:

  • Smoothing – the sensor fusion algorithm should produce a smooth output. If the device is setting stationary on a table, the inputs should not “jitter”. For example, using My View, the terrain should be accurate and not jitter
  • Accurate – prioritise the input from the compass over all over input – we should be using the other sensors to smooth and enhance the compass, not using the compass to provide stability to the gyroscope
  • Fast – The sensor data is read from the device at either 50 times or 60 times a second (50 on android, 60 on iOS). This means the CPU has 20 to 25ms to process each package of sensor data to ensure that we are keeping up.

With these requirements in mind we set about writing our own sensor fusion algorithm. The pseudo code for the algorithm we ended up with looks little like:

  • Calculate a ‘smoothed heading’
    • Calculate change between current heading value and last heading value
    • If change > THRESHOLD
      • smoothedHeading = lastHeading +LARGE_OFFSET
    • Else
      • smoothedHeading = lastHeading + SMALL_OFFSET
  • Maximise accuracy of the compass component
    • Remove current heading component from the output of the manufactures sensor fusion
    • Multiply by smoothed heading from above
  • Normalize

When smoothing, if we have a big change in magnetic heading (ie if change is > THRESHOLD) the ‘Smoothed Heading’ will respond to that quickly, however small changes will be smoothed over by the small offset. The values for THRESHOLD, LARGE_OFFSET and SMALL_OFFSET are device specific and have been found by running appropriate testing on each device.


To give an example of how the smoothing function of our sensor fusion works, I did some work to export the raw data to excel and do some analysis on it. The graph below shows the input (raw heading) plotted next two the output of the sensor fusion. This graph shows the input to the sensor fusion classes in blue and the output in orange. The x axis shows the number of data points (were receiving approximately 50 per second so this shows 2000 data points over about 40 seconds). The y axis shows the heading of the device is shown on the left (values between 0° and 360°).

40 of smoothing data from Linknode Sensor fusion implementation

40 seconds of smoothing data from Linknode Sensor fusion implementation

This shows our smoothing functions working correctly:

  • The peak and troughs on the graph are less extreme
  • Curves are smoother so there is less jitter
  • The peak is delayed by approximately 15 samples. This equates to approximately 0.25s which we have decided to be acceptable performance.

Below are two crops of the My View function of VentusAR. The left hand (red wireline) shows the jitter as seen in v2.1. While the right video (black wireline) shows much less jitter in v2.2

Our custom sensor fusion has been a considerable bit of work at Linknode which we hope is useful for other people who want to understand the way the sensors work on these types of devices.

Your Phone has Attitude!

The axis on a mobile device

The axis on a mobile device

Sorry, this post isn’t about your phone or tablets bad attitude and the way it doesn’t let you do what you want – that’s just working with Android that does that. Instead, this post is about how we at Linknode use the sensors built into your device to understand the direction it is orientated to and how that can be used to do interesting things.

This is a core piece of technology we use within VentusAR. We have spent a lot of time and effort interfacing with the sensors within your devices. This experience and skill goes into several of our mobile apps to provide a more intuitive and useful mobile experience.

In this post, we’ll talk about attitude (or geospatial orientation), sensors and sensor fusion, then show some example code of how to get this attitude information on each of the major platforms. I’ll write a follow up post that will dig more deeply into what sensor fusion is and how we have customised it in VentusAR, to provide a better user experience in our augmented reality applications.


To allow the device to present useful information about its surroundings, we need to know the direction the device is looking. This provides key information that you must know to be able to do any proper augmented reality. The direction your device is looking  is called ‘the attitude’ (or geographic orientation) of the device. In essence, this is a value that represents the rotation of the device in real world coordinates.  In mathematics, this rotation value can be represented in a number of ways: a quaternion, a rotation matrix or as three separate values for yaw, pitch and roll. We use a quaternion to represent this rotation because this is smaller, involves simpler maths to work with and avoids known problems with rotation matrices – I’ll cover that in a separate blog post some time.


Modern phones and tablets have lots of sensors in them – they allow app developers get an insight into the world around them. In terms of attitude, the ones we are interested in this post are:

  • Compass – gives the direction of magnetic north in 3D space
  • Gyroscope – this measures angular rotation – how far you have rotated the device
  • Accelerometer – measure the direction of gravity in 3D space

There are a couple of limitations of these sensors that are worth knowing about:

  • Digital compasses are very noisy and susceptible to interference so often they jump during real world use. This is down to the characteristics of the sensor – as an app developer, there is not much you can do about it.
  • Gyroscopes tend to drift. There is no real world reference for the gyroscope, it is just measuring rotation. If you did a complete 360°, you would expect the gyroscope to give the same result. Unfortunately it doesn’t, after a while of running it tends to drift.

For these reasons, some very clever people came up with the concept of sensor fusion.

 Sensor Fusion

These sensors can be merged through software into a single “virtual” sensor using a process called Sensor Fusion. Many people have written in-depth articles about what Sensor Fusion is and how it works – but you may need a PhD to understand them. I think it is easiest to see it as a mathematical process that takes input from the three physical sensors (Compass, Gyro and accelerometer) and provides one unified quaternion representing the attitude of the device.

Sensor Fusion block diagram

Sensor Fusion block diagram

To provide a more detailed example, if you were standing in the northern hemisphere with the device perpendicular to the ground facing the north pole (i.e. level on a tripod, facing a heading of 0 degrees), the devices attitude would be:

0 degrees 45 degrees 90 degrees 180 degrees
x  0  0  0  0
y  -1  -0.9238795  – 0.7071068  0
z  0  0  0  0
w  0  0.3826834   0.7071068  -1

How does it help

As I said at the start, the integration to the sensors is at the core of what we do at Linknode. We have several apps that read data from the sensors and provide a real time view across a 3D world. We can pull in a real world terrain model and show what the terrain looks like in a particular direction.


Each device manufacture / OS vendor provides their own implementation of sensor fusion within their devices. These are usually good enough for general or gaming purpose – they tend to have an emphasis on speed of response instead of absolute accuracy. Below I have shown some code that allows you to get a quaternion out of the API provided by the OS.

All code below is c# as all the code we write is c#. For more information on running c# on iOS or Android have a look at what Xamarin are up to.

Apple (iOS)

Apple provide the CMMotionManager classes that can be used on iOS.

public class IOSSensorFusionExample
  public void Start()
    CMMotionManager _motionManager = new CMMotionManager();
    _motionManager.DeviceMotionUpdateInterval = 1/60; //request 60 updates a second
      delegate (CMDeviceMotion motionData, NSError error)
        CMQuaternion cMQuatAttitude = (CMQuaternion)motionData.Attitude.Quaternion;
        //do something useful with the quaternion here

(See Xamarin API for mode details)


Android provides the RotationVector sensor type accessible from their SensorManager class:

public class AndroidSensorFusionExample : Java.Lang.Object, ISensorEventListener
  public void Start()
    SensorManager sensorManager = this.GetSystemService(Context.SensorService);
    var defaultRotationVectorSensor = sensorManager.GetDefaultSensor(SensorType.RotationVector);
    sensorManager.RegisterListener(this, defaultRotationVectorSensor, SensorDelay.Game);

  public void OnSensorChanged(SensorEvent e)
    float[] q = new float[4];
    SensorManager.GetQuaternionFromVector(q, e.Value.Values.ToArray());
    Quaternion quaternion = new Quaternion(q[1], q[2], q[3], q[0]);
    //do something useful with the quaternion here

(see Xamain Android API and Android Docs for more information)

Windows Phone

Windows Phone provides the motion classes:

Motion sensor = new Microsoft.Devices.Sensors.Motion();
sensor.CurrentValueChanged += (sender, args) =>
    var quaternion = args.SensorReading.Attitude.Quaternion;
    //do something useful with the quaternion here

(see MSDN for more details)

Windows 8

Windows 8 uses the motion class:

var sensor = Windows.Devices.Sensors.OrientationSensor.GetDefault();
sensor.ReadingChanged += (sender, args) =>
    var quaternion = args.Reading.Quaternion;
    //do something useful with the quaternion here
sensor.ReportInterval = 16;

(see MSDN for more details)

VentusAR 2.2 Release

We are pleased to announce the release of VentusAR v2.2 for Android, iPad (and associated portal updates).

Setup DSLR photography for use in VentusAR

Setup DSLR photography for use in VentusAR

For iPad and Android users this release includes many new features including:

  • Adds Earth Curvature and Atmospheric refraction calculations to the My View and Gallery Visualisations. This is required to produce SNH compliant output.
  • Support for external photography. Photography taken on an DSLR camera can be loaded into the portal and visualisations generated. This is required to produce SNH compliant output.
  • Reorganised gallery to associate photography with a viewpoint. This makes it quicker and easier to find the photography and generate the wirelines after use.

We recommend updating the new version of VentusAR to take advantage of the new features that have been introduced.

See the full release notes for Android, iPad and the portal.