Introduction

FusionHub is a software application that has the purpose of combining a number of sensor data inputs to create a higher level information output. LPVR-POS is based on FusionHub and combines odometry, GPS and IMU data from a vehicle to calculate high-accuracy and low-latency global localization information. Typical applications: Automobile localization, robot localization

The diagram below shows the general structure of FusionHub. Sources and sinks are connected by a filter unit. The sensor fusion functionality is contained in this filter unit. The filter parameters as well as the parameters of input and output blocks can be configured via a configuration script or the graphical user interface.

The graphical user interface is detached from the main FusionHub application and both applications can therefore run on separate computers. This provides flexibility for running FusionHub on devices with limited monitoring capabilities like a head mounted display.

General

Starting FusionHub

FusionHub consists of two components:

• The main application

• A graphical user interface application

Insert the security dongle into a USB port of your computer.

The main FusionHub application is started by running FusionHub.exe. No specific installation is needed, the application can be run directly out of its deployment directory. It is a command line application that uses the file config.json for its configuration. We will explan the contents and options of the configuration file further below.

Please install the graphical user interface by running lp-fusionhub-dashboard_0.1.0_x64_en-US.msi. It installs lp-fusionhub-dashboard in your start menu, launch the application from there. Press the Connect button after starting FusionHub.exe to connect client and server. In case you are running FusionHub on a separate machine make sure to enter the correct IP address.

The screenshot below shows the connection elements of the GUI.

Licensing

FusionHub has two options for license protection:

Hardware dongle

License authentication using a hardware dongle; This is especially interesting for air-gapped installations that are not connected to the internet. As long as the dongle is inserted into a USB slot of the host system, FusionHub will run. Please note that for the Android (Quest 2 HMD) version of FusionHub, the GUI running on the streaming host is dongle protected, see more detailed information in the specific manual chapter.

Online license

License authentication using a software, online license; This makes sense for systems that are connected to the internet at least during the initial installation of FusionHub. The software checks its license status with our license server with following sequence:

1. Enter license key in configuration file. You receive your personal license key from us.

2. Send license key and machine code to server

3. Server checks if license is valid and returns response code, if it is valid

4. Copy the response code from the log and enter it in the config file to the ResponseKey parameter. Save the config file.

5. This allows FusionHub to run on this specific machine without reconnecting to the internet. One license unit will be subtracted from your license account. Please ask us for assitance if you’d like to move your license.

If your default configuration file config.json doesn't contain it already, add the LicenseInfo block as shown below. Enter your personal key you received from us as LicenseKey.

{
  ...

  "LicenseInfo": {
    "LicenseKey": "EKKCO-GZYLT-NJKET-SASDC",
    "ResponseKey": ""
  }
  
  ...
}

LPVR-POS Filter Configuration

LPVR-POS combines odometry, GPS and IMU data from a vehicle to calculate high-accuracy and low-latency global localization information. While GPS or RTK-GPS measurements alone provide similar positioning accuracy the output frequency of these systems is relatively low, making them unsuitable for applications where localization information at higher framerates is required, such as positioning objects in an augmented reality environment.

By additionally using odometry (wheel speeds, steering angle etc.) information, the localization data from the GPS measurements is interpolated to achieve framerates limited only by IMU and odometry sampling speeds.

The LPVR-POS filter has two operation modes with different configuration blocks in config.json and different output formats. The two modes are:

• Low-dynamics filter (LD)

• High-dynamics filter (HD)

The diagram below shows an overview of a simple LPVR-POS filter setup.

Installation of Hardware Components

Inertial Measurement Unit (IMU)

LPMS-IG1P needs to be installed in the vehicle in a known orientation ideally with the coordinate axes of the IMU arranged in parallel to the vehicle coordinate system. As vehicle reference frame we are using the VW coordinate system shown in the image below. Connect the USB connector of LPMS-IG1P to the host computer. If needed an active or passive USB extension can be used. Make sure to check data integrity with the LpmsControl 2 data acquisition tool, we have noticed communication issues with some passive USB extensions.

VW frame
x: back
y: right
z: up

Global Positioning System (GPS)

The GPS receiver is integrated with the LPMS-IG1P sensor. Connect the antenna cable and place the GPS antenna on top of the vehicle. If RTK-GPS is reqquired a standalone RTK GPS module can be used as a GPS input source as well.

CAN Bus Connection

FusionHub can be connected to the vehicle CAN bus by using one of the following CAN bus interfaces:

• PCAN USB interface

• Vector CAN USB interface

LPVR-POS Sensor Fusion Filter

Filter Inputs

Both the LD and the HD filter need the following sources as input. One option is to operate the filter with our LPMS-IG1P sensor that contains IMU and a GPS receiver. This allows for standard GPS absolute position accuracy. Relative accuracy and update rate are higher, based on odometry and IMU data. The other option is to use a separate RTK-GPS unit to get high accuracy RTK-GPS readings. We will look at how to set up an RTK-GPS system for LPVR-POS in a following chapter.

Option 1 - LPMS-IG1P data source for IMU and GPS data

LPMS-IG1P Source

"imuP": {
    "type": "DualRtk",
    "settings": {
        "sensor1": {
            // If specification needed, insert first IG1 sensor name here
            //"name": "ig1p232800650050",
            "autodetectType": "ig1p"
        },
        "rtcm": true,
        "imuEndpoint": "tcp://*:8802"
    }
}

Option 2 - Separate LPMS-IG1 IMU and RTK GPS sources (with NTRIP caster for RTK correction)

LPMS-IG1 Source

"imu": {
  "type": "OpenZen",
  "settings": {
    "autodetectType": "ig1"
  }
}

RTCM Source

"RTCM": {
    "type": "NTRIP",
    "settings": {
        "host": "some-host-name",
        "port": "2101",
        "mountpoint": "some-mount-point",
        "user": "some-user",
        "password": "some-password",
        "userAgent": "LPVR",
        "initialLatitude": 35.65736,
        "initialLongitude": 139.73239,
        "forwardGnss":  true
    }
}

GNSS Source

"gnss": {
    "type": "NMEA",
    "settings": {
        "port": "/dev/ttyUSB0",
        "baudrate": 115200,
        "rtcm": true
    }
}

CAN bus and vehicle decoder source

"vehicle": {
    "type": "Automotive",
    "vehicleStateEndpoint": "tcp://*:8999",
    "settings": {
        "canInterface": "PeakCAN",
        "vehicleType": "R56"
    }
}

Low-dynamics Filter

General

The low dynamics (LD) filter combines odometry, GPS and IMU data to calculate the global position of a vehicle. This filter was initially concipated to work only with odometry and GPS data to calculate the 2D position and yaw angle of a vehicle. It is therefore suitable for simple tracking scenarios. For more complicated, for example augmented reality, applications we recommed using the HD filter as it outputs globally referenced 3D orientation additionally to globally referenced position. This filter relies heavily on the wheel velocity data of the car published on the car’s CAN bus. Therefore the quality of the output of this filter depends on the accuracy of this information.

Notes on Data Output

The LD filter outputs fusedVehiclePose. The FusedVehiclePose contains a 3D acceleration vector. The acceleration is defined in the following manner: There's a configuration flag imuToCarRotation which takes a quaternion used to rotate vectors in the IMU frame to the car frame. By default it is the identity quaternion. For the LD model, the measured IMU acceleration is simply rotated by the imuToCarRotation and written to the output.

In the LD filter, pitch and roll has to be derived from the acceleration data based on a model of the stiffness of the chassis. That assumes a flat surface. The HD model offers the full 6-DOF, and we are planning to unify them to have all data available at all times.

Notes on IMU Arrangement

LPMS-IG1P needs to be installed in the vehicle in a known orientation ideally with the coordinate axes of the IMU arranged in parallel to the vehicle coordinate system. The LD filter uses the imuTurnRateAxis parameter to determine which axis it should use to calculate the vehicle’s orientation. For example if the IMU is installed in the car so that the Z axis is pointing upwards, imuTurnRateAxis should be set to 0, 0, 1.

Configuration Block

Insert the following configuration block into your config.json file to activate the LD filter. The filter's node name is vehicularFusion.

"vehicularFusion": {
    "echoFusedPose": false,
    "endpoint": "tcp://*:8801",
    "fuser": {
        "fitModel": "SimpleCarModel",
        "driveModel": "Differential",
        "velError": 0.277777778,
        "omegaError": 0.5,
        "measurementError": 0.1,
        "smoothFit": true,
        "useImuTurnRate": true,
        "imuTurnRateAxis": {
            "x": 1,
            "y": 0,
            "z": 0
        },
        "imuToCarRotation": {
            "w": 1,
            "x": 0,
            "y": 0,
            "z": 0
        }
    }
}

See below a description of the parameter options of the LD filter.

Output Format

See a technical description of FusionHub’s communication interface in one of the following chapters.

JSON

{
    "fusedVehiclePose": {
        "acceleration": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "globalPosition": {
            "x": 0.0,
            "y": 0.0
        },
        "lastDataTime": {
            "timestamp": 0
        },
        "position": {
            "x": 0,
            "y": 0
        },
        "timestamp": {
            "timestamp": 0
        },
        "utmZone": "31T",
        "yaw": 0
      }
}

Protobuf

syntax = "proto3";

package Fusion.proto;

message Vector2 {
    double x = 2;
    double y = 3;
}

message Vector {
    double x = 2;
    double y = 3;
    double z = 4;
}

message FusedVehiclePose {
    int64 timestamp = 1;
    Vector2 position = 2;
    Vector2 global_position = 3;
    double yaw = 4;
    string utm_zone = 5;
    int64 timecode = 6; // Optional: if 0 not set.    
    Vector acceleration = 7;
}

message StreamData {
    int32 sequence_number = 1;
    FusedVehiclePose fused_vehicle_pose = 9;
}

High-Dynamics Filter

General

The high dynamics filter combines IMU and GPS data to calculate the global position of a vehicle. Instead of using the odometry it uses IMU data to determine the orientation changes of a car on the X, Y and Z axis. The direction of the vehicle is globally referenced from the GPS system. For increasing the direction reference quality a dual-antenna GPS system can be used.

The high dynamics filter works well for scenarios with agressive driving maneuvers such as drifting and cornering. During such maneuvers the turning motion of the wheels generally doesn’t directly correspond with the direction of the vehicle. Therefore for this filter don’t rely on wheel velocity measurements. This filter uses information from the wheels only to determine if the car has come to a full stop.

As the filter relies heavily on GPS measurements it doesn’t deliver good results indoors. The better GPS reception, the better the resulting output of the filter. In it’s current state it therefore only works outdoors. In a future version that combines LD and HD filters, this issue will be resolved.

Notes on Filter Output

The HD filter outputs the following data:

• fusedVehiclePose (2D pose): Output equivalent to the LD filter output. Includes position in meters relative to starting point, global position (lon, lat) and heading.

• fusedPose (3D pose): relative to starting point, x, y (in meters) + z (height) + 3D orientation quaternion

• globalFusedPose: globally referenced 3D position (longitude, latitude, height) + 3D orientation quaternion in ENU frame

Which filter output works best depends on your application. For augmented reality applications we reommend using globalFusedPose.

Notes on IMU Arrangement

The used car frame is the Volkswagen (VW) coordinate frame convention:

VW frame x: back y: right z: up

The IMU sensor can be mounted in any way but the ImuToCarRotation quaternion needs to be provided to transform the IMU data into VW frame. For example, if the IMU is mounted like follows:

IMU mounting
x: forward
y: left
z: up

To match the VW frame, we need a 180° rotation around the z axis (clockwise). Therefore, the rotation matrix would be:

[ -1,  0,  0;
   0, -1,  0;
   0,  0,  1 ] 

And the orientation quaternion woud be [x, y, z, w] = [ 0, 0, 1, 0 ] which can be specified in the configuration like below:

"imuToCarRotation": {
    "w": 0,
    "x": 0,
    "y": 0,
    "z": 1
}  

Check this page for more information on how to calculate the orientation quaternion.

Configuration Block

Insert the following configuration block into your config.json file to activate the HD filter. The filter's node name is gnssImuFusion.

"gnssImuFusion": {
    "echoFusedPose": false,
    "endpoint": "tcp://*:8803",
    "fuser": {
        "fitModel": "ModelGnssImu",
        "accelError": 0.01,
        "omegaError": 0.02,
        "measurementError": 0.05,
        "imuToCarRotation": {
            "w": 1,
            "x": 0,
            "y": 0,
            "z": 0
        }
    }
}

See below a description of the parameter options for the HD filter.

Output data format

FusedVehiclePose

JSON

{
    "fusedVehiclePose": {
        "acceleration": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "globalPosition": {
            "x": 0.0,
            "y": 0.0
        },
        "lastDataTime": {
            "timestamp": 0
        },
        "position": {
            "x": 0.0,
            "y": 0.0
        },
        "timestamp": {
            "timestamp": 0
        },
        "utmZone": "31T",
        "yaw": 0.0
    }
}

Protobuf

syntax = "proto3";

package Fusion.proto;

message Vector2 {
  double x = 2;
  double y = 3;
}

message Vector {
  double x = 2;
  double y = 3;
  double z = 4;
}

message FusedVehiclePose {
  int64 timestamp = 1;
  Vector2 position = 2;
  Vector2 global_position = 3;
  double yaw = 4;
  string utm_zone = 5;
  int64 timecode = 6; // Optional: if 0 not set.
  Vector acceleration = 7;
}

message StreamData {
  int32 sequence_number = 1;
  FusedVehiclePose fused_vehicle_pose = 9;
}

FusedPose

JSON

{
	"fusedPose": {
		"lastDataTime": {
			"timestamp": 0
		},
		"orientation": {
			"w": 1.0,
			"x": 0.0,
			"y": 0.0,
			"z": 0.0
		},
		"position": {
			"x": 0.0,
			"y": 0.0,
			"z": 0.0
		},
		"timestamp": {
			"timestamp": 0
		}
    }
}

Protobuf

syntax = "proto3";

package Fusion.proto;

message Quaternion {
  double w = 1;
  double x = 2;
  double y = 3;
  double z = 4;
}

message Vector {
  double x = 2;
  double y = 3;
  double z = 4;
}

message FusedPose {
  int64 timestamp = 1;
  Vector position = 2;
  Quaternion orientation = 3;
  Vector angular_velocity = 4;
  int64 timecode = 5; // Optional: if 0 not set.  
}

message StreamData {
  int32 sequence_number = 1;
  FusedPose fused_pose = 4;
}

GlobalFusedPose

{
    "globalFusedPose" {
        "orientation": {
			"w": 1.0,
			"x": 0.0,
			"y": 0.0,
			"z": 0.0
		},
        "timestamp": {
			"timestamp": 0
		},
        "position": {
            "longitude": 0.0,
            "latitude": 0.0,
            "height": 0.0
		}
	}

Information regarding the ENU coordinate system is here: https://en.wikipedia.org/wiki/Local_tangent_plane_coordinates

Data Playback [DEPRECATED, switch over to ReplayExecutable]

Data from a log file can played back and forwarded to a fusion filter using the fileReader block. An example of how to use this node we are showing below:

"sources": {
    "filereader": {
        "filename": "sampleDriveData.json",
        "playbackInterval": 0.001
    }
},

Replay Node [DEPRECATED, switch over to ReplayExecutable]

Replay data from disk file.

  "sources": {
    "replay": {
      "filename": "./MiniRide10.json",
      "replaySpeed": 1,          // replay speed is adjustable
      "readMultipleLines": 10    // disk reader would read in multiple lines every time.
    }
  }

Replay Executable

This is a separate executable that can be built from FusionHub project. In Visual Studio build target dropdown there will be an option to build ReplayExecutable.exe.

The replay executable will read in from file, push data to replay queue and send them to the network (tcp://localhost:9921 by default). To run the ReplayExecutable,

ReplayExecutable.exe -r <path/to/file.json> [--replay-speed 1] [--queue-size 100] [--echo-data] [--verbose]

A normal FusionHub program can then receive the file data by having an endpoints source defined in the configuration file:

{
  ...,
  "sources": {
    "endpoints": ["tcp://localhost:9921"]
  }
}

Graphical User Interface

Map View

Data Playback and Recording

Data playback and recording works in the same way for all FusionHub versions. It has been described in the previous chapters, but I’ll add a recap here to give it its dedicated chapter, as it’s a very important feature for data analysis and serialization.

Data Recording

Record node

You can record the output from FusionHub to a file by adding

"record": {
    "filename": "driveData.json",
    "format": "json"
}

to the sink section of config.json.

File Logger

"logger": {
    // "endpoints": ["inproc://file_reader_1"],  // from file reader
    "endpoints": ["tcp://localhost:9921"],  // from Replay executable
    "format": "json",
    "filename": "recTest" // this will become the postfix of the filename
}

Data Playback

Data from a log file can played back and forwarded to a fusion filter using the fileReader block. An example of how to use this node we are showing below:

"sources": {
    "filereader": {
        "filename": "driveData.json",
        "playbackInterval": 0.001
    }
},

Communication with External Applications

WebSocket APIs

Apart from manual editing the config.json configuration script or modifying it through the GUI, FusionHub also offers a WebSocket API for external application to change its configuratuion. In fact the GUI uses this interface to access FusionHub’s settings.

Note that the websocket communication is currently not encrypted, it is not secure. Please take your own precautions to make sure network traffic for the configuration isn’t intercepted in some way. We might add an option for secure communication in future releases.

The WebSocket server can be accessed via 19358 port on the machine hosting the FusionHub service. To accelerate development download the Simple WebSocket Client Chrome plugin. This allows you to manually enter API commands and check the replies from the server.

{
  "command": "getConfig"
}

{
  "command": "getSavedConfig"
}

{
  "command": "saveConfig"
}

{
  "command": "setConfig",
  "data": {
    "sources": {
      "optical": {
        "settings": {
          "port": 5005
        }
      }
    }
  }
}

{
  "command": "overwriteConfig",
  "data": {
    "settings": { ... },
    "sources": { ... },
    "sinks": { ... }
  }
}

{
  "command": "getIntercalibrationStatus"
}

{
  "command": "applyIntercalibrationResults"
}

{
  "command": "restartBackend"
}

{
    "command": "startRecording",
    "data": {
        "endpoints": ["inproc://imu_data_source_1"],
        "format": "json",
        "filename": "subaruDrive"
    }
}

{
    "command": "stopRecording",
}

{
    "command": "listRecording",
}

{
  "command": "getVersion"
}

{
  "command": "getVersion",
  "status": "ok",
  "data": {
    "version": "1.0.0"
  }
}

Sending FusionHub Data to External Applications via the ZeroMQ Interface

FusionHub emits data resulting from the sensor fusion through the local network interface.

Output Ports

The network port that this information is output to can be configured in the JSON parameter file config.json of FusionHub.

Data Format

As low level protocol to emit the output data we use ZeroMQ (publisher / subscriber). The data itself is in JSON format and is encoded as Protocol Buffers. Protocol Buffers are documented here. Message are defined in the Protobuf (.protoc) format as defined in the file stream_data.proto. This file is contained in the installation folder of FusionHub.

Python Resources

Download a Python example that shows how to decode messaged from FusionHub from this repository.

Prerequisites can be installed in your Python 3 environment with this:

pip install zmq
pip install protobuf

Make sure to set the input port in FusionHubPythonExample.py correctly. For example for the Antilatency source definition like below, the port needs to be set to 8899.

"optical": {
    "type": "Antilatency",
    "settings": {
        // Use this for access from an external process eg. ALVR
        "endpoint": "tcp://*:8899",
        
        "environmentLink": "AntilatencyAltEnvironmentHorizontalGrid~AgAEBLhTiT_cRqA-r45jvZqZmT4AAAAAAAAAAACamRk_AQQCAwICAgICAQICAAI",
        "placementLink": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
    }
}

C# Resources

On parsing Protobuf files: https://github.com/5argon/protobuf-unity

How to subscribe to ZeroMQ messages: https://github.com/gench23/unity-zeromq-client and https://tech.uqido.com/2020/09/29/zeromq-in-unity/

VRPN Output

VRPN output is set in the following part in the sinks section of config.json. The device name will be referenced by the plugin for Unreal engine.

"VRPN": {
  "settings": {
    "deviceName": "Fusion Hub"
  }
}	  

Please see below how we achieve data input via VRPN in the Unreal engine. First, install the VRPN LiveLink plugin:

Configure the VRPN source with the correct device and subject name:

Apply the output from the fusion hub to an Unreal object eg. a cine camera actor.

Hardware Preparation

Inertial Measurement Units

General documentation for LPMS IMUs is here.

Switching LPMS-IG1(P) to USBxpress Mode

Note: These instructions work for LPMS-IG1 (IMU only) and LPMS-IG1P (IMU + GPS).

First, download LpmsControl 2 from here and install it.

Connect LPMS-IG1(P) to your computer and start LpmsControl 2.

In LpmsControl 2 select one of the LPMS-IG1(P) sensors and connect to it.

In case the sensor is in VCP (virtual COM port) mode as shown below, click on Convert to switch the sensor to USBxpress mode. This is required for communication with FusionHub.

After converting the sensor to USBxpress mode it should be displayed as such.

The image below shows typical output from LPMS-IG1(P) after connecting.

Close LpmsControl 2 to disconnect from the sensor. You are now ready to use LPMS-IG1(P) in FusionHub.