Introduction

FusionHub is a software application that has the purpose of combining a number of sensor data inputs to create a higher level information output. LPVR-POS is based on FusionHub and combines odometry, GPS and IMU data from a vehicle to calculate high-accuracy and low-latency global localization information. There is a range of typical applications such as automotive and robot localization.

The diagram below shows the general structure of FusionHub. Sources and sinks are connected by a filter unit. The sensor fusion functionality is contained in this filter unit. The filter parameters as well as the parameters of input and output blocks can be configured via a configuration script or a graphical user interface. Note that this image is a general block diagram of FusionHub showing more items than the functionality needed for LPVR-POS.

The graphical user interface is detached from the main FusionHub application and both applications can therefore run on separate computers. This provides flexibility for running FusionHub on devices with limited monitoring capabilities like a head mounted display.

Open

General

Starting FusionHub

FusionHub consists of two components:

• The main application

• A graphical user interface application

The main FusionHub application is started by running FusionHub.exe. No specific installation is needed, the application can be run directly out of its deployment directory. It is a command line application that uses the file config.json for its configuration. We will explain the contents and options of the configuration file further below.

Please install the graphical user interface by running lp-fusionhub-dashboard_0.1.0_x64_en-US.msi. It installs lp-fusionhub-dashboard in your start menu, launch the application from there. Press the Connect button after starting FusionHub.exe to connect client and server. In case you are running FusionHub on a separate machine make sure to enter the correct IP address.

The screenshot below shows the connection elements of the GUI dashboard.

Open

Licensing

FusionHub has two options for license protection. The application will only run correctly if one of the two license options is satisfied.

Hardware dongle

License authentication using a hardware dongle; This is especially interesting for air-gapped installations that are not connected to the internet. As long as the dongle is inserted into a USB slot of the host system, FusionHub will run. Please note that for the Android (eg. on Quest 2 HMD) version of FusionHub, the GUI running on the streaming host is dongle protected, see more detailed information in the specific manual chapter.

Online license

License authentication using a software, online license; This makes sense for systems that are connected to the internet at least during the initial installation of FusionHub. The software checks its license status with our license server with following sequence:

1. Enter license key in configuration file. You receive your personal license key from us.

2. FusionHub application sends license key and machine code to server.

3. Server checks if license is valid and returns response code, if it is valid.

4. Copy the response code from the log and enter it in the config file to the ResponseKey parameter. Save the config file.

5. This allows FusionHub to run on this specific machine without reconnecting to the internet. One license unit will be subtracted from your license account. Please ask us for assitance if you’d like to move your license.

If your default configuration file config.json doesn't contain it already, add the LicenseInfo block as shown below. Enter your personal key you received from us as LicenseKey.

LPVR-POS Filter Configuration

LPVR-POS combines odometry, GPS and IMU data from a vehicle to calculate high-accuracy and low-latency global localization information. While GPS or RTK-GPS measurements alone provide similar positioning accuracy, the output frequency of these systems is relatively low, making them unsuitable for applications where localization information at higher framerates is required, such as positioning objects in an augmented reality environment.

By additionally using odometry (wheel speeds, steering angle etc.) information, the localization data from the GPS measurements is interpolated to achieve framerates limited only by IMU and odometry sampling speeds.

The LPVR-POS filter has two operation modes with different configuration blocks in config.json and different output formats. The two modes are:

• Low-dynamics filter (LD)

• High-dynamics filter (HD)

The diagram below shows a general overview of the LPVR-POS filter setup.

Installation of Hardware Components

Inertial Measurement Unit (IMU)

LPMS-IG1P needs to be installed in the vehicle in a known orientation ideally with the coordinate axes of the IMU arranged in parallel to the vehicle coordinate system. As vehicle reference frame we are using the VW coordinate system shown in the image below.

Connect the USB connector of LPMS-IG1P to the host computer. If needed an active or passive USB extension can be used. Make sure to check data integrity with the LpmsControl 2 data acquisition tool, we have noticed communication issues with some passive USB extensions.

See the appendix for further information how to set up LPMS-IG1.

Global Positioning System (GPS)

We provide two options for using a global position reference:

• The GPS receiver integrated with the LPMS-IG1P sensor; Connect the antenna cable and place the GPS antenna on top of the vehicle

• Standalone multi-channel RTK GPS receiver module; This requires setting up an RTK-GPS base station. Details are provided in this chapter.

Note on the antenna placement: We recommend placing the GPS antenna on top, in the center of the car.

CAN Bus Connection

FusionHub can be connected to the vehicle CAN bus by using one of the following CAN bus interfaces:

• PCAN USB interface

• Vector CAN USB interface

Both LPVR-POS filter types require reading out wheel velocities from the vehicle’s CAN bus. This means we need to connect the CAN interface that’s being used with FusionHub to the vehicle’s CAN lines. This can happen for example through the car’s OBD2 connector, or by directly tapping into its wire harness.

Once access to the CAN bus is available the CAN messages from the car need to be decoded to calculate the correct wheel velocity values. This decoding will in the future be configurable. At the moment we offer decoder configuration presets for a few vehicle types. Please contact us for details.

LPVR-POS Sensor Fusion Filter

Filter Inputs

Both the LD and the HD filter need the following sources as input. One option is to operate the filter with our LPMS-IG1P sensor that contains IMU and a GPS receiver. This allows for standard GPS absolute position accuracy. Relative accuracy and update rate are higher, based on odometry and IMU data. The other option is to use a separate RTK-GPS unit to get high accuracy RTK-GPS readings. We will look at how to set up an RTK-GPS system for LPVR-POS in this chapter.

IMU & GPS Option 1 - LPMS-IG1P Data Source for IMU and GPS Data

LPMS-IG1P Source

LPMS-IG1P combines IMU sensor and GPS data source.

IMU & GPS Option 2 - Separate LPMS-IG1 IMU and RTK GPS Sources (with NTRIP Caster for RTK Correction)

LPMS-IG1 Source

This node connects to an IMU sensor. For LPVR-POS use LPMS-IG1P. Note that this node doesn’t read out GPS data from LPMS-IG1P.

RTCM Source

Node to connect to an NTRIP caster and read in RTCM information from there.

GNSS Source

This is the connection to the actual GNSS module such as the ZED-F9P module. The nodes receives data in NMEA format and sends RTK correction information to the node using RTCM.

For details about the overall configuration of the RTK-GPS system refer to this section.

GNSS Output Format

JSON

Protobuf

Important Output Data

Odometry

The vehicle source is the data source that reads raw CAN data messages from a given CAN bus interface and parses the vehicle data using to the provided CAN protocol.

This section explains how to set up the configuration for a custom CAN parsing node. Depending on the available information about the vehicle CAN protocol, there are three types of CAN parsing available.

• Minimal parsing where we only have information about the vehicle speed CAN message.

• External separate four wheel speed parsing, velocity sign information may be added if the gear stick reading is available.

• ExternalWithSteering parsing with steering model information, there are two ways to specify the steering tables depending on the information available: steering wheel angle vs (right or left) wheel angles or steering wheel angle vs average wheel angles. Using this data we can perform linear interpolation to compute the wheel angles for any given steering angle and use it for Ackermann steering.

Minimal Vehicle Parsing

This parser only requires information about the overall vehicle speed message. So it only provide this data as input to the fusion algorithm. Even though this can be enough for some use cases, it may be better to provide separate wheel speeds to extract and use the vehicle angular velocity information as well.

Configuration example

Vehicle Parsing with Separate Wheel Velocities

Configuration example

Additional Parameters

Gear Stick Position for Velocity Sign

If available, the gear stick reading data can be parsed to determine the velocity sign.

Configuration template

Steering information for Ackermann model

If available, the steering wheel message can also be parsed but using it in the drive model requires input about the vehicle steering model as detailed below. Although we provide this option, using simpler drive models proved to work just fine.

Configuration template

If the input steering table is like follows (note that the input table should contain values for both right and left turns aka: positive and negative steering wheel angles),

The steeringTable configuration block would look something like below.

Predefined Vehicle Parsing

For a small number of test vehicles we created a “hardcoded” parser to translate the CAN bus data of these vehicles to input values for our sensor fusion. This predefined parsing is now mostly obsolete because of our new flexible parser. We’ll put the definition of this parser here for completeness.

Vehicle Source Output Data Types

The vehicle source published the raw CANData it reads from the CAN bus in addition to VehicleState and VehicleSpeed message. The latter is used as input to the sensor fusion algorithms.

Fusion Filter Option 1 - Low-dynamics Filter

General

The low dynamics (LD) filter combines odometry, GPS and IMU data to calculate the global position of a vehicle. This filter was initially concipated to work only with odometry and GPS data to calculate the 2D position and yaw angle of a vehicle. It is therefore suitable for simple tracking scenarios. For more complicated, for example augmented reality, applications we recommed using the HD filter as it outputs globally referenced 3D orientation additionally to globally referenced position. This filter relies heavily on the wheel velocity data of the car published on the car’s CAN bus. Therefore the quality of the output of this filter depends on the accuracy of this information.

Notes on Data Output

The LD filter outputs fusedVehiclePose. The fusedVehiclePose contains a 3D acceleration vector. The acceleration is defined in the following manner: There's a configuration flag imuToCarRotation which takes a quaternion used to rotate vectors in the IMU frame to the car frame. By default it is the identity quaternion. For the LD model, the measured IMU acceleration is simply rotated by the imuToCarRotation and written to the output.

In the LD filter, pitch and roll has to be derived from the acceleration data based on a model of the stiffness of the chassis. That assumes a flat surface. The HD model offers the full 6-DOF, and we are planning to unify them to have all data available at all times.

Notes on IMU Arrangement

LPMS-IG1P needs to be installed in the vehicle in a known orientation ideally with the coordinate axes of the IMU arranged in parallel to the vehicle coordinate system. The LD filter uses the imuTurnRateAxis parameter to determine which axis it should use to calculate the vehicle’s orientation. For example if the IMU is installed in the car so that the Z axis is pointing upwards, imuTurnRateAxis should be set to 0, 0, 1.

Configuration Block

Insert the following configuration block into your config.json file in the sinks section to activate the LD filter. The filter's node name is vehicularFusion.

See below a description of the parameter options of the LD filter.

Output Format

See a technical description of FusionHub’s communication interface in one of the following chapters.

JSON

Protobuf

Fusion Filter Option 2 - High-Dynamics Filter

General

The high dynamics filter combines IMU and GPS data to calculate the global position of a vehicle. Instead of using the odometry it uses IMU data to determine the orientation changes of a car on the X, Y and Z axis. The direction of the vehicle is globally referenced from the GPS system. For increasing the direction reference quality a dual-antenna GPS system can be used.

The high dynamics filter works well for scenarios with agressive driving maneuvers such as drifting and cornering. During such maneuvers the turning motion of the wheels generally doesn’t directly correspond with the direction of the vehicle. Therefore this filter doesn’t rely on wheel velocity measurements for localization like the low-dynamics (LD) filter. This filter uses information from the wheels to determine if the car has come to a full stop.

As the filter relies heavily on GPS measurements it doesn’t deliver good results indoors. The better the GPS reception, the better the resulting output of the filter. In it’s current state it therefore only works outdoors. In a future version that combines LD and HD filters, this issue will be resolved.

Notes on Filter Output

The HD filter outputs the following data:

• fusedVehiclePose (2D pose): Output equivalent to the LD filter output. Includes position in meters relative to starting point, global position (lon, lat) and heading.

• fusedPose (3D pose): relative to starting point, x, y (in meters) + z (height in meters) + 3D orientation quaternion

• globalFusedPose: globally referenced 3D position (longitude, latitude, height in meters) + 3D orientation quaternion in ENU frame

Which filter output works best depends on your application. For augmented reality applications we recommend using globalFusedPose.

You can find more information regarding the ENU coordinate system here.

Notes on IMU Arrangement

For the car frame we use the Volkswagen (VW) coordinate frame convention:

The IMU sensor can be mounted in any way but the ImuToCarRotation quaternion needs to be provided to transform the IMU data into VW frame. For example, if the IMU is mounted like follows:

To match the VW frame, we need a 180° rotation around the z axis (clockwise). Therefore, the rotation matrix would be:

And the orientation quaternion woud be [w, x, y, z] = [ 0, 0, 0, 1 ] which can be specified in the configuration like below:

Check this page for more information on how to calculate the orientation quaternion.

Note: The orientation of the IMU for a unity quaternion [ 1, 0, 0, 0 ] would be

Configuration Block

Insert the following configuration block into your config.json file in the sinks section to activate the HD filter. The filter's node name is gnssImuFusion.

See below a description of the parameter options for the HD filter.

Output Data Format

FusedVehiclePose

JSON

Protobuf

FusedPose

JSON

Protobuf

GlobalFusedPose

JSON

Protobuf

Recording and Replaying Data

Data Recording

You can record the output from FusionHub to a file by adding the following lines to the sink section of config.json. All JSON output that's printed to the screen during the operation of FusionHub will be written into the log file.

Data Replay

Replay executable

In order to replay data from a JSON file a separate application ReplayExecutable.exe that we deliver with LPVR-POS is used. The replay executable reads the JSON data from a defined file, pushes the data to a replay queue and sends them to the network (tcp://localhost:9921 by default). To run the ReplayExecutable use the following command:

FusionHub can receive data from the replay application by adding the replay application’s endpoint source to the configuration file:

Compatibility with old data recordings

Starting from 19/10/2023, the odometry message type used as input to the filter nodes has changed.

This way the recorded CAN messages are reprocessed to produce the expected input.

Output replay

This is an alternative replay method included in the FusionHub application itself. It can be used to publish the fusion output with a desired frequency. It doesn’t run the fusion but simulates the output instead. The recorded data need to be pre-processed by keeping the desired output message type only and removing the input messages. The configuration for this replay looks like follows, note that the “sinks“ block is empty here except for “echo” if needed.

Graphical User Interface

Dashboard Elements

As described in a previous chapter, the input field for the address of the FusionHub main application is in the top section of the dashboard GUI. Standard connection port is 19358. In case FusionHub is running on the same machine leave the address at localhost.

There are multiple subsections to the graphical user interface. For LPVR-POS the Fusion Configuration, Settings and Map sections are most important.

Sensor Fusion Configuration

This section shows the FusionHub configuration as well as several key values reported by the FusionHub main application. After editing the configuration script make sure to press “Set” and “Save” to load the new configuration into FusionHub.

General Settings

This section works as additional connection dialog. It shows the status of the connection between the GUI client and the FusionHub executable.

Map View

The map view is specific to LPVR-POS. It shows the current position and yaw orientation output of FusionHub. This section works well to check if the data currently calculated by LPVR-POS makes sense.

Communication with External Applications

There are multiple ways that LPVR-POS can communicate with external applications:

• Connect via WebSocket API to FusionHub WebSocket server to configure FusionHub node parameters and to stream data to a client application. This method is being used by the FusionHub frontend application.

• Stream data to a client application via JSON / Protobuf-encoded ZeroMQ. This method is used internally by FusionHub to connect its various nodes. Each output node can be configured to stream data not only internally but also externally.

• Forward pose data to a client application such as the Unreal Engine via VRPN. This is in many cases the easiest way of streaming data to a client, but it is not as flexible as using the Websocket API or ZeroMQ.

WebSocket APIs

Apart from manual editing the config.json configuration script or modifying it through the GUI, FusionHub also offers a WebSocket API for external applications to change its configuration. The GUI uses this interface to access FusionHub’s settings.

Note that the websocket communication is currently not encrypted, it is not secure. Please take your own precautions to make sure network traffic for the configuration isn’t intercepted in some way. We might add an option for secure communication in future releases.

The WebSocket server can be accessed via 19358 port on the machine hosting the FusionHub service. To accelerate development download the Simple WebSocket Client Chrome plugin. This allows you to manually enter API commands and check the replies from the server.

Sending FusionHub Data to External Applications via the ZeroMQ Interface

FusionHub emits the data resulting from the sensor fusion through the local network interface.

Output Ports

The network port that this information is output to can be configured in the JSON parameter file config.json of FusionHub. Usually each FusionHub node contains a parameter setting such as "endpoint": "tcp://*:8899" that allows settings the output port. The output from the node can then be accessed by connecting to the IP of the computer that FusionHub is running on the defined port.

Data Format

As low level protocol to emit the output data we use ZeroMQ (publisher / subscriber). The data itself is in JSON format and is encoded as Protocol Buffers. Protocol Buffers are documented here. Messages are defined in the Protobuf (.protoc) format as defined in the file stream_data.proto. This file is contained in the installation folder of FusionHub.

Python Resources

Download a Python example that shows how to decode messages from FusionHub from this repository.

Prerequisites can be installed in your Python 3 environment with this:

Make sure to set the input port in FusionHubPythonExample.py correctly. For example for the Antilatency source definition like below, the port needs to be set to 8899.

C# Resources

On parsing Protobuf files: https://github.com/5argon/protobuf-unity

How to subscribe to ZeroMQ messages: https://github.com/gench23/unity-zeromq-client and https://tech.uqido.com/2020/09/29/zeromq-in-unity/

VRPN Output

VRPN output is set in the following part in the sinks section of config.json. The device name will be referenced by the plugin for Unreal engine.

Please see below how we achieve data input via VRPN in the Unreal engine. First, install the VRPN LiveLink plugin:

Configure the VRPN source with the correct device and subject name:

Apply the output from the fusion hub to an Unreal object eg. a cine camera actor.

RTK-GPS Setup

Overview

RTK is a technique used to improve the accuracy of a standalone GNSS receiver. Traditional GNSS receivers, as used for instance in a smartphone, can only determine the device’s position with 2-4 meters accuracy. RTK can give you centimeter accuracy.

GNSS receivers measure how long it takes for a signal to travel from a satellite to the receiver. Transmitted signals travel through the ionosphere and atmosphere and are slowed down and perturbed on the way. For example, travel time on a cloudy day and in clear sky conditions would be different. That is why it is difficult for a standalone receiver to precisely determine its position. RTK is a technology that solves this issue.

High real-time precision​

Two receivers are used in RTK. One of them is stationary, another moves freely. They are called base station and rover.

The base's mission is to stay in one place and send corrections to a moving receiver. Rover uses that data to achieve centimeter precise position. Any number of rovers can connect to one base if their input settings match the base's output.

Corrections over NTRIP​

The correction data is sent from the base station to the rover as RTCM data packets, that are sent using the NTRIP protocol. Base and rover are connected via a caster service, that relays the data from the base to the rover.

For LPVR-POS we use a free NTRIP caster service provided by the company Emlid. The caster service handles the NTRIP communication between base station and LPVR-POS in the car, without the need of these systems knowing the IP address of each other. Besides the Emlid caster, alternatively another caster service could be used, or you could even easily set up your own caster service.

How to Set-up RTK-GPS for LPVR-POS

In the case of LPVR-POS the above-mentioned rover is the vehicle that contains the computer running LPVR-POS. To enable GPS with centimeter accuracy, we need to install a base station in a fixed location in the vicinity of the vehicle, or alternatively use an available NTRIP service. We will focus here on setting up a base station as in this way we are location and infrastructure independent.

Components

In order to set up an RTK GPS system to work with LPVR-POS follow the below steps. You need to do these steps only once, after that the system should be ready to work when you turn it on:

• Connect the ZED-F9P RTK GNSS receiver board to the computer running FusionHub.

• The RTK-GPS board communicates with the computer using a virtual COM port. In case of using a Windows system, determine the address of the COM port by looking at the device manager. In the example below the ZED-F9P, our RTK-GPS module, has the address COM4. Enter this address in the GNSS node configuration.

• Configure the ZED-F9P RTK-GPS module (only needs to be done once):

  • Download u-center.

  • Make sure GGA, GSA and VTG messages are enabled.

  • Before that, you need to choose “Receiver“ tab in menu bar, navigate to “Connection“ and connect to the correct COM port, then set the baudrate to 115200.

  • Next, navigate to “View“ → “Messages View“, right-click “GxGGA”, “GxGSA“ and “GxVTG“ text to enable them.

  • Then open “Configuration View“, choose “RATE (Rates)” to set frequency to 5Hz by changing measurement period to 200ms.

  • You can checkout the output of the RTK-GPS with “Text Console” to verify if the settings is effective.

  • The last step is to save the settings. Open “Configuration View“, choose “CFG (Configuration)“, select “Save current configuration“ and apply for all devices and then click “Send“ in the bottom left-hand corner to save the settings.

• Set up an NTRIP caster service by creating a free Emlid Flow account. Note that you only need the free version.

• Set up the Emlid RS2+ base station. A detailed guide how to connect it to the internet using a SIM card is here: Connecting Reach to the Internet via built-in modem | Reach RS2/RS2+

• Next, adjust the base station settings so that the base station outputs correction data to our NTRIP caster: Base and rover setup | Reach RS2/RS2+ Note that we only need to follow the instructions on setting up the base station, the rover we set up via the FusionHub interface.

• Configure the base station to talk to our Emlid Flow account: https://docs.emlid.com/emlid-caster/emlid-caster-base-rover-connection

• Set up FusionHub to receive RTCM data from the Emlid Flow caster.

For example if your caster configuration is the following:

Set the parameters in your config.json to the following:

• Once you successfully connect your base, you will see the ONLINE notification next to the mount point name.

• Once you start FusionHub and it successfully connects to our NTRIP caster, you will see the ONLINE notification and the number of the connected rovers in the My rovers section.

• FusionHub should now output the status of the RTK-GPS connection to the command line. Note that currently we don’t yet output the status in the GUI. This will follow soon. There are three states to the RTK-GPS system:

  • GPS (RTK not connected, output has normal GPS accuracy)

  • Float (RTK connected and working, but not enough data to get high accuracy output)

  • Fix (RTK connected and in high-accuracy mode)

These three states are reported by the GNSS node as GnssData messages with the quality field:

Preferably we always want the system to operate in Fix mode. This requires working connection to the NTRIP caster, decent antenna placement and a sufficient number of satellites in view of the car’s antenna and the base station antenna.

As for LPVR-POS we are using a multi-channel GPS system, performance should be relatively robust. Check the output of the Emlid software to check the number of satellites the base station is seeing.

• If you managed to get an RTK-GPS fix, your LPVR-POS system is now ready to operate with RTK-GPS as global centimeter-accurate position reference.

Using Alternate NTRIP Servers

In principle the FusionHub RTCM node can be used with any NTRIP server. We’ve so far only tested it using the Emlid caster described above and the Softbank Ichimili service in Japan. When configuring FusionHub for an alternate NTRIP server make sure to correctly modify all fields of the RTCM node. Make sure that the initialLatitude and initialLongitude coordinates point roughly to the area where you would like to operate the system. Check the command line output of FusionHub, if the connection to the NTRIP server was successful:

• There should be a line saying "Output to NTRIP server: .." with the initial GET command that's sent to the NTRIP server

• The output "Successfully connected to NTRIP Caster" should follow a few lines further on

If you see these messages, we know that FusionHub connected correctly to the NTRIP server. Check the GnssData message as described above for connection state.

References

Emlid caster documentation

Emlid Reach RS2+ documentation

Ublox ZED-F9P module documentation

Appendix

Example Configuration Scripts

LPVR-POS with LD filter and RTK-GPS node

LPVR-POS with HD filter and LPMS-IG1P GPS node

LPMS-IG1 Setup

General documentation for LPMS IMUs is here.

Switching LPMS-IG1(P) to USBxpress Mode

Note: These instructions work for LPMS-IG1 (IMU only) and LPMS-IG1P (IMU + GPS).

• First, download LpmsControl 2 from here and install it.

• Connect LPMS-IG1(P) to your computer and start LpmsControl 2.

• In LpmsControl 2 select one of the LPMS-IG1(P) sensors and connect to it.

• In case the sensor is in VCP (virtual COM port) mode as shown below, click on Convert to switch the sensor to USBxpress mode. This is required for communication with FusionHub.

• After converting the sensor to USBxpress mode it should be displayed as such.

• The image below shows typical output from LPMS-IG1(P) after connecting.

• Close LpmsControl 2 to disconnect from the sensor. You are now ready to use LPMS-IG1(P) in FusionHub.