Introduction

FusionHub is a software application that has the purpose of combining a number of sensor data inputs to create a higher level information output. LPVR-POS is based on FusionHub and combines odometry, GPS and IMU data from a vehicle to calculate high-accuracy and low-latency global localization information. There is a range of typical applications such as automotive and robot localization.

The diagram below shows the general structure of FusionHub. Sources and sinks are connected by a filter unit. The sensor fusion functionality is contained in this filter unit. The filter parameters as well as the parameters of input and output blocks can be configured via a configuration script or a graphical user interface. Note that this image is a general block diagram of FusionHub showing more items than the functionality needed for LPVR-POS.

The graphical user interface is detached from the main FusionHub application and both applications can therefore run on separate computers. This provides flexibility for running FusionHub on devices with limited monitoring capabilities like a head mounted display.

Open

General

Starting FusionHub

FusionHub consists of two components:

• The main application

• A graphical user interface application

The main FusionHub application is started by running FusionHub.exe. No specific installation is needed, the application can be run directly out of its deployment directory. It is a command line application that uses the file config.json for its configuration. We will explain the contents and options of the configuration file further below.

Please install the graphical user interface by running lp-fusionhub-dashboard_0.1.0_x64_en-US.msi. It installs lp-fusionhub-dashboard in your start menu, launch the application from there. Press the Connect button after starting FusionHub.exe to connect client and server. In case you are running FusionHub on a separate machine make sure to enter the correct IP address.

The screenshot below shows the connection elements of the GUI dashboard.

Open

Licensing

FusionHub has two options for license protection. The application will only run correctly if one of the two license options is satisfied.

Hardware dongle

License authentication using a hardware dongle; This is especially interesting for air-gapped installations that are not connected to the internet. As long as the dongle is inserted into a USB slot of the host system, FusionHub will run. Please note that for the Android (eg. on Quest 2 HMD) version of FusionHub, the GUI running on the streaming host is dongle protected, see more detailed information in the specific manual chapter.

Online license

License authentication using a software, online license; This makes sense for systems that are connected to the internet at least during the initial installation of FusionHub. The software checks its license status with our license server with following sequence:

1. Enter license key in configuration file. You receive your personal license key from us.

2. FusionHub application sends license key and machine code to server.

3. Server checks if license is valid and returns response code, if it is valid.

4. Copy the response code from the log and enter it in the config file to the ResponseKey parameter. Save the config file.

5. This allows FusionHub to run on this specific machine without reconnecting to the internet. One license unit will be subtracted from your license account. Please ask us for assitance if you’d like to move your license.

If your default configuration file config.json doesn't contain it already, add the LicenseInfo block as shown below. Enter your personal key you received from us as LicenseKey.

LPVR-POS Filter Configuration

LPVR-POS combines odometry, GPS and IMU data from a vehicle to calculate high-accuracy and low-latency global localization information. While GPS or RTK-GPS measurements alone provide similar positioning accuracy, the output frequency of these systems is relatively low, making them unsuitable for applications where localization information at higher framerates is required, such as positioning objects in an augmented reality environment.

By additionally using odometry (wheel speeds, steering angle etc.) information, the localization data from the GPS measurements is interpolated to achieve framerates limited only by IMU and odometry sampling speeds.

The LPVR-POS filter has two operation modes with different configuration blocks in config.json and different output formats. The two modes are:

• Low-dynamics filter (LD)

• High-dynamics filter (HD)

The diagram below shows a general overview of the LPVR-POS filter setup.

Open

Installation of Hardware Components

Inertial Measurement Unit (IMU)

LPMS-IG1P needs to be installed in the vehicle in a known orientation ideally with the coordinate axes of the IMU arranged in parallel to the vehicle coordinate system. As vehicle reference frame we are using the VW coordinate system shown in the image below.

Connect the USB connector of LPMS-IG1P to the host computer. If needed an active or passive USB extension can be used. Make sure to check data integrity with the LpmsControl 2 data acquisition tool, we have noticed communication issues with some passive USB extensions.

See the appendix for further information how to set up LPMS-IG1.

Open

VW coordinate frame

Global Positioning System (GPS)

We provide two options for using a global position reference:

• The GPS receiver integrated with the LPMS-IG1P sensor; Connect the antenna cable and place the GPS antenna on top of the vehicle

• Standalone multi-channel RTK GPS receiver module; This requires setting up an RTK-GPS base station. Details are provided in this chapter.

Note on the antenna placement: We recommend placing the GPS antenna on top, in the center of the car.

CAN Bus Connection

FusionHub can be connected to the vehicle CAN bus by using one of the following CAN bus interfaces:

• PCAN USB interface

• Vector CAN USB interface

Both LPVR-POS filter types require reading out wheel velocities from the vehicle’s CAN bus. This means we need to connect the CAN interface that’s being used with FusionHub to the vehicle’s CAN lines. This can happen for example through the car’s OBD2 connector, or by directly tapping into its wire harness.

Once access to the CAN bus is available the CAN messages from the car need to be decoded to calculate the correct wheel velocity values. This decoding will in the future be configurable. At the moment we offer decoder configuration presets for a few vehicle types. Please contact us for details.

LPVR-POS Sensor Fusion Filter

Filter Inputs

Both the LD and the HD filter need the following sources as input. One option is to operate the filter with our LPMS-IG1P sensor that contains IMU and a GPS receiver. This allows for standard GPS absolute position accuracy. Relative accuracy and update rate are higher, based on odometry and IMU data. The other option is to use a separate RTK-GPS unit to get high accuracy RTK-GPS readings. We will look at how to set up an RTK-GPS system for LPVR-POS in this chapter.

IMU & GPS Option 1 - LPMS-IG1P Data Source for IMU and GPS Data

LPMS-IG1P Source

LPMS-IG1P combines IMU sensor and GPS data source.

Open

IMU & GPS Option 2 - Separate LPMS-IG1 IMU and RTK GPS Sources (with NTRIP Caster for RTK Correction)

LPMS-IG1 Source

This node connects to an IMU sensor. For LPVR-POS use LPMS-IG1P. Note that this node doesn’t read out GPS data from LPMS-IG1P.

RTCM Source

Node to connect to an NTRIP caster and read in RTCM information from there.

GNSS Source

This is the connection to the actual GNSS module such as the ZED-F9P module. The nodes receives data in NMEA format and sends RTK correction information to the node using RTCM.

For details about the overall configuration of the RTK-GPS system refer to this section.

GNSS Output Format

JSON

Protobuf

Important Output Data

Odometry

The vehicle source is the data source that reads raw CAN data messages from a given CAN bus interface and parses the vehicle data using to the provided CAN protocol.

This section explains how to set up the configuration for a custom CAN parsing node. Depending on the available information about the vehicle CAN protocol, there are three types of CAN parsing available.

• Minimal parsing where we only have information about the vehicle speed CAN message.

• External separate four wheel speed parsing, velocity sign information may be added if the gear stick reading is available.

• ExternalWithSteering parsing with steering model information, there are two ways to specify the steering tables depending on the information available: steering wheel angle vs (right or left) wheel angles or steering wheel angle vs average wheel angles. Using this data we can perform linear interpolation to compute the wheel angles for any given steering angle and use it for Ackermann steering.

Minimal Vehicle Parsing

This parser only requires information about the overall vehicle speed message. So it only provide this data as input to the fusion algorithm. Even though this can be enough for some use cases, it may be better to provide separate wheel speeds to extract and use the vehicle angular velocity information as well.

Configuration example

Vehicle Parsing with Separate Wheel Velocities

Configuration example

Additional Parameters

Gear Stick Position for Velocity Sign

If available, the gear stick reading data can be parsed to determine the velocity sign.

Configuration template

Steering information for Ackermann model

If available, the steering wheel message can also be parsed but using it in the drive model requires input about the vehicle steering model as detailed below. Although we provide this option, using simpler drive models proved to work just fine.

Configuration template

If the input steering table is like follows (note that the input table should contain values for both right and left turns aka: positive and negative steering wheel angles),

The steeringTable configuration block would look something like below.

Predefined Vehicle Parsing

For a small number of test vehicles we created a “hardcoded” parser to translate the CAN bus data of these vehicles to input values for our sensor fusion. This predefined parsing is now mostly obsolete because of our new flexible parser. We’ll put the definition of this parser here for completeness.

Vehicle Source Output Data Types

The vehicle source published the raw CANData it reads from the CAN bus in addition to VehicleState and VehicleSpeed message. The latter is used as input to the sensor fusion algorithms.

Fusion Filter Option 1 - Low-dynamics Filter

General

The low dynamics (LD) filter combines odometry, GPS and IMU data to calculate the global position of a vehicle. This filter was initially concipated to work only with odometry and GPS data to calculate the 2D position and yaw angle of a vehicle. It is therefore suitable for simple tracking scenarios. For more complicated, for example augmented reality, applications we recommed using the HD filter as it outputs globally referenced 3D orientation additionally to globally referenced position. This filter relies heavily on the wheel velocity data of the car published on the car’s CAN bus. Therefore the quality of the output of this filter depends on the accuracy of this information.

Notes on Data Output

The LD filter outputs fusedVehiclePose. The fusedVehiclePose contains a 3D acceleration vector. The acceleration is defined in the following manner: There's a configuration flag imuToCarRotation which takes a quaternion used to rotate vectors in the IMU frame to the car frame. By default it is the identity quaternion. For the LD model, the measured IMU acceleration is simply rotated by the imuToCarRotation and written to the output.

In the LD filter, pitch and roll has to be derived from the acceleration data based on a model of the stiffness of the chassis. That assumes a flat surface. The HD model offers the full 6-DOF, and we are planning to unify them to have all data available at all times.

Notes on IMU Arrangement

LPMS-IG1P needs to be installed in the vehicle in a known orientation ideally with the coordinate axes of the IMU arranged in parallel to the vehicle coordinate system. The LD filter uses the imuTurnRateAxis parameter to determine which axis it should use to calculate the vehicle’s orientation. For example if the IMU is installed in the car so that the Z axis is pointing upwards, imuTurnRateAxis should be set to 0, 0, 1.

Configuration Block

Insert the following configuration block into your config.json file in the sinks section to activate the LD filter. The filter's node name is vehicularFusion.

See below a description of the parameter options of the LD filter.

Output Format

See a technical description of FusionHub’s communication interface in one of the following chapters.

JSON

Protobuf

Fusion Filter Option 2 - High-Dynamics Filter

General

The high dynamics filter combines IMU and GPS data to calculate the global position of a vehicle. Instead of using the odometry it uses IMU data to determine the orientation changes of a car on the X, Y and Z axis. The direction of the vehicle is globally referenced from the GPS system. For increasing the direction reference quality a dual-antenna GPS system can be used.

The high dynamics filter works well for scenarios with agressive driving maneuvers such as drifting and cornering. During such maneuvers the turning motion of the wheels generally doesn’t directly correspond with the direction of the vehicle. Therefore this filter doesn’t rely on wheel velocity measurements for localization like the low-dynamics (LD) filter. This filter uses information from the wheels to determine if the car has come to a full stop.

As the filter relies heavily on GPS measurements it doesn’t deliver good results indoors. The better the GPS reception, the better the resulting output of the filter. In it’s current state it therefore only works outdoors. In a future version that combines LD and HD filters, this issue will be resolved.

Notes on Filter Output

The HD filter outputs the following data:

• fusedVehiclePose (2D pose): Output equivalent to the LD filter output. Includes position in meters relative to starting point, global position (lon, lat) and heading.

• fusedPose (3D pose): relative to starting point, x, y (in meters) + z (height in meters) + 3D orientation quaternion

• globalFusedPose: globally referenced 3D position (longitude, latitude, height in meters) + 3D orientation quaternion in ENU frame

Which filter output works best depends on your application. For augmented reality applications we recommend using globalFusedPose.

You can find more information regarding the ENU coordinate system here.

Notes on IMU Arrangement

For the car frame we use the Volkswagen (VW) coordinate frame convention:

VW frame x: back y: right z: up

Open

The IMU sensor can be mounted in any way but the ImuToCarRotation quaternion needs to be provided to transform the IMU data into VW frame. For example, if the IMU is mounted like follows:

To match the VW frame, we need a 180° rotation around the z axis (clockwise). Therefore, the rotation matrix would be:

And the orientation quaternion woud be [w, x, y, z] = [ 0, 0, 0, 1 ] which can be specified in the configuration like below:

Check this page for more information on how to calculate the orientation quaternion.

Note: The orientation of the IMU for a unity quaternion [ 1, 0, 0, 0 ] would be

Configuration Block

Insert the following configuration block into your config.json file in the sinks section to activate the HD filter. The filter's node name is gnssImuFusion.

See below a description of the parameter options for the HD filter.