LPVR-POS Manual

Introduction

FusionHub is a software application that has the purpose of combining a number of sensor data inputs to create a higher level information output. LPVR-POS is based on FusionHub and combines odometry, GPS and IMU data from a vehicle to calculate high-accuracy and low-latency global localization information. There is a range of typical applications such as automotive and robot localization.

The diagram below shows the general structure of FusionHub. Sources and sinks are connected by a filter unit. The sensor fusion functionality is contained in this filter unit. The filter parameters as well as the parameters of input and output blocks can be configured via a configuration script or a graphical user interface. Note that this image is a general block diagram of FusionHub showing more items than the functionality needed for LPVR-POS.

The graphical user interface is detached from the main FusionHub application and both applications can therefore run on separate computers. This provides flexibility for running FusionHub on devices with limited monitoring capabilities like a head mounted display.

General

Starting FusionHub

FusionHub consists of two components:

  • The main application

  • A graphical user interface application

The main FusionHub application is started by running FusionHub.exe. No specific installation is needed, the application can be run directly out of its deployment directory. It is a command line application that uses the file config.json for its configuration. We will explain the contents and options of the configuration file further below.

Please install the graphical user interface by running lp-fusionhub-dashboard_0.1.0_x64_en-US.msi. It installs lp-fusionhub-dashboard in your start menu, launch the application from there. Press the Connect button after starting FusionHub.exe to connect client and server. In case you are running FusionHub on a separate machine make sure to enter the correct IP address.

The screenshot below shows the connection elements of the GUI dashboard.

Licensing

FusionHub has two options for license protection. The application will only run correctly if one of the two license options is satisfied.

Hardware dongle

License authentication using a hardware dongle; This is especially interesting for air-gapped installations that are not connected to the internet. As long as the dongle is inserted into a USB slot of the host system, FusionHub will run. Please note that for the Android (eg. on Quest 2 HMD) version of FusionHub, the GUI running on the streaming host is dongle protected, see more detailed information in the specific manual chapter.

Online license

License authentication using a software, online license; This makes sense for systems that are connected to the internet at least during the initial installation of FusionHub. The software checks its license status with our license server with following sequence:

  1. Enter license key in configuration file. You receive your personal license key from us.

  2. FusionHub application sends license key and machine code to server.

  3. Server checks if license is valid and returns response code, if it is valid.

  4. Copy the response code from the log and enter it in the config file to the ResponseKey parameter. Save the config file.

  5. This allows FusionHub to run on this specific machine without reconnecting to the internet. One license unit will be subtracted from your license account. Please ask us for assitance if you’d like to move your license.

If your default configuration file config.json doesn't contain it already, add the LicenseInfo block as shown below. Enter your personal key you received from us as LicenseKey.

{ ... "LicenseInfo": { "LicenseKey": "EKKCO-GZYLT-NJKET-SASDC", "ResponseKey": "" } ... }

LPVR-POS Filter Configuration

LPVR-POS combines odometry, GPS and IMU data from a vehicle to calculate high-accuracy and low-latency global localization information. While GPS or RTK-GPS measurements alone provide similar positioning accuracy, the output frequency of these systems is relatively low, making them unsuitable for applications where localization information at higher framerates is required, such as positioning objects in an augmented reality environment.

By additionally using odometry (wheel speeds, steering angle etc.) information, the localization data from the GPS measurements is interpolated to achieve framerates limited only by IMU and odometry sampling speeds.

The LPVR-POS filter has two operation modes with different configuration blocks in config.json and different output formats. The two modes are:

  • Low-dynamics filter (LD)

  • High-dynamics filter (HD)

The diagram below shows a general overview of the LPVR-POS filter setup.

Installation of Hardware Components

Inertial Measurement Unit (IMU)

LPMS-IG1P needs to be installed in the vehicle in a known orientation ideally with the coordinate axes of the IMU arranged in parallel to the vehicle coordinate system. As vehicle reference frame we are using the VW coordinate system shown in the image below.

Connect the USB connector of LPMS-IG1P to the host computer. If needed an active or passive USB extension can be used. Make sure to check data integrity with the LpmsControl 2 data acquisition tool, we have noticed communication issues with some passive USB extensions.

See the appendix for further information how to set up LPMS-IG1.

VW frame x: back y: right z: up

Global Positioning System (GPS)

We provide two options for using a global position reference:

  • The GPS receiver integrated with the LPMS-IG1P sensor; Connect the antenna cable and place the GPS antenna on top of the vehicle

  • Standalone multi-channel RTK GPS receiver module; This requires setting up an RTK-GPS base station. Details are provided in this chapter.

Note on the antenna placement: We recommend placing the GPS antenna on top, in the center of the car.

CAN Bus Connection

FusionHub can be connected to the vehicle CAN bus by using one of the following CAN bus interfaces:

Both LPVR-POS filter types require reading out wheel velocities from the vehicle’s CAN bus. This means we need to connect the CAN interface that’s being used with FusionHub to the vehicle’s CAN lines. This can happen for example through the car’s OBD2 connector, or by directly tapping into its wire harness.

Once access to the CAN bus is available the CAN messages from the car need to be decoded to calculate the correct wheel velocity values. This decoding will in the future be configurable. At the moment we offer decoder configuration presets for a few vehicle types. Please contact us for details.

LPVR-POS Sensor Fusion Filter

Filter Inputs

Both the LD and the HD filter need the following sources as input. One option is to operate the filter with our LPMS-IG1P sensor that contains IMU and a GPS receiver. This allows for standard GPS absolute position accuracy. Relative accuracy and update rate are higher, based on odometry and IMU data. The other option is to use a separate RTK-GPS unit to get high accuracy RTK-GPS readings. We will look at how to set up an RTK-GPS system for LPVR-POS in this chapter.

IMU & GPS Option 1 - LPMS-IG1P Data Source for IMU and GPS Data

LPMS-IG1P Source

LPMS-IG1P combines IMU sensor and GPS data source.

"imuP": { "type": "DualRtk", "settings": { "sensor1": { // If specification needed, insert first IG1 sensor name here //"name": "ig1p232800650050", "autodetectType": "ig1p" }, "rtcm": false, "imuEndpoint": "tcp://*:8802" } }

Parameter name

Description

Default

Parameter name

Description

Default

type

Type of GPS receiver. Currently only DualRtk is allowed.

DualRTK

name

The name of the LPMS-IG1P sensor used in this setup. This parameter is optional. If FusionHub is operated at the same time with LPVR-DUO, we recommend specifying the sensor name. Look up the sensor name in LpmsControl 2.

n/a

autodetectType

Type of sensor to be autodetcted

ig1p

rtcm

Set to true if RTCM input is to be received eg. from an NTRIP source.

false

imuEndpoint

Output endpoint of IMU data. This parameter is optional.

tcp://*:8802

We currently don’t have an elegant way to copy & paste the name of the IG1 sensor to be used by FusionHub to the configuration script from LPMS-Control. Please see the image below where to (manually) copy the name from.

IMU & GPS Option 2 - Separate LPMS-IG1 IMU and RTK GPS Sources (with NTRIP Caster for RTK Correction)

LPMS-IG1 Source

This node connects to an IMU sensor. For LPVR-POS use LPMS-IG1P. Note that this node doesn’t read out GPS data from LPMS-IG1P.

Parameter name

Description

Default

Parameter name

Description

Default

type

Type of IMU. At the moment only OpenZen IMUs are supported.

OpenZen

name

The name of the LPMS-IG1 sensor used in this setup. This parameter is optional. If FusionHub is operated at the same time with LPVR-DUO, we recommend specifying the sensor name. Look up the sensor name in LpmsControl 2.

n/a

autodetectType

Type of sensor to be autodetcted

ig1

imuEndpoint

Output endpoint of IMU data. This parameter is optional.

tcp://*:8802

RTCM Source

Node to connect to an NTRIP caster and read in RTCM information from there.

Parameter name

Description

Default

Parameter name

Description

Default

type

Type of RTCM correction data source. Currently only NTRIP is allowed.

NTRIP

host

NTrip caster host.

192.168.1.1

port

NTrip caster port.

2101

mountpoint

NTrip mountpoint or stream to receive rtcm correction data.

 

user

NTrip caster username.

 

password

NTrip caster password.

 

userAgent

Name of user agent when connecting to NTrip caster.

LPVR-POS

initialLatitude

Latitude to forward to Ntrip caster on first connect.

0.0

initialLongitude

Longitude to forward to Ntrip caster on first connect.

0.0

forwardGnss

Set true if gnss data from gnss source is to be forwarded to NTRIP caster. This is useful if Ntrip caster offers dynamic switching of RTCM correction data based on forwarded location.

false

GNSS Source

This is the connection to the actual GNSS module such as the ZED-F9P module. The nodes receives data in NMEA format and sends RTK correction information to the node using RTCM.

Parameter name

Description

Default

Parameter name

Description

Default

type

Data output format for gnss data source. Currently only NMEA is allowed.

NMEA

port

Serial port number for gnss source.

  • For Linux this parameter needs to be in format /dev/ttyXXX

  • For Windows it is COMxxx

 

baudrate

Serial port baudrate to connect to GNSS source.

 

rtcm

Set true to enable RTCM correction data forwarding from RTCM source to gnss module.

false

For details about the overall configuration of the RTK-GPS system refer to this section.

Odometry

The vehicle source is the data source that reads raw CAN data messages from a given CAN bus interface and parses the vehicle data using to the provided CAN protocol.

This section explains how to set up the configuration for a custom CAN parsing node. Depending on the available information about the vehicle CAN protocol, there are three types of CAN parsing available.

  • Minimal parsing where we only have information about the vehicle speed CAN message.

  • External separate four wheel speed parsing, velocity sign information may be added if the gear stick reading is available.

  • ExternalWithSteering parsing with steering model information, there are two ways to specify the steering tables depending on the information available: steering wheel angle vs (right or left) wheel angles or steering wheel angle vs average wheel angles. Using this data we can perform linear interpolation to compute the wheel angles for any given steering angle and use it for Ackermann steering.

In most cases, having the overall vehicle speed message or the separate wheel speed messages is more than enough and no steering information is needed. If available, the gear stick reading is useful to help identify the reverse driving.

Minimal Vehicle Parsing

This parser only requires information about the overall vehicle speed message. So it only provide this data as input to the fusion algorithm. Even though this can be enough for some use cases, it may be better to provide separate wheel speeds to extract and use the vehicle angular velocity information as well.

Parameter name

Description

Default

Parameter name

Description

Default

canInterface

CAN bus interface, supported values are "PeakCAN" and "Vector"

PeakCAN

vehicleType

"Minimal"

n/a

velocityCanId

CAN id for the velocities message

n/a

velocityScale

Value to multiply the parsed raw velocity by in order to get velocities in m/s.

1.0

velocityStartBit

Start bit of the vehicle wheel velocity.

0

velocityBitsLength

Number of bits used for wheel velocity encoding.

16

Configuration example

Vehicle Parsing with Separate Wheel Velocities

Parameter name

Description

Default

Parameter name

Description

Default

canInterface

CAN bus interface, supported values are "PeakCAN" and "Vector"

PeakCAN

vehicleType

Used to specify the parser type, available values are:

"Minimal", "External" and "ExternalWithSteering"

n/a

wheelBase

Distance between front and back wheels.

n/a

trackWidth

Distance between right and left back wheels.

n/a

velocityCanId

CAN id for the velocities message

n/a

velocityScale

Value to multiply the parsed raw velocities by in order to get velocities in m/s.

1.0

velocityStartBits

Within this parameter, one need to specify the starting bits of each wheel velocity "FrontLeftWheel", "FrontRightWheel", "RearLeftWheel"and "RearRightWheel" .

0, 16, 32, 48

respectively.

velocityBitsLength

Number of bits used for each wheel velocity encoding.

16

endianness

Specify if the CAN data endianness is "little" or "big" .

little

driveModel

When no steering information is available, only Differential model can be used.

Differential

canEndpoint

Output endpoint of Vehicle data. This parameter is optional.

tcp://*:8802

Configuration example

Additional Parameters

Gear Stick Position for Velocity Sign

If available, the gear stick reading data can be parsed to determine the velocity sign.

Parameter name

Description

Default

Parameter name

Description

Default

gearCanId

Gear stick message CAN id

n/a

gearStartBit

Gear stick message start bit

n/a

gearBitsLength

Number of bits used for gear stick message encoding.

n/a

gearReverseValue

Reverse driving gear stick value.

n/a

Configuration template

Steering information for Ackermann model

If available, the steering wheel message can also be parsed but using it in the drive model requires input about the vehicle steering model as detailed below. Although we provide this option, using simpler drive models proved to work just fine.

Parameter name

Description

Default

Parameter name

Description

Default

steeringCanId

CAN id for the steering wheel angle message

n/a

steeringScale

Value to multiply the parsed raw steering wheel angle by in order to get angle in degrees.

1.0

steeringWheelAngles

The steering table column containing sample steering wheel angles.

n/a

wheelAngleType

Specification of provided wheelAngles type, valid options are "Right", "Left" or "Average"

n/a

wheelAngles

Steering table column containing corresponding wheel angle values.

n/a

driveModel

In addition to Differential model, Ackermann or SimpleAckermann can be used. the latter is a version of Ackermann steering that uses the average wheel angles.

Differential

Configuration template

If the input steering table is like follows (note that the input table should contain values for both right and left turns aka: positive and negative steering wheel angles),

steering wheel angles

right wheel angles

steering wheel angles

right wheel angles

-360

-30.0

-340

-25.5

340

35.5

360

40.0

The steeringTable configuration block would look something like below.

Predefined Vehicle Parsing

For a small number of test vehicles we created a “hardcoded” parser to translate the CAN bus data of these vehicles to input values for our sensor fusion. This predefined parsing is now mostly obsolete because of our new flexible parser. We’ll put the definition of this parser here for completeness.

Parameter name

Description

Default

Parameter name

Description

Default

type

Type of vehicle. Currently only Automotive allowed.

Automotive

vehicleStateEndpoint

Endpoint for vehicle state output

tcp://*:8999

canInterface

CAN interface used for readin odometry data. Allowed options:

  • PeakCAN

  • Vector

PeakCAN

vehicleType

Type of vehicle. Currently supported vehicles have to be manually added. Contact us for details.

R56 (BMW Mini)

Vehicle Source Output Data Types

The vehicle source published the raw CANData it reads from the CAN bus in addition to VehicleState and VehicleSpeed message. The latter is used as input to the sensor fusion algorithms.

Fusion Filter Option 1 - Low-dynamics Filter

General

The low dynamics (LD) filter combines odometry, GPS and IMU data to calculate the global position of a vehicle. This filter was initially concipated to work only with odometry and GPS data to calculate the 2D position and yaw angle of a vehicle. It is therefore suitable for simple tracking scenarios. For more complicated, for example augmented reality, applications we recommed using the HD filter as it outputs globally referenced 3D orientation additionally to globally referenced position. This filter relies heavily on the wheel velocity data of the car published on the car’s CAN bus. Therefore the quality of the output of this filter depends on the accuracy of this information.

Notes on Data Output

The LD filter outputs fusedVehiclePose. The fusedVehiclePose contains a 3D acceleration vector. The acceleration is defined in the following manner: There's a configuration flag imuToCarRotation which takes a quaternion used to rotate vectors in the IMU frame to the car frame. By default it is the identity quaternion. For the LD model, the measured IMU acceleration is simply rotated by the imuToCarRotation and written to the output.

In the LD filter, pitch and roll has to be derived from the acceleration data based on a model of the stiffness of the chassis. That assumes a flat surface. The HD model offers the full 6-DOF, and we are planning to unify them to have all data available at all times.

Notes on IMU Arrangement

LPMS-IG1P needs to be installed in the vehicle in a known orientation ideally with the coordinate axes of the IMU arranged in parallel to the vehicle coordinate system. The LD filter uses the imuTurnRateAxis parameter to determine which axis it should use to calculate the vehicle’s orientation. For example if the IMU is installed in the car so that the Z axis is pointing upwards, imuTurnRateAxis should be set to 0, 0, 1.

Configuration Block

Insert the following configuration block into your config.json file in the sinks section to activate the LD filter. The filter's node name is vehicularFusion.

 

See below a description of the parameter options of the LD filter.

Parameter name

Description

Default

Parameter name

Description

Default

echoFusedPose

fusedVehiclePose output is printed to command line

false

endpoint

Output port for the fusion result

8801

fitModel

Model to use for fusion. At the moment only SimpleCarModel is supported.

SimpleCarModel

driveModel

Model used to calculate the car trajectory from CAN bus data. If the steering wheel data and steering model are provided, Ackermann model can be used.

Differential

velError

Velocity error for Kalman filter. Keep default value.

0.277777778

omegaError

Omega error for Kalman filter. Keep default value.

0.5

measurementError

Measurement error for Kalman filter. Keep default value.

0.1

smoothFit

Enable this option to prevent filter output from jumping between odometry data and GPS measurement. Keep enabled.

true

useImuTurnRate

If enabled the IMU turn rate is used instead of the wheel velocity based turn rate. Recommended.

false

imuTurnRateAxis

The IMU axis to use for the Turn rate if useImuTurnRate is enabled.

1, 0, 0

imuToCarRotation

Rotation that is applied to accelerometer data from IMU before output

1, 0, 0, 0

Output Format

See a technical description of FusionHub’s communication interface in one of the following chapters.

JSON

Protobuf

Parameter name

Description

Unit

Parameter name

Description

Unit

acceleration

3D acceleration vector as measured by IMU. Describes the orientation of the vehicle in the vehicle coordinate system.

m/s^2

globalPosition

Longitude and latitude in degrees

degrees

lastDataTime

Unused

s

position

Position relative to starting point with X pointing North and Y pointing East in the current UTM frame

m

timestamp

Timestamp of data acquisition

ns

utmZone

UTM zone

UTM string

yaw

Globally referenced yaw angle

rad

After starting FusionHub, while the car is static, the filter will not deliver a correct yaw angle. The angle will be adjusted to the correct direction after a few seconds of driving the vehicle.

Fusion Filter Option 2 - High-Dynamics Filter

General

The high dynamics filter combines IMU and GPS data to calculate the global position of a vehicle. Instead of using the odometry it uses IMU data to determine the orientation changes of a car on the X, Y and Z axis. The direction of the vehicle is globally referenced from the GPS system. For increasing the direction reference quality a dual-antenna GPS system can be used.

The high dynamics filter works well for scenarios with agressive driving maneuvers such as drifting and cornering. During such maneuvers the turning motion of the wheels generally doesn’t directly correspond with the direction of the vehicle. Therefore for this filter doesn’t rely on wheel velocity measurements. This filter uses information from the wheels to determine if the car has come to a full stop.

As the filter relies heavily on GPS measurements it doesn’t deliver good results indoors. The better the GPS reception, the better the resulting output of the filter. In it’s current state it therefore only works outdoors. In a future version that combines LD and HD filters, this issue will be resolved.

Notes on Filter Output

The HD filter outputs the following data:

  • fusedVehiclePose (2D pose): Output equivalent to the LD filter output. Includes position in meters relative to starting point, global position (lon, lat) and heading.

  • fusedPose (3D pose): relative to starting point, x, y (in meters) + z (height) + 3D orientation quaternion

  • globalFusedPose: globally referenced 3D position (longitude, latitude, height) + 3D orientation quaternion in ENU frame

Which filter output works best depends on your application. For augmented reality applications we recommend using globalFusedPose.

You can find more information regarding the ENU coordinate system here.

Notes on IMU Arrangement

The used car frame is the Volkswagen (VW) coordinate frame convention:

 

The IMU sensor can be mounted in any way but the ImuToCarRotation quaternion needs to be provided to transform the IMU data into VW frame. For example, if the IMU is mounted like follows:

To match the VW frame, we need a 180° rotation around the z axis (clockwise). Therefore, the rotation matrix would be:

And the orientation quaternion woud be [x, y, z, w] = [ 0, 0, 1, 0 ] which can be specified in the configuration like below:

Check this page for more information on how to calculate the orientation quaternion.

Configuration Block

Insert the following configuration block into your config.json file in the sinks section to activate the HD filter. The filter's node name is gnssImuFusion.

See below a description of the parameter options for the HD filter.

Parameter name

Description

Default

Parameter name

Description

Default

echoFusedPose

fusedVehiclePose output is printed to command line

false

endpoint

Output port for the fusion result

8801

fitModel

Model to use for fusion.

ModelGnssImu

accelError

Acceleration error for Kalman filter. Keep default value.

0.01

omegaError

Omega error for Kalman filter. Keep default value.

0.02

measurementError

Measurement error for Kalman filter. Keep default value.

0.05

imuToCarRotation

Orientation quaternion of IMU relative to car frame

1, 0, 0, 0

smoothFit

Enable this option to prevent filter output from jumping between IMU data and GPS measurement. Keep enabled.

true

singleEndpoint

  • If enabled, the different fusion output messages will be published to the same port "endpoint"

  • If disabled, the "endpoint" parameter above, is the output port for the FusedVehiclePose msg type only.

true

poseEndpoint

Output port for the FusedPose msg type.

8804 if singleEndpoint=false

globalPoseEndpoint

Output port for the GlobalFusedPose msg type.

8805 if singleEndpoint=false

outputRawGnssData

Publishes raw Gnss data position instead of the fusion output. Useful for debugging.

false

outputWhenFilterNotReady

Publishes a temporary raw Gnss data output while the filter is initializing. Useful for a minimal check before moving the vehicle.

false

Output Data Format

FusedVehiclePose

JSON
Protobuf

Parameter name

Description

Unit

Parameter name

Description

Unit

acceleration

3D acceleration vector as measured by IMU. Describes the orientation of the vehicle.

m/s^2

globalPosition

Longitude and latitude in degrees

degrees

lastDataTime

Unused

s

position

Position within UTM zone

m

timestamp

Timestamp of data acquisition

ns

utmZone

UTM zone

UTM string

yaw

Globally referenced yaw angle

rad

FusedPose

JSON
Protobuf

Parameter name

Description

Unit

Parameter name

Description

Unit

lastDataTime

Unused

s

orientation

Orientation quaternion in ENU coordinate frame

without unit

position

X, y position + height

m

timestamp

Time of data acqusition

ns

GlobalFusedPose

JSON
Protobuf

Parameter name

Description

Unit

Parameter name

Description

Unit

orientation

Orientation quaternion

without unit

position

Longitude, latitude, height

deg, deg, m

timestamp

Time of data acqusition

ns

Recording and Replaying Data

Data Recording

You can record the output from FusionHub to a file by adding the following lines to the sink section of config.json. All JSON output that's printed to the screen during the operation of FusionHub will be written into the log file.

Parameter name

Description

Default

Parameter name

Description

Default

filename

Filename of the file to be recorded.

driveData.json

format

The file format. At the moment only JSON is possible.

json

Data Replay

Replay executable

In order to replay data from a JSON file a separate application ReplayExecutable.exe that we deliver with LPVR-POS is used. The replay executable reads the JSON data from a defined file, pushes the data to a replay queue and sends them to the network (tcp://localhost:9921 by default). To run the ReplayExecutable use the following command:

Key

Description

Type

Example value

Key

Description

Type

Example value

-r

Path to read in file

String

“log.json”

--replay-speed

Speed to the actual recording

Double

1

--queue-size

The size of queue that file reader would stop pushing new data to the replay queue.

Increase this value when you see lots of data is published at the same time when running with --verbose

Integer

100

--echo-data

Listen to the publishing endpoint and display the replayed data

N/A

N/A

--verbose

Print the debugging information, i.e., the timestamp a packet is added to the replay queue, replayed from the replay queue, and discarded from the replay queue.

N/A

N/A

FusionHub can receive data from the replay application by adding the replay application’s endpoint source to the configuration file:

 

Compatibility with old data recordings

Starting from 19/10/2023, the odometry message type used as input to the filter nodes has changed.

This way the recorded CAN messages are reprocessed to produce the expected input.

Output replay

This is an alternative replay method included in the FusionHub application itself. It can be used to publish the fusion output with a desired frequency. It doesn’t run the fusion but simulates the output instead. The recorded data need to be pre-processed by keeping the desired output message type only and removing the input messages. The configuration for this replay looks like follows, note that the “sinks“ block is empty here except for “echo” if needed.

Parameter name

Description

Default

Parameter name

Description

Default

filename

Filename of the file to be replayed.

N/A

playbackInterval

Time interval in seconds, controls how fast the data is published.

0.001

loop

Whether to loop over the input file.

false

endpoint

Output endpoint.

““

Graphical User Interface

Dashboard Elements

As described in a previous chapter, the input field for the address of the FusionHub main application is in the top section of the dashboard GUI. Standard connection port is 19358. In case FusionHub is running on the same machine leave the address at localhost.

There are multiple subsections to the graphical user interface. For LPVR-POS the Fusion Configuration, Settings and Map sections are most important.

Sensor Fusion Configuration

This section shows the FusionHub configuration as well as several key values reported by the FusionHub main application. After editing the configuration script make sure to press “Set” and “Save” to load the new configuration into FusionHub.

General Settings

This section works as additional connection dialog. It shows the status of the connection between the GUI client and the FusionHub executable.

Map View

The map view is specific to LPVR-POS. It shows the current position and yaw orientation output of FusionHub. This section works well to check if the data currently calculated by LPVR-POS makes sense.

Communication with External Applications

There are multiple ways that LPVR-POS can communicate with external applications:

  • Connect via WebSocket API to FusionHub WebSocket server to configure FusionHub node parameters and to stream data to a client application. This method is being used by the FusionHub frontend application.

  • Stream data to a client application via JSON / Protobuf-encoded ZeroMQ. This method is used internally by FusionHub to connect its various nodes. Each output node can be configured to stream data not only internally but also externally.

  • Forward pose data to a client application such as the Unreal Engine via VRPN. This is in many cases the easiest way of streaming data to a client, but it is not as flexible as using the Websocket API or ZeroMQ.

WebSocket APIs

Apart from manual editing the config.json configuration script or modifying it through the GUI, FusionHub also offers a WebSocket API for external applications to change its configuration. The GUI uses this interface to access FusionHub’s settings.

Note that the websocket communication is currently not encrypted, it is not secure. Please take your own precautions to make sure network traffic for the configuration isn’t intercepted in some way. We might add an option for secure communication in future releases.

The WebSocket server can be accessed via 19358 port on the machine hosting the FusionHub service. To accelerate development download the Simple WebSocket Client Chrome plugin. This allows you to manually enter API commands and check the replies from the server.

Endpoint

Sample Requests

Sample Response / Description

Endpoint

Sample Requests

Sample Response / Description

getConfig

Get in memory configurations.

getSavedConfig

Get on disk configurations.

saveConfig

Save the in-memory configurations to the disk.

setConfig

Update in-memory configurations. This api create new key-value pairs, or update the existing values.

It does not save configurations to the disk.

Note that in "data" you just need to specify the path to the json key to update: the exmaple on the right would change the port to 5005 while everything else is left unchanged.

overwriteConfig

Overwrite the in-memory configurations. This is suitable when user want to remove a key from the configuration.

getIntercalibrationStatus

Get the current intercalibration status. Useful for refetching current status when the frontnend accidentally disconnects.

applyIntercalibrationResults

Apply the current intercalibration quaternion to the in-memory copy of config. This does NOT save to disk.

restartBackend

Restart the backend. Internally the while loop reset the DataBlock, causing all sources and sinks to be freed from memory, and instantiate them again.

startRecording

Listen to data published to endpoints, and write to a file of YYYYMMDD-HHMMSS-{comment}.{format}

stopRecording

Stop the current recording.

listRecording

List the recorded filenames since the FusionHub booted up.

getVersion

Sending FusionHub Data to External Applications via the ZeroMQ Interface

FusionHub emits the data resulting from the sensor fusion through the local network interface.

Output Ports

The network port that this information is output to can be configured in the JSON parameter file config.json of FusionHub. Usually each FusionHub node contains a parameter setting such as "endpoint": "tcp://*:8899" that allows settings the output port. The output from the node can then be accessed by connecting to the IP of the computer that FusionHub is running on the defined port.

Data Format

As low level protocol to emit the output data we use ZeroMQ (publisher / subscriber). The data itself is in JSON format and is encoded as Protocol Buffers. Protocol Buffers are documented here. Messages are defined in the Protobuf (.protoc) format as defined in the file stream_data.proto. This file is contained in the installation folder of FusionHub.

Python Resources

Download a Python example that shows how to decode messages from FusionHub from this repository.

Prerequisites can be installed in your Python 3 environment with this:

Make sure to set the input port in FusionHubPythonExample.py correctly. For example for the Antilatency source definition like below, the port needs to be set to 8899.

C# Resources

On parsing Protobuf files: https://github.com/5argon/protobuf-unity

How to subscribe to ZeroMQ messages: https://github.com/gench23/unity-zeromq-client and https://tech.uqido.com/2020/09/29/zeromq-in-unity/

VRPN Output

VRPN output is set in the following part in the sinks section of config.json. The device name will be referenced by the plugin for Unreal engine.

Please see below how we achieve data input via VRPN in the Unreal engine. First, install the VRPN LiveLink plugin:

Configure the VRPN source with the correct device and subject name:

Apply the output from the fusion hub to an Unreal object eg. a cine camera actor.

RTK-GPS Setup

Overview

RTK is a technique used to improve the accuracy of a standalone GNSS receiver. Traditional GNSS receivers, as used for instance in a smartphone, can only determine the device’s position with 2-4 meters accuracy. RTK can give you centimeter accuracy.

GNSS receivers measure how long it takes for a signal to travel from a satellite to the receiver. Transmitted signals travel through the ionosphere and atmosphere and are slowed down and perturbed on the way. For example, travel time on a cloudy day and in clear sky conditions would be different. That is why it is difficult for a standalone receiver to precisely determine its position. RTK is a technology that solves this issue.

High real-time precision

Two receivers are used in RTK. One of them is stationary, another moves freely. They are called base station and rover.

The base's mission is to stay in one place and send corrections to a moving receiver. Rover uses that data to achieve centimeter precise position. Any number of rovers can connect to one base if their input settings match the base's output.

Corrections over NTRIP

The correction data is sent from the base station to the rover as RTCM data packets, that are sent using the NTRIP protocol. Base and rover are connected via a caster service, that relays the data from the base to the rover.

For LPVR-POS we use a free NTRIP caster service provided by the company Emlid. The caster service handles the NTRIP communication between base station and LPVR-POS in the car, without the need of these systems knowing the IP address of each other. Besides the Emlid caster, alternatively another caster service could be used, or you could even easily set up your own caster service.

How to Set-up RTK-GPS for LPVR-POS

In the case of LPVR-POS the above-mentioned rover is the vehicle that contains the computer running LPVR-POS. To enable GPS with centimeter accuracy, we need to install a base station in a fixed location in the vicinity of the vehicle, or alternatively use an available NTRIP service. We will focus here on setting up a base station as in this way we are location and infrastructure independent.

Components

In order to set up an RTK GPS system to work with LPVR-POS follow the below steps. You need to do these steps only once, after that the system should be ready to work when you turn it on:

  • Connect the ZED-F9P RTK GNSS receiver board to the computer running FusionHub.

  • The RTK-GPS board communicates with the computer using a virtual COM port. In case of using a Windows system, determine the address of the COM port by looking at the device manager. In the example below the ZED-F9P, our RTK-GPS module, has the address COM4. Enter this address in the GNSS node configuration.

  • Configure the ZED-F9P RTK-GPS module (only needs to be done once):

    • Download u-center.

    • Make sure GGA, GSA and VTG messages are enabled.

    • Before that, you need to choose “Receiver“ tab in menu bar, navigate to “Connection“ and connect to the correct COM port, then set the baudrate to 115200.

    • Next, navigate to “View“ → “Messages View“, right-click “GxGGA”, “GxGSA“ and “GxVTG“ text to enable them.

    • Then open “Configuration View“, choose “RATE (Rates)” to set frequency to 5Hz by changing measurement period to 200ms.

    • You can checkout the output of the RTK-GPS with “Text Console” to verify if the settings is effective.

    • The last step is to save the settings. Open “Configuration View“, choose “CFG (Configuration)“, select “Save current configuration“ and apply for all devices and then click “Send“ in the bottom left-hand corner to save the settings.

  • Set up an NTRIP caster service by creating a free Emlid Flow account. Note that you only need the free version.

  • Set up the Emlid RS2+ base station. A detailed guide how to connect it to the internet using a SIM card is here: https://docs.emlid.com/reachrs2/rtk-quickstart/connecting-reach-to-the-internet/enabling-cellular-modem

  • Next, adjust the base station settings so that the base station outputs correction data to our NTRIP caster: Note that we only need to follow the instructions on setting up the base station, the rover we set up via the FusionHub interface.

  • Configure the base station to talk to our Emlid Flow account:

  • Set up FusionHub to receive RTCM data from the Emlid Flow caster.

For example if your caster configuration is the following:

Set the parameters in your config.json to the following:

  • Once you successfully connect your base, you will see the ONLINE notification next to the mount point name.

  • Once you start FusionHub and it successfully connects to our NTRIP caster, you will see the ONLINE notification and the number of the connected rovers in the My rovers section.

  • FusionHub should now output the status of the RTK-GPS connection to the command line. Note that currently we don’t yet output the status in the GUI. This will follow soon. There are three states to the RTK-GPS system:

    • GPS (RTK not connected, output has normal GPS accuracy)

    • Float (RTK connected and working, but not enough data to get high accuracy output)

    • Fix (RTK connected and in high-accuracy mode)

These three states are reported by the GNSS node as GnssData messages with the quality field:

Quality flag value

Name

Result

Quality flag value

Name

Result

2

DGPS

The GNSS unit is operating as normal GPS unit. It is not receiving correction information from an NTRIP server or other RTCM source.

4

RTK fixed

The RTK-GPS system is locked and operating with ~2cm positioning accuracy. This is the desired state.

5

RTK float

The GNSS unit is receiving correction information from an RTCM source, but so far hasn’t been able to lock to high precision mode. There might not be enough satellites in view or the system needs more time to stabilize.

Preferably we always want the system to operate in Fix mode. This requires working connection to the NTRIP caster, decent antenna placement and a sufficient number of satellites in view of the car’s antenna and the base station antenna.

As for LPVR-POS we are using a multi-channel GPS system, performance should be relatively robust. Check the output of the Emlid software to check the number of satellites the base station is seeing.

  • If you managed to get an RTK-GPS fix, your LPVR-POS system is now ready to operate with RTK-GPS as global centimeter-accurate position reference.

Using Alternate NTRIP Servers

In principle the FusionHub RTCM node can be used with any NTRIP server. We’ve so far only tested it using the Emlid caster described above and the Softbank Ichimili service in Japan. When configuring FusionHub for an alternate NTRIP server make sure to correctly modify all fields of the RTCM node. Make sure that the initialLatitude and initialLongitude coordinates point roughly to the area where you would like to operate the system. Check the command line output of FusionHub, if the connection to the NTRIP server was successful:

  • There should be a line saying "Output to NTRIP server: .." with the initial GET command that's sent to the NTRIP server

  • The output "Successfully connected to NTRIP Caster" should follow a few lines further on

If you see these messages, we know that FusionHub connected correctly to the NTRIP server. Check the GnssData message as described above for connection state.

References

Emlid caster documentation

Emlid Reach RS2+ documentation

Ublox ZED-F9P module documentation

Appendix

Example Configuration Scripts

LPVR-POS with LD filter and RTK-GPS node

LPVR-POS with HD filter and LPMS-IG1P GPS node

LPMS-IG1 Setup

General documentation for LPMS IMUs is here.

Switching LPMS-IG1(P) to USBxpress Mode

Note: These instructions work for LPMS-IG1 (IMU only) and LPMS-IG1P (IMU + GPS).

  • First, download LpmsControl 2 from here and install it.

  • Connect LPMS-IG1(P) to your computer and start LpmsControl 2.

  • In LpmsControl 2 select one of the LPMS-IG1(P) sensors and connect to it.

  • In case the sensor is in VCP (virtual COM port) mode as shown below, click on Convert to switch the sensor to USBxpress mode. This is required for communication with FusionHub.

  • After converting the sensor to USBxpress mode it should be displayed as such.

  • The image below shows typical output from LPMS-IG1(P) after connecting.

  • Close LpmsControl 2 to disconnect from the sensor. You are now ready to use LPMS-IG1(P) in FusionHub.