FusionHub Manual
To make it easier to read, we’ve split the FusionHub documentation into two sub-manuals:
LPVR-AIR Manual - LPVR-AIR Manual (Client-Side Version)
LPVR-POS Manual - LPVR-POS Manual
Please refer to these for the most up-to-date documentation.
- 1 Introduction
- 2 General
- 2.1 Starting FusionHub
- 2.2 Licensing
- 2.2.1 Hardware dongle
- 2.2.2 Online license
- 3 BASE Filter Configuration
- 3.1 Setup
- 3.2 Operation
- 3.3 Calibration
- 3.4 IMU-Optical Fusion Filter
- 3.4.1 Example Configuration
- 3.4.2 Configuration Block
- 3.4.3 Output Data Format
- 3.5 Source Options
- 3.5.1 Optical Tracking Source Options
- 3.5.1.1 Advanced Realtime Tracking (ART)
- 3.5.1.2 Optitrack
- 3.5.1.3 VICON
- 3.5.1.4 Antilatency
- 3.5.2 IMU Source
- 3.5.2.1 LPMS-IG1
- 3.5.2.2 LPMS-CURS3
- 3.5.1 Optical Tracking Source Options
- 3.6 Graphical User Interface
- 3.6.1 Dashboard
- 3.6.2 3D Viewer
- 3.6.3 Sensor Fusion Configuration and Calibration Status
- 3.6.4 General Settings
- 4 MOVE Filter Configuration
- 5 FLOW Filter Configuration
- 5.1 Installation of Hardware Components
- 5.2 Low-dynamics Filter (Odometry + GPS + (some) IMU)
- 5.2.1 Configuration Block
- 5.2.2 Output Data Format
- 5.2.3 Additional Notes
- 5.2.4 Example Configuration
- 5.3 High-Dynamics Filter (IMU + GPS)
- 5.3.1 Configuration block example (in sinks section)
- 5.3.1.1 Setting up the ImuToCarRotation parameter
- 5.3.1.2 Example
- 5.3.2 Output data format
- 5.3.2.1 FusedVehiclePose
- 5.3.2.2 FusedPose
- 5.3.2.3 GlobalFusedPose
- 5.3.3 Example Configuration
- 5.3.1 Configuration block example (in sinks section)
- 5.4 Data Playback [DEPRECATED, switch over to ReplayExecutable]
- 5.5 Replay Node [DEPRECATED, switch over to ReplayExecutable]
- 5.6 Replay Executable
- 5.7 Graphical User Interface
- 5.7.1 Map View
- 6 Data Playback and Recording
- 6.1 Data Recording
- 6.1.1 Record node
- 6.1.2 File Logger
- 6.2 Data Playback
- 6.1 Data Recording
- 7 Communication with External Applications
- 8 Hardware Preparation
- 9 FusionHub on OpenXR HMDs
- 9.1 General
- 9.2 System Components
- 9.2.1 Applications
- 9.2.2 Authentification
- 9.3 Running the Solution
- 9.3.1 Installation
- 9.3.1.1 Meta Quest 2 / Pro
- 9.3.1.2 VIVE Focus 3
- 9.3.1.3 nReal Glasses
- 9.3.2 Start-up
- 9.3.1 Installation
- 9.4 Optical Tracking Systems
- 9.4.1 Marker Adjustment
- 9.4.2 Optitrack Notes
- 9.5 Troubleshooting
- 10 Release Notes
- 10.1 Version 1.2
- 10.2 Version 1.1
- 10.3 Version 1.0
Introduction
FusionHub is a software application that has the purpose of combining a number of sensor data inputs to create a higher level information output. There are 3 basic versions of FusionHub:
FusionHub BASE combines data from an outside-in tracking system with inertial measurements done by an inertial measurement unit (IMU). Typical applications: Head-mounted display tracking for VR/AR applications, camera tracking for virtual production
FusionHub MOVE adds an additional platform IMU to the BASE configuration. It combines data from both IMUs to calculate poses relative to a moving platform. Typical applications: AR/VR in a vehicle, aircraft, or on a simulator platform
FusionHub FLOW combines odometry, GPS and IMU data from a vehicle to calculate high-accuracy and low-latency global localization information. Typical applications: Automobile localization, robot localization
The diagram below shows the general structure of FusionHub. Sources and sinks are connected by a filter unit. The sensor fusion functionality is contained in this filter unit. The filter parameters as well as the parameters of input and output blocks can be configured via a configuration script or the graphical user interface.
The graphical user interface is detached from the main FusionHub application and both applications can therefore run on separate computers. This provides flexibility for running FusionHub on devices with limited monitoring capabilities like a head mounted display.
General
Starting FusionHub
FusionHub consists of two components:
The main application
A graphical user interface application
Insert the security dongle into a USB port of your computer.
The main FusionHub application is started by running FusionHub.exe
. No specific installation is needed, the application can be run directly out of its deployment directory. It is a command line application that uses the file config.json
for its configuration. We will explan the contents and options of the configuration file further below.
Please install the graphical user interface by running lp-fusionhub-dashboard_0.1.0_x64_en-US.msi
. It installs lp-fusionhub-dashboard
in your start menu, launch the application from there. Press the Connect
button after starting FusionHub.exe
to connect client and server. In case you are running FusionHub on a separate machine make sure to enter the correct IP address.
The screenshot below shows the connection elements of the GUI.
Licensing
FusionHub has two options for license protection:
Hardware dongle
License authentication using a hardware dongle; This is especially interesting for air-gapped installations that are not connected to the internet. As long as the dongle is inserted into a USB slot of the host system, FusionHub will run. Please note that for the Android (Quest 2 HMD) version of FusionHub, the GUI running on the streaming host is dongle protected, see more detailed information in the specific manual chapter.
Online license
License authentication using a software, online license; This makes sense for systems that are connected to the internet at least during the initial installation of FusionHub. The software checks its license status with our license server with following sequence:
Enter license key in configuration file. You receive your personal license key from us.
Send license key and machine code to server
Server checks if license is valid and returns response code, if it is valid
Copy the response code from the log and enter it in the config file to the
ResponseKey
parameter. Save the config file.This allows FusionHub to run on this specific machine without reconnecting to the internet. One license unit will be subtracted from your license account. Please ask us for assitance if you’d like to move your license.
If your default configuration file config.json
doesn't contain it already, add the LicenseInfo
block as shown below. Enter your personal key you received from us as LicenseKey.
{
...
"LicenseInfo": {
"LicenseKey": "EKKCO-GZYLT-NJKET-SASDC",
"ResponseKey": ""
}
...
}
BASE Filter Configuration
FusionHub BASE combines data from an outside-in tracking system with inertial measurements by an inertial measurement unit (IMU). The BASE filter integrates the angular velocity measured by the IMU’s gyroscope and corrects it by the pose of the optical target that is determined with the optical tracking system. This references the calculated pose to the coordinate system of the optical tracking system and avoids drift while maintaining the high frequency and responsiveness of the gyroscope data. The diagram below shows an overview of a BASE filter system.
The position output of the BASE filter is transferred directly from the optical measurements without modification. Pose prediction and interpolation of position measurements by accelerometer integration are under development.
Setup
Setup your optical tracking system. Attach the IMU to the optical target or attach both to the same rigid object eg. an HMD. Initialize the optical tracking body in your motion capture software and note the object ID.
Connect your IMU to the computer running FusionHub. Make sure your computer can connect to the IMU and read data by using LpmsControl 2. Make sure to disconnect from LpmsControl before running FusionHub.
Modify
config.json
to contain the correct information for your IMU and optical tracking system. See below how to configure the blocks in the configuration file. The configuration file can also be modified through the FusionHub GUI as shown further below.
Operation
If all components are connected and the configuration file is valid, FusionHub should work right away after starting the application. The console output shows a log of the initialization of the various components. Note that you can log the output from FusionHub to a file by adding
"record": {
"filename": "log.a",
"format": "json"
}
to the sink
section of config.json
.
After starting and connecting the GUI the Auto Calibration
section of Fusion Config
should show increasing numbers for nImu
(number of recorded IMU samples) and nOptical
(number of recorded optical samples).
Calibration
There are two calibration steps that are required to operate the BASE filter:
Gyroscope Autocalibration
Gyroscope sensors have a built-in measurement bias that changes over time and is temperature-dependent. Good, permanent temperature calibration of MEMS gyroscopes is hard to achieve, therefore FusionHub offers the possibility to run-time calibrate this offset. This calibration is semi-automatic.
The measurement bias of the gyroscope attached to the tracked object is calculated as an average of the data acquired over a certain time interval. Requirement for this sampling to happen is for the object to be in a non-moving / static state. The state of the object is determined by input data from the optical tracking. So once the optical tracking system (eg. ART DTrack) reports the optical target to be static, gyroscope data will be sampled, averaged and a new bias compensation vector calculated.
The result of the autocalibration is saved in autocalibValue.json
. When starting FusionHub for the first time, this offset is set to (0, 0, 0)
. Make sure to place the target, with the IMU attached, within the tracking volume and keep it static eg. by putting it on the floor.
IMU-Optical Intercalibration
The IMU-optical intercalibration calibrates the orientation difference between IMU and optical tracking body. When setting up a new system or after modifying the optical target a (re-)calibration is needed. The calibration is started by running FusionHub with the runIntercalibration
option set to true
.
Rotate the target with the IMU attached slowly within the tracking volume. You can monitor the status of the intercalibration in the Intercalibration section on the Fusion Config
page of the GUI. After around 50 sampled poses the intercalibration should be finished and the GUI should show the resulting calibration quaternion.
Click Apply Intercalibration Result
to automatically insert the result into the configuration file. Click Set
and Save
at the bottom of the editor to save the result and restart FusionHub.
Check the 3D View
page to confirm if the intercalibration result is correct. The red and white cube should overlap alpmost exactly at all times when you rotate your object inside the tracking volume. Note that after a restart it might take a few seconds for optical and fused pose to converge.
IMU-Optical Fusion Filter
Example Configuration
Real-time IMU-optical fusion with LPMS-IG1 and ART Dtrack: imuOpticalFusion.json
Configuration Block
Node name: fusion
"fusion": {
"type": "ImuOpticalFusion",
"settings": {
"echoFusedPose": false,
"echoOpticalPose": true,
"runIntercalibration": true,
"Autocalibration": {
"minAgeS": 60.0,
"nSamplesForAutocalibration": 1500,
"nSamplesForSteady": 256,
"noiseRmsLimit": 0.02,
"steadyThresholdAverage": 0.2,
"steadyThresholdRms": 1.0
},
"MotionDetection": {
"omegaLimit": 2.0,
"positionSampleInterval": 1000,
"rotationFilterAlpha": 0.9,
"timeToUnknown": 500
},
"SensorFusion": {
"alignment": {
"w": 1.0,
"x": 0.0,
"y": 0.0,
"z": 0.0
},
"orientationWeight": 0.005,
"tiltCorrection": null,
"yawWeight": 0.01,
"predictionInterval": 0.01,
"sggPointsEachSide": 5,
"sggPolynomialOrder": 5
}
}
}
Parameter name | Description | Default |
---|---|---|
type | Type of sensor fusion. At the moment only default option possible. | ImuOpticalFusion |
echoFusedPose | Print fused pose like it is output | false |
echoOpticalPose | Print optical pose like it is received by fusion | false |
runIntercalibration | Starts intercalibration between IMU and optical target | true |
minAgeS | Minimum time between two autocalibrations | 60.0 |
nSamplesForAutocalibration | Number of samples used by autocalibration | 1500 |
nSamplesForSteady | Number of samples needed below threshold to trigger calibration | 256 |
noiseRmsLimit | Noise limit | 0.02 |
steadyThresholdAverage | Threshold average limit | 0.2 |
steadyThresholdRms | Threshold RMS limit | 1.0 |
omegaLimit | Omega limit | 2.0 |
positionSampleInterval | Interval between two position samples for motion detection | 1000 |
rotationFilterAlpha | Weight for rotation low-pass filter | 0.9 |
timeToUnknown | Interval to autocalibration “unknown” state | 500 |
alignment | Alignment quaternion between IMU and optical target. Insert the result of the intercalibration here. | 1, 0, 0, 0 |
orientationWeight | Amount of correction of angle calculated from gyroscope data by optical measurements (roll, pitch, yaw) | 0.005 |
tiltCorrection | Specify for correcting tilt of angle calculated from gyroscope data by vertical calculated from gravity measurements. This feature is not available yet. | null |
yawWeight | Amount of yaw correction by optical data, if tilt correction is active | 0.01 |
predictionInterval | Time to look into the future for calculation of the output quaternion | 0.01 |
sggPointsEachSide | Smoothing filter points each side | 5 |
sggPolynomialOrder | Smoothing filter polynomial order | 5 |
This filter needs as input:
Optical tracking source
IMU source
This Filter outputs:
fusedPose
Output Data Format
Parameter name | Description | Unit |
---|---|---|
lastDataTime | Unused | s |
orientation | Orientation quaternion | without unit |
position | Unused | m |
timestamp | Time of data acqusition | ns |
Source Options
Optical Tracking Source Options
Advanced Realtime Tracking (ART)
FusionHub works with all ART tracking systems, based on their DTrack tracking software.
Optitrack
FusionHub works with all Optitrack tracking systems based on their Motive tracking software.
VICON
FusionHub consumes VICON’s DataStream protocol. Communication has been tested with their Shogun software.
Antilatency
FusionHub connects directly to Antilatency’s USB or wireless trackers.
IMU Source
FusionHub supports all LP-RESEARCH IMUs.
See a description of how to prepare LPMS-IG1 for operation with FusionHub further below.
LPMS-IG1
LPMS-CURS3
Graphical User Interface
Dashboard
3D Viewer
Sensor Fusion Configuration and Calibration Status
General Settings
MOVE Filter Configuration
FusionHub MOVE adds an additional platform IMU to the BASE configuration. It combines data from both IMUs to calculate poses relative to a moving platform.
The MOVE filter section of FusionHub is still under development. Refer to LPVR-DUO for an implementation of the filter for specific virtual / augmented reality headsets.
FLOW Filter Configuration
FusionHub FLOW combines odometry, GPS and IMU data from a vehicle to calculate high-accuracy and low-latency global localization information. While GPS or RTK-GPS measurements alone provide similar positioning accuracy the output frequency of these systems is relatively low, making them unsuitable for applications where localization information at higher framerates is required, such as positioning objects in an augmented reality environment.
By additionally using odometry (wheel speeds, steering angle etc.) information, the localization data from the GPS measurements is interpolated to achieve framerates limited only by IMU and odometry sampling speeds.
The FLOW filter has two operation modes with different configuration blocks in config.json
and different output formats. The two modes are:
Low-dynamics filter (LD)
High-dynamics filter (HD)
The diagram below shows an overview of a simple FLOW filter setup.
Installation of Hardware Components
Inertial Measurement Unit (IMU)
LPMS-IG1P needs to be installled in the vehicle in a known orientation ideally with the coordinate axes of the IMU arranged in parallel to the vehicle coordinate system. As vehicle reference frame we are using the VW coordinate system as shown in the image below. Connect the USB connector of LPMS-IG1P to the host computer. If needed an active or passive USB extension can be used. Make sure to check data integrity with the LpmsControl 2 data acquisition tool, we have noticed communication issues with some passive USB extensions.
Global Positioning System (GPS)
The GPS receiver is integrated with the LPMS-IG1P sensor. Connect the antenna cable and place the GPS antenna on top of the vehicle.
Alternatively, a standalone RTK gps module can be used as a gps input source as well.
CAN Bus Connection
FusionHub can be connected to the vehicle CAN bus by using one of the following CAN bus interfaces:
Low-dynamics Filter (Odometry + GPS + (some) IMU)
Configuration Block
Node name: vehicularFusion
Parameter name | Description | Default |
---|---|---|
echoFusedPose | fusedVehiclePose output is printed to command line | false |
endpoint | Output port for the fusion result | 8801 |
fitModel | Model to use for fusion. At the moment only | SimpleCarModel |
driveModel | Model used to calculate the car trajectory from CAN bus data. If the steering wheel data and steering model are provided, | Differential |
velError | Velocity error for Kalman filter. Keep default value. | 0.277777778 |
omegaError | Omega error for Kalman filter. Keep default value. | 0.5 |
measurementError | Measurement error for Kalman filter. Keep default value. | 0.1 |
smoothFit | Enable this option to prevent filter output from jumping between odometry data and GPS measurement. Keep enabled. | true |
useImuTurnRate | If enabled the IMU turn rate is used instead of the wheel velocity based turn rate. Recommended. | false |
imuTurnRateAxis | The IMU axis to use for the Turn rate if | 1, 0, 0 |
This filter needs as input:
LPMS-IG1P data source for IMU and GPS data
Parameter name | Description | Default |
---|---|---|
type | Type of GPS receiver. Currently only | DualRTK |
name | The name of the LPMS-IG1P sensor used in this setup. This parameter is optional. If FusionHub is operated at the same time with LPVR-DUO, we recommend specifying the sensor name. Look up the sensor name in LpmsControl 2. | n/a |
autodetectType | Type of sensor to be autodetcted | ig1p |
rtcm | Set to true if RTCM input is to be received eg. from an NTRIP source. | false |
imuEndpoint | Output endpoint of IMU data. This parameter is optional. | tcp://*:8802 |
Alternatively for case with separate IMU and RTK GPS sources (with NTRIP Caster for RTK correction)
RTCM Source
Parameter name | Description | Default |
---|---|---|
type | Type of RTCM correction data source. Currently only | NTRIP |
host | NTrip caster host. | 192.168.1.1 |
port | NTrip caster port. | 2101 |
mountpoint | NTrip mountpoint or stream to receive rtcm correction data. |
|
user | NTrip caster username. |
|
password | NTrip caster password. |
|
userAgent | Name of user agent when connecting to NTrip caster. | LPVR-POS |
initialLatitude | Latitude to forward to Ntrip caster on first connect. | 0.0 |
initialLongitude | Longitude to forward to Ntrip caster on first connect. | 0.0 |
forwardGnss | Set true if gnss data from gnss source is to be forwarded to NTRIP caster. This is useful if Ntrip caster offers dynamic switching of RTCM correction data based on forwarded location. | false |
GNSS Source
Parameter name | Description | Default |
---|---|---|
type | Data output format for gnss data source. Currently only | NMEA |
port | Serial port number for gnss source. |
|
baudrate | Serial port baudrate to connect to gnss source. |
|
rtcm | Set true to enable RTCM correction data forwarding from RTCM source to gnss module. | false |
CAN bus and vehicle decoder source
Parameter name | Description | Default |
---|---|---|
type | Type of vehicle. Currently only | Automotive |
vehicleStateEndpoint | Endpoint for vehicle state output | tcp://*:8999 |
canInterface | CAN interface used for readin odometry data. Allowed options: | PeakCAN |
vehicleType | Type of vehicle. Currently supported vehicles have to be manually added. Contact us for details. | R56 (BMW Mini) |
This Filter outputs:
fusedVehiclePose
Output Data Format
Parameter name | Description | Unit |
---|---|---|
acceleration | 3D acceleration vector as measured by IMU. Describes the orientation of the vehicle in the vehicle coordinate system. | m/s^2 |
globalPosition | Longitude and latitude in degrees | degrees |
lastDataTime | Unused | s |
position | Position relative to starting point with X pointing North and Y pointing East in the current UTM frame | m |
timestamp | Timestamp of data acquisition | ns |
utmZone | UTM zone | UTM string |
yaw | Globally referenced yaw angle | rad |
Additional Notes
The FusedVehiclePose contains a 3D acceleration vector. The acceleration is defined in the following manner: There's a configuration flag imuToCarRotation which takes a quaternion used to rotate vectors in the IMU frame to the car frame. By default it is the identity quaternion. For the LD model, the measured IMU acceleration is simply rotated by the imuToCarRotation and written to the output.
In the LD filter, pitch and roll has to be derived from the acceleration data based on a model of the stiffness of the chassis. That assumes a flat surface. The HD model offers the full 6-DOF, and we are planning to unify them to have all data available at all times.
As the filter relies heavily on GPS measurements it doesn’t deliver good results indoors. The better GPS reception, the better the resulting output of the filter. The yaw angle of the vehicle is calculated based on several GPS and odometry measurements when the car is moving. Therefore, after starting FusionHub, while the car is static, the filter will not deliver a correct yaw angle. The angle will be adjusted to the correct direction after a few seconds of driving the vehicle.
Example Configuration
Playback and fusion of prerecorded data: gpsImuFusionPlayback.json
Real-time fusion: gpsOdometryFusion.json
High-Dynamics Filter (IMU + GPS)
Node name: gnssImuFusion
Configuration block example (in sinks
section)
Parameter name | Description | Default |
---|---|---|
echoFusedPose |
| false |
endpoint | Output port for the fusion result (more than one endpoint can be used if needed, check the endpoint parameters below). | 8803 |
fitModel | Model to use for fusion. | ModelGnssImu |
accelError | Acceleration error for Kalman filter. Keep default value. | 0.01 |
omegaError | Omega error for Kalman filter. Keep default value. | 0.02 |
measurementError | Measurement error for Kalman filter. Keep default value. | 0.05 |
imuToCarRotation | Orientation quaternion of IMU relative to car frame | 1, 0, 0, 0 |
smoothFit | Enable this option to prevent filter output from jumping between IMU data and GPS measurement. Keep enabled. | true |
singleEndpoint |
| true |
poseEndpoint | Output port for the |
|
globalPoseEndpoint | Output port for the |
|
outputRawGnssData | Publishes raw Gnss data position instead of the fusion output. Useful for debugging. | false |
outputWhenFilterNotReady | Publishes a temporary raw Gnss data output while the filter is initializing. Useful for a minimal check before moving the vehicle. | false |
Setting up the ImuToCarRotation parameter
The used car frame is VW coordinate frame,
The IMU sensor can be mounted in any way but the ImuToCarRotation
quaternion need to be provided to transform the IMU data into VW frame.
Example
If the IMU is mounted like follows,
To match the VW frame, we need a 180° rotation around the z axis (clockwise). Therefore, the rotation matrix would be,
And the orientation quaternion woud be [x, y, z, w] = [ 0, 0, 1, 0 ]
which can be specified in the configuration like below,
This filter needs as input:
LPMS-IG1P data source for IMU and GPS data
Alternatively for case with separate IMU and RTK GPS sources (with NTRIP Caster for RTK correction)
RTCM Source
Parameter name | Description | Default |
---|---|---|
type | Type of RTCM correction data source. Currently only | NTRIP |
host | NTrip caster host. | 192.168.1.1 |
port | NTrip caster port. | 2101 |
mountpoint | NTrip mountpoint or stream to receive rtcm correction data. |
|
user | NTrip caster username. |
|
password | NTrip caster password. |
|
userAgent | Name of user agent when connecting to NTrip caster. | LPVR-POS |
initialLatitude | Latitude to forward to Ntrip caster on first connect. | 0.0 |
initialLongitude | Longitude to forward to Ntrip caster on first connect. | 0.0 |
forwardGnss | Set true if gnss data from gnss source is to be forwarded to NTRIP caster. This is useful if Ntrip caster offers dynamic switching of RTCM correction data based on forwarded location. | false |
GNSS Source
Parameter name | Description | Default |
---|---|---|
type | Data output format for gnss data source. Currently only | NMEA |
port | Serial port number for gnss source. |
|
baudrate | Serial port baudrate to connect to gnss source. |
|
rtcm | Set true to enable RTCM correction data forwarding from RTCM source to gnss module. | false |
CAN bus and vehicle decoder source
This Filter outputs:
fusedVehiclePose (2D pose):
Output equivalent to the LD filter output. Includes position in meters relative to starting point, global position (lon, lat) and heading.fusedPose (3D pose):
relative to starting point, x, y (in meters) + z (height) + 3D orientation quaternionglobalFusedPose:
globally referenced 3D position (longitude, latitude, height) + 3D orientation quaternion in ENU frame
Output data format
FusedVehiclePose
Parameter name | Description |
---|