Info

To make it easier to read, we’ve split the FusionHub documentation into two sub-manuals:
LPVR-AIR Manual - LPVR-AIR Manual

LPVR-POS Manual - LPVR-POS Manual

Please refer to these for the most up-to-date documentation.

Table of Contents

Introduction

FusionHub is a software application that has the purpose of combining a number of sensor data inputs to create a higher level information output. There are 3 basic versions of FusionHub:

...

Code Block

"fusion": {
    "type": "ImuOpticalFusion",
    "settings": {
        "echoFusedPose": false,
        "echoOpticalPose": true,

        "runIntercalibration": true,

        "Autocalibration": {
            "minAgeS": 60.0,
            "nSamplesForAutocalibration": 1500,
            "nSamplesForSteady": 256,
            "noiseRmsLimit": 0.02,
            "steadyThresholdAverage": 0.2,
            "steadyThresholdRms": 1.0
        },

        "MotionDetection": {
            "omegaLimit": 2.0,
            "positionSampleInterval": 1000,
            "rotationFilterAlpha": 0.9,
            "timeToUnknown": 500
        },
        
        "SensorFusion": {
            "alignment": {
                "w": 1.0,
                "x": 0.0,
                "y": 0.0,
                "z": 0.0
            },

            "orientationWeight": 0.005,
            "tiltCorrection": null,
            "yawWeight": 0.01,
            
            "predictionInterval": 0.01,
            "sggPointsEachSide": 5,
            "sggPolynomialOrder": 5
            
        }
    }
}

Parameter name	Description	Default
type	Type of sensor fusion. At the moment only default option possible.	ImuOpticalFusion
echoFusedPose	Print fused pose like it is output	false
echoOpticalPose	Print optical pose like it is received by fusion	false
runIntercalibration	Starts intercalibration between IMU and optical target	true
minAgeS	Minimum time between two autocalibrations	60.0
nSamplesForAutocalibration	Number of samples used by autocalibration	1500
nSamplesForSteady	Number of samples needed below threshold to trigger calibration	256
noiseRmsLimit	Noise limit	0.02
steadyThresholdAverage	Threshold average limit	0.2
steadyThresholdRms	Threshold RMS limit	1.0
omegaLimit	Omega limit	2.0
positionSampleInterval	Interval between two position samples for motion detection	1000
rotationFilterAlpha	Weight for rotation low-pass filter	0.9
timeToUnknown	Interval to autocalibration “unknown” state	500
alignment	Alignment quaternion between IMU and optical target. Insert the result of the intercalibration here.	1, 0, 0, 0
orientationWeight	Amount of correction of angle calculated from gyroscope data by optical measurements (roll, pitch, yaw)	0.005
tiltCorrection	Specify for correcting tilt of angle calculated from gyroscope data by vertical calculated from gravity measurements. This feature is not available yet.	null
yawWeight	Amount of yaw correction by optical data, if tilt correction is active	0.01
predictionInterval	Time to look into the future for calculation of the output quaternion	0.01
sggPointsEachSide	Smoothing filter points each side	5
sggPolynomialOrder	Smoothing filter polynomial order	5

This filter needs as input:

...

Code Block
{ "fusedPose": { "lastDataTime": { "timestamp": 0 }, "orientation": { "w": 1.0, "x": 0.0, "y": 0.0, "z": 0.0 }, "position": { "x": 0.0, "y": 0.0, "z": 0.0 }, "timestamp": { "timestamp": 0 } } }

Parameter name	Description	Unit
lastDataTime	Unused	s
orientation	Orientation quaternion	without unit
position	Unused	m
timestamp	Time of data acqusition	ns

Source Options

Optical Tracking Source Options

...

Code Block

// Sensor fusion config
"vehicularFusion": {
    "echoFusedPose": false,
    "endpoint": "tcp://*:8801",
    "fuser": {
        "fitModel": "SimpleCarModel",
        "driveModel": "Differential",
        "velError": 0.277777778,
        "omegaError": 0.5,
        "measurementError": 0.1,
        "smoothFit": true,
        "useImuTurnRate": true,
        "imuTurnRateAxis": {
            "x": 1,
            "y": 0,
            "z": 0
        }
    }
}

Parameter name	Description	Default
echoFusedPose	fusedVehiclePose output is printed to command line	false
endpoint	Output port for the fusion result	8801
fitModel	Model to use for fusion. At the moment only `SimpleCarModel` is supported.	SimpleCarModel
driveModel	Model used to calculate the car trajectory from CAN bus data. If the steering wheel data and steering model are provided, `Ackermann` model can be used.	Differential
velError	Velocity error for Kalman filter. Keep default value.	0.277777778
omegaError	Omega error for Kalman filter. Keep default value.	0.5
measurementError	Measurement error for Kalman filter. Keep default value.	0.1
smoothFit	Enable this option to prevent filter output from jumping between odometry data and GPS measurement. Keep enabled.	true
useImuTurnRate	If enabled the IMU turn rate is used instead of the wheel velocity based turn rate. Recommended.	false
imuTurnRateAxis	The IMU axis to use for the Turn rate if `useImuTurnRate` is enabled.	1, 0, 0

This filter needs as input:

...

Code Block

"imuP": {
    "type": "DualRtk",
    "settings": {
        "sensor1": {
            // If specification needed, insert first IG1 sensor name here
            //"name": "ig1p232800650050",
            "autodetectType": "ig1p"
        },
        "rtcm": true,
        "imuEndpoint": "tcp://*:8802"
    }
}

Parameter name	Description	Default
type	Type of GPS receiver. Currently only `DualRtk` is allowed.	DualRTK
name	The name of the LPMS-IG1P sensor used in this setup. This parameter is optional. If FusionHub is operated at the same time with LPVR-DUO, we recommend specifying the sensor name. Look up the sensor name in LpmsControl 2.	n/a
autodetectType	Type of sensor to be autodetcted	ig1p
rtcm	Set to true if RTCM input is to be received eg. from an NTRIP source.	false
imuEndpoint	Output endpoint of IMU data. This parameter is optional.	tcp://*:8802

Alternatively for case with separate IMU and RTK GPS sources (with NTRIP Caster for RTK correction)

Code Block

"imu": {
  "type": "OpenZen",
  "settings": {
    "autodetectType": "ig1"
  }
},

"RTCM": {
    "type": "NTRIP",
    "settings": {
        "host": "some-host-name",
        "port": "2101",
        "mountpoint": "some-mount-point",
        "user": "some-user",
        "password": "some-password",
        "userAgent": "LPVR",
        "initialLatitude": 35.65736,
        "initialLongitude": 139.73239,
        "forwardGnss":  true
    }
},

"gnss": {
    "type": "NMEA",
    "settings": {
        "port": "/dev/ttyUSB0",
        "baudrate": 115200,
        "rtcm": true
    }
},

RTCM Source

Parameter name	Description	Default
type	Type of RTCM correction data source. Currently only `NTRIP` is allowed.	NTRIP
host	NTrip caster host.	192.168.1.1
port	NTrip caster port.	2101
mountpoint	NTrip mountpoint or stream to receive rtcm correction data.
user	NTrip caster username.
password	NTrip caster password.
userAgent	Name of user agent when connecting to NTrip caster.	LPVR-POS
initialLatitude	Latitude to forward to Ntrip caster on first connect.	0.0
initialLongitude	Longitude to forward to Ntrip caster on first connect.	0.0
forwardGnss	Set true if gnss data from gnss source is to be forwarded to NTRIP caster. This is useful if Ntrip caster offers dynamic switching of RTCM correction data based on forwarded location.	false

GNSS Source

Parameter name	Description	Default
type	Data output format for gnss data source. Currently only `NMEA` is allowed.	NMEA
port	Serial port number for gnss source.
baudrate	Serial port baudrate to connect to gnss source.
rtcm	Set true to enable RTCM correction data forwarding from RTCM source to gnss module.	false

CAN bus and vehicle decoder source

Code Block
"vehicle": { "type": "Automotive", "vehicleStateEndpoint": "tcp://*:8999", "settings": { "canInterface": "PeakCAN", "vehicleType": "R56" } }

Parameter name	Description	Default
type	Type of vehicle. Currently only `Automotive` allowed.	Automotive
vehicleStateEndpoint	Endpoint for vehicle state output	tcp://*:8999
canInterface	CAN interface used for readin odometry data. Allowed options: - `PeakCAN` - `Vector`	PeakCAN
vehicleType	Type of vehicle. Currently supported vehicles have to be manually added. Contact us for details.	R56 (BMW Mini)

This Filter outputs:

fusedVehiclePose

...

Code Block

{
    "fusedVehiclePose": {
        "acceleration": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "globalPosition": {
            "x": 0.0,
            "y": 0.0
        },
        "lastDataTime": {
            "timestamp": 0
        },
        "position": {
            "x": 0,
            "y": 0
        },
        "timestamp": {
            "timestamp": 0
        },
        "utmZone": "31T",
        "yaw": 0
      }
}

Parameter name	Description	Unit
acceleration	3D acceleration vector as measured by IMU. Describes the orientation of the vehicle in the vehicle coordinate system.	m/s^2
globalPosition	Longitude and latitude in degrees	degrees
lastDataTime	Unused	s
position	Position relative to starting point with X pointing North and Y pointing East in the current UTM frame	m
timestamp	Timestamp of data acquisition	ns
utmZone	UTM zone	UTM string
yaw	Globally referenced yaw angle	rad

Additional Notes

The FusedVehiclePose contains a 3D acceleration vector. The acceleration is defined in the following manner: There's a configuration flag imuToCarRotation which takes a quaternion used to rotate vectors in the IMU frame to the car frame. By default it is the identity quaternion. For the LD model, the measured IMU acceleration is simply rotated by the imuToCarRotation and written to the output.

...

Code Block

"gnssImuFusion": {
    "echoFusedPose": false,
    "endpoint": "tcp://*:8803",
    "fuser": {
        "fitModel": "ModelGnssImu",
        "accelError": 0.01,
        "omegaError": 0.02,
        "measurementError": 0.05,
        "imuToCarRotation": {
            "w": 1,
            "x": 0,
            "y": 0,
            "z": 0
        }
    }
}

Parameter name	Description	Default
echoFusedPose	`fusedVehiclePose` output is printed to command line	false
endpoint	Output port for the fusion result

8801

(more than one endpoint can be used if needed, check the endpoint parameters below).	8803
fitModel	Model to use for fusion.	ModelGnssImu
accelError	Acceleration error for Kalman filter. Keep default value.	0.01
omegaError	Omega error for Kalman filter. Keep default value.	0.02
measurementError	Measurement error for Kalman filter. Keep default value.	0.05
imuToCarRotation	Orientation quaternion of IMU relative to car frame	1, 0, 0, 0
smoothFit	Enable this option to prevent filter output from jumping between IMU data and GPS measurement. Keep enabled.	true

Setting up the ImuToCarRotation parameter

...

singleEndpoint	If enabled, the different fusion output messages will be published to the same port `"endpoint"` If disabled, the `"endpoint"` parameter above, is the output port for the `FusedVehiclePose` msg type only.	true
poseEndpoint	Output port for the `FusedPose` msg type.	`8804` if `singleEndpoint=false`
globalPoseEndpoint	Output port for the `GlobalFusedPose` msg type.	`8805` if `singleEndpoint=false`
outputRawGnssData	Publishes raw Gnss data position instead of the fusion output. Useful for debugging.	false
outputWhenFilterNotReady	Publishes a temporary raw Gnss data output while the filter is initializing. Useful for a minimal check before moving the vehicle.	false

Setting up the ImuToCarRotation parameter

The used car frame is VW coordinate frame,

Code Block
VW frame x: back y: right z: up

...

Code Block

"imu": {
  "type": "OpenZen",
  "settings": {
    "autodetectType": "ig1"
  }
},

"RTCM": {
    "type": "NTRIP",
    "settings": {
        "host": "some-host-name",
        "port": "2101",
        "mountpoint": "some-mount-point",
        "user": "some-user",
        "password": "some-password",
        "userAgent": "LPVR",
        "initialLatitude": 35.65736,
        "initialLongitude": 139.73239,
        "forwardGnss":  true
    }
},

"gnss": {
    "type": "NMEA",
    "settings": {
        "port": "/dev/ttyUSB0",
        "baudrate": 115200,
        "rtcm": true
    }
},

RTCM Source

Parameter name	Description	Default
type	Type of RTCM correction data source. Currently only `NTRIP` is allowed.	NTRIP
host	NTrip caster host.	192.168.1.1
port	NTrip caster port.	2101
mountpoint	NTrip mountpoint or stream to receive rtcm correction data.
user	NTrip caster username.
password	NTrip caster password.
userAgent	Name of user agent when connecting to NTrip caster.	LPVR-POS
initialLatitude	Latitude to forward to Ntrip caster on first connect.	0.0
initialLongitude	Longitude to forward to Ntrip caster on first connect.	0.0
forwardGnss	Set true if gnss data from gnss source is to be forwarded to NTRIP caster. This is useful if Ntrip caster offers dynamic switching of RTCM correction data based on forwarded location.	false

GNSS Source

Parameter name	Description	Default
type	Data output format for gnss data source. Currently only `NMEA` is allowed.	NMEA
port	Serial port number for gnss source.
baudrate	Serial port baudrate to connect to gnss source.
rtcm	Set true to enable RTCM correction data forwarding from RTCM source to gnss module.	false

CAN bus and vehicle decoder source

...

Code Block

{
    "fusedVehiclePose": {
        "acceleration": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "globalPosition": {
            "x": 0.0,
            "y": 0.0
        },
        "lastDataTime": {
            "timestamp": 0
        },
        "position": {
            "x": 0.0,
            "y": 0.0
        },
        "timestamp": {
            "timestamp": 0
        },
        "utmZone": "31T",
        "yaw": 0.0
    }
}

Parameter name	Description	Unit
acceleration	3D acceleration vector as measured by IMU. Describes the orientation of the vehicle.	m/s^2
globalPosition	Longitude and latitude in degrees	degrees
lastDataTime	Unused	s
position	Position within UTM zone	m
timestamp	Timestamp of data acquisition	ns
utmZone	UTM zone	UTM string
yaw	Globally referenced yaw angle	rad

FusedPose

Code Block
{ "fusedPose": { "lastDataTime": { "timestamp": 0 }, "orientation": { "w": 1.0, "x": 0.0, "y": 0.0, "z": 0.0 }, "position": { "x": 0.0, "y": 0.0, "z": 0.0 }, "timestamp": { "timestamp": 0 } } }

Parameter name	Description	Unit
lastDataTime	Unused	s
orientation	Orientation quaternion in ENU coordinate frame	without unit
position	X, y position + height	m
timestamp	Time of data acqusition	ns

GlobalFusedPose

Code Block
{ "globalFusedPose" { "orientation": { "w": 1.0, "x": 0.0, "y": 0.0, "z": 0.0 }, "timestamp": { "timestamp": 0 }, "position": { "longitude": 0.0, "latitude": 0.0, "height": 0.0 } }

Parameter name	Description	Unit
orientation	Orientation quaternion	without unit
position	Longitude, latitude, height	deg, deg, m
timestamp	Time of data acqusition	ns

Information regarding the ENU coordinate system is here: https://en.wikipedia.org/wiki/Local_tangent_plane_coordinates

...

Code Block
"sources": { "filereader": { "filename": "sampleDriveData.json", "playbackInterval": 0.001 } },

Parameter name	Description	Unit
filename	Name of the file to be played back	n/a
playbackInterval	Time interval between each line of the playback file	s

Replay Node [DEPRECATED, switch over to ReplayExecutable]

...

Code Block
"sources": { "replay": { "filename": "./MiniRide10.json", "replaySpeed": 1, // replay speed is adjustable "readMultipleLines": 10 // disk reader would read in multiple lines every time. } }

Key	Description	Type	Example value
filepath	Path to read in file	String	“log.json”
replaySpeed	Speed to the actual recording	Double	1
readMultipleLines	Number of lines to read each time	Integer	10

Replay Executable

This is a separate executable that can be built from FusionHub project. In Visual Studio build target dropdown there will be an option to build ReplayExecutable.exe.

...

Code Block
ReplayExecutable.exe -r <path/to/file.json> [--replay-speed 1] [--queue-size 100] [--echo-data] [--verbose]

Key	Description	Type	Example value
-r	Path to read in file	String	“log.json”
--replay-speed	Speed to the actual recording	Double	1
--queue-size	The size of queue that file reader would stop pushing new data to the replay queue. Increase this value when you see lots of data is published at the same time when running with `--verbose`	Integer	100
--echo-data	Listen to the publishing endpoint and display the replayed data	N/A	N/A
--verbose	Print the debugging information, i.e., the timestamp a packet is added to the replay queue, replayed from the replay queue, and discarded from the replay queue.	N/A	N/A

A normal FusionHub program can then receive the file data by having an endpoints source defined in the configuration file:

...

Code Block
"sources": { "filereader": { "filename": "driveData.json", "playbackInterval": 0.001 } },

Parameter name	Description	Unit
filename	Name of the file to be played back	n/a
playbackInterval	Time interval between each line of the playback file	s

Communication with External Applications

...

The WebSocket server can be accessed via 19358 port on the machine hosting the FusionHub service. To accelerate development download the Simple WebSocket Client Chrome plugin. This allows you to manually enter API commands and check the replies from the server.

...

Endpoint

...

Sample Requests

...

Sample Response / Description

...

getConfig

...

Code Block

language	json

{
  "command": "getConfig"
}

...

Get in memory configurations.

...

getSavedConfig

...

Code Block

language	json

{
  "command": "getSavedConfig"
}

...

Get on disk configurations.

...

saveConfig

...

Code Block

language	json

{
  "command": "saveConfig"
}

...

Save the in-memory configurations to the disk.

...

development download the Simple WebSocket Client Chrome plugin. This allows you to manually enter API commands and check the replies from the server.

Command

Sample Requests

Sample Response / Description

getConfig

Code Block

language	json

{
  "command": "getConfig"
}

Gets in memory configurations.

getSavedConfig

Code Block

language	json

{
  "command": "getSavedConfig"
}

Gets on disk configurations.

saveConfig

Code Block

language	json

{
  "command": "saveConfig"
}

Saves the in-memory configurations to the disk.

setConfig

Code Block

language	json

{
  "command": "setConfig",
  "data": {
    "sources": {
      "optical": {
        "settings": {
          "port": 5005
        }
      }
    }
  }
}

Updates in-memory configurations. This API creates new key-value pairs, or updates the existing values. It doesn't save configurations to disk.

Note that in "data" you just need to specify the path to the json key to update:the example on the right would change the port to 5005 while everything else is left unchanged.

setConfigJsonPath

(available for JVC branch)

Code Block

language	json

{
  "command": "

setConfig

setConfigJsonPath",
  "data":

{

"sources":

 {
      "

optical

path":

{

 "/sinks/prediction/0/settings

": {

/predictionIntervalMs",
      "

port

value":

5005

40
    }

]
}

} } }

Update in-memory configurations. This api create new key-value pairs, or update the existing values.

It does not save configurations to the disk.

Note that in "data" you just need to specify the path to the json key to update: the exmaple on the right would change the port to 5005 while everything else is left unchanged.

overwriteConfig

Updates the in-memory configuration given JSON path, and new value.

For more info about JSON path, please refer JSON Pointer - JSON for Modern C++ (nlohmann.me).

overwriteConfig

(disabled in JVC branch)

Code Block

language	json

{
  "command": "overwriteConfig",
  "data": {
    "settings": { ... },
    "sources": { ... },
    "sinks": { ... }
  }
}

Overwrite

Overwrites the in-memory configurations. This is suitable when user want to remove a key from the configuration.

getIntercalibrationStatus

Code Block

language	json

{
  "command": "getIntercalibrationStatus"
}

Get

Gets the current intercalibration status. Useful for refetching current status when the frontnend accidentally disconnects.

applyIntercalibrationResults

Code Block

language	json

{
  "command": "applyIntercalibrationResults"
}

Apply

Applies the current intercalibration quaternion to the in-memory copy of config. This does

NOT

not save to disk.

restartBackend

Code Block

language	json

{
  "command": "restartBackend"
}

Restart

Restarts the backend. Internally the while loop reset the DataBlock, causing all sources and sinks to be freed from memory, and instantiate them again.

startRecording

Code Block

language	json

{
    "command": "startRecording",
    "data": {
        "endpoints": ["inproc://imu_data_source_1"],
        "format": "json",
        "filename": "subaruDrive"
    }
}

Listen

Listens to data published to endpoints, and write to a file of YYYYMMDD-HHMMSS-{comment}.{format}

stopRecording

Code Block
{ "command": "stopRecording", }

Stop

Stops the current recording.

listRecording

Code Block

language	json

{
    "command": "listRecording",
}

List

Lists the recorded filenames since the FusionHub booted up.

getVersion

Code Block

language	json

{
  "command": "getVersion"
}

Code Block

language	json

{
  "command": "getVersion",
  "status": "ok",
  "data": {
    "version": "1.0.0"
  }
}

Sending FusionHub Data to External Applications via the ZeroMQ Interface

...

The following applications need to be started on the head mounted display and the host computer. They should all the included in the installation package that you received from us. We will discuss the order of starting these applications and what their status output should be below.

On the headset:

Application

Purpose

Name

FusionHub server

Receives IMU data from ALVR client
Receives optical tracking data
Sends sensor fusion result to ALVR client

FusionHub-v1.2-Launcher.apk

ALVR client

Receives image data from ALVR server
Connects to FusionHub server
Sends pose information to ALVR server

FusionHub-v1.2-ALVR-Client.apk

On the host computer:

Application	Purpose	Name
FusionHub GUI	Connects to FusionHub server on HMD Configures FusionHub Authenticates system	`lp-fusionhub-dashboard.exe`
ALVR server	Receives pose information from client Sends pose information to SteamVR Sends image data to ALVR client Receives image information from SteamVR	`ALVR Launcher.exe`

Authentification

FusionHub authentificates itself via the GUI client application. In order to run the client application make sure to insert the LPVR USB dongle into the host computer. After the GUI client is connected to FusionHub on the HMD, FusionHub will start streaming pose data to ALVR.

...

Start FusionHub on the HMD. A window showing the FusionHub console output should open.
Start the FusionHub GUI client on the host computer
Connect the GUI client to FusionHub on the HMD. Make sure HMD and host are in the same subnet. Enter the the correct IP of the HMD in the client before pressing connect.
Adjust parameter blocks as needed. Refer to the description of FusionHub BASE for configuration options. Note the following input and output ports that are hard-coded in the ALVR FusionHub API layer. These are already correctly set in the default configuration file installed with the FusionHub APK, so usually there is no need to change them.

Endpoint	Direction	Purpose
`tcp://*:8799`	Output	Fused pose data
`tcp://localhost:8898`	Input	IMU data

If it’s not running yet make sure to start and configure your optical tracking system. Once optical data is streamed to FusionHub, the nOptical counter in the GUI should be increasing.

...

The optical system is the tracking reference, its pose is what is received by the visualization backend. The orientation of the IMU sensor is calibrated relative to the optical markers on the HMD. Therefore it is important to set up the tracking body or rigid body in the optical tracking software (DTrack, Motive etc.) in a way that its axes align with the optical axes of the head mounted display.

Image Modified

...

We will add to this section soon. In the meantime refer to this page for ART setups and this page for OptiTrack setups from the LPVR documentation.

...

Version	Old Version 48	New Version Current
Changes made by	Klaus Petersen	Chong Yong Kin (Unlicensed)
Saved on	Sept 14, 2023	Apr 08, 2024

Page Comparison

Versions Compared

Key

Introduction

Source Options

Optical Tracking Source Options

Additional Notes

Setting up the ImuToCarRotation parameter

Setting up the ImuToCarRotation parameter

FusedPose

GlobalFusedPose

Replay Node [DEPRECATED, switch over to ReplayExecutable]

Replay Executable

Communication with External Applications

Sending FusionHub Data to External Applications via the ZeroMQ Interface

Authentification