Introduction

FusionHub is a software application that has the purpose of combining a number of sensor data inputs to create a higher level information output. LPVR-POS is based on FusionHub and combines odometry, GPS and IMU data from a vehicle to calculate high-accuracy and low-latency global localization information. Typical applications: Automobile localization, robot localization

The diagram below shows the general structure of FusionHub. Sources and sinks are connected by a filter unit. The sensor fusion functionality is contained in this filter unit. The filter parameters as well as the parameters of input and output blocks can be configured via a configuration script or the graphical user interface.

The graphical user interface is detached from the main FusionHub application and both applications can therefore run on separate computers. This provides flexibility for running FusionHub on devices with limited monitoring capabilities like a head mounted display.

General

Starting FusionHub

FusionHub consists of two components:

• The main application

• A graphical user interface application

Insert the security dongle into a USB port of your computer.

The main FusionHub application is started by running FusionHub.exe. No specific installation is needed, the application can be run directly out of its deployment directory. It is a command line application that uses the file config.json for its configuration. We will explan the contents and options of the configuration file further below.

Please install the graphical user interface by running lp-fusionhub-dashboard_0.1.0_x64_en-US.msi. It installs lp-fusionhub-dashboard in your start menu, launch the application from there. Press the Connect button after starting FusionHub.exe to connect client and server. In case you are running FusionHub on a separate machine make sure to enter the correct IP address.

The screenshot below shows the connection elements of the GUI.

Licensing

FusionHub has two options for license protection:

Hardware dongle

License authentication using a hardware dongle; This is especially interesting for air-gapped installations that are not connected to the internet. As long as the dongle is inserted into a USB slot of the host system, FusionHub will run. Please note that for the Android (Quest 2 HMD) version of FusionHub, the GUI running on the streaming host is dongle protected, see more detailed information in the specific manual chapter.

Online license

License authentication using a software, online license; This makes sense for systems that are connected to the internet at least during the initial installation of FusionHub. The software checks its license status with our license server with following sequence:

1. Enter license key in configuration file. You receive your personal license key from us.

2. Send license key and machine code to server

3. Server checks if license is valid and returns response code, if it is valid

4. Copy the response code from the log and enter it in the config file to the ResponseKey parameter. Save the config file.

5. This allows FusionHub to run on this specific machine without reconnecting to the internet. One license unit will be subtracted from your license account. Please ask us for assitance if you’d like to move your license.

If your default configuration file config.json doesn't contain it already, add the LicenseInfo block as shown below. Enter your personal key you received from us as LicenseKey.

{
  ...

  "LicenseInfo": {
    "LicenseKey": "EKKCO-GZYLT-NJKET-SASDC",
    "ResponseKey": ""
  }
  
  ...
}

LPVR-POS Filter Configuration

LPVR-POS combines odometry, GPS and IMU data from a vehicle to calculate high-accuracy and low-latency global localization information. While GPS or RTK-GPS measurements alone provide similar positioning accuracy the output frequency of these systems is relatively low, making them unsuitable for applications where localization information at higher framerates is required, such as positioning objects in an augmented reality environment.

By additionally using odometry (wheel speeds, steering angle etc.) information, the localization data from the GPS measurements is interpolated to achieve framerates limited only by IMU and odometry sampling speeds.

The LPVR-POS filter has two operation modes with different configuration blocks in config.json and different output formats. The two modes are:

• Low-dynamics filter (LD)

• High-dynamics filter (HD)

The diagram below shows an overview of a simple LPVR-POS filter setup.

Installation of Hardware Components

Inertial Measurement Unit (IMU)

LPMS-IG1P needs to be installled in the vehicle in a known orientation ideally with the coordinate axes of the IMU arranged in parallel to the vehicle coordinate system. As vehicle reference frame we are using the VW coordinate system shown in the image below. Connect the USB connector of LPMS-IG1P to the host computer. If needed an active or passive USB extension can be used. Make sure to check data integrity with the LpmsControl 2 data acquisition tool, we have noticed communication issues with some passive USB extensions.

VW frame
x: back
y: right
z: up

Global Positioning System (GPS)

The GPS receiver is integrated with the LPMS-IG1P sensor. Connect the antenna cable and place the GPS antenna on top of the vehicle.

Alternatively, a standalone RTK gps module can be used as a gps input source as well.

CAN Bus Connection

FusionHub can be connected to the vehicle CAN bus by using one of the following CAN bus interfaces:

• PCAN USB interface

• Vector CAN USB interface

Low-dynamics Filter (Odometry + GPS + (some) IMU)

The low dynamics (LD) filter combines odometry, GPS and IMU data to calculate the global position of a vehicle. This filter was initially concipated wokring only with odometry and GPS data to calculate the 2D position and yaw angle of a vehicle. It is therefore suitable for simple tracking scenarios. For more complicated, for example augmented reality, applications we recommed using the HD filter as it outputs globally referenced 3D orientation additionally to position. This filter relies heavily on the wheel velocity data of the car published on the car’s CAN bus. Therefore the quality of the output of this filter depends on the accuracy of this information.

Configuration Block

Insert the following configuration block into your config.json file to activate the LD filter. The filter's node name is vehicularFusion.

"vehicularFusion": {
    "echoFusedPose": false,
    "endpoint": "tcp://*:8801",
    "fuser": {
        "fitModel": "SimpleCarModel",
        "driveModel": "Differential",
        "velError": 0.277777778,
        "omegaError": 0.5,
        "measurementError": 0.1,
        "smoothFit": true,
        "useImuTurnRate": true,
        "imuTurnRateAxis": {
            "x": 1,
            "y": 0,
            "z": 0
        }
    }
}

See below a description of the parameter options of the LD filter.

This filter needs the following sources as input. One option is to operate the filter with our with our LPMS-IG1P sensor that contains IMU and a GPS receiver. This allows for stadard GPS absolute position accuracy. Relative accuracy and update rate are higher, based on odometry and IMU data. The other option is to use a separate RTK-GPS unit to get high accuracy RTK-GPS readings. We will look at how to set up an RTK-GPS system for LPVR-POS in a following chapter.

• LPMS-IG1P data source for IMU and GPS data

"imuP": {
    "type": "DualRtk",
    "settings": {
        "sensor1": {
            // If specification needed, insert first IG1 sensor name here
            //"name": "ig1p232800650050",
            "autodetectType": "ig1p"
        },
        "rtcm": true,
        "imuEndpoint": "tcp://*:8802"
    }
}

• Alternatively for case with separate IMU and RTK GPS sources (with NTRIP Caster for RTK correction)

"imu": {
  "type": "OpenZen",
  "settings": {
    "autodetectType": "ig1"
  }
},

"RTCM": {
    "type": "NTRIP",
    "settings": {
        "host": "some-host-name",
        "port": "2101",
        "mountpoint": "some-mount-point",
        "user": "some-user",
        "password": "some-password",
        "userAgent": "LPVR",
        "initialLatitude": 35.65736,
        "initialLongitude": 139.73239,
        "forwardGnss":  true
    }
},

"gnss": {
    "type": "NMEA",
    "settings": {
        "port": "/dev/ttyUSB0",
        "baudrate": 115200,
        "rtcm": true
    }
},

RTCM Source

GNSS Source

• CAN bus and vehicle decoder source

"vehicle": {
    "type": "Automotive",
    "vehicleStateEndpoint": "tcp://*:8999",
    "settings": {
        "canInterface": "PeakCAN",
        "vehicleType": "R56"
    }
}

This Filter outputs:

• fusedVehiclePose

Output Data Format

{
    "fusedVehiclePose": {
        "acceleration": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "globalPosition": {
            "x": 0.0,
            "y": 0.0
        },
        "lastDataTime": {
            "timestamp": 0
        },
        "position": {
            "x": 0,
            "y": 0
        },
        "timestamp": {
            "timestamp": 0
        },
        "utmZone": "31T",
        "yaw": 0
      }
}

Additional Notes

The FusedVehiclePose contains a 3D acceleration vector. The acceleration is defined in the following manner: There's a configuration flag imuToCarRotation which takes a quaternion used to rotate vectors in the IMU frame to the car frame. By default it is the identity quaternion. For the LD model, the measured IMU acceleration is simply rotated by the imuToCarRotation and written to the output.

In the LD filter, pitch and roll has to be derived from the acceleration data based on a model of the stiffness of the chassis. That assumes a flat surface. The HD model offers the full 6-DOF, and we are planning to unify them to have all data available at all times.

As the filter relies heavily on GPS measurements it doesn’t deliver good results indoors. The better GPS reception, the better the resulting output of the filter. The yaw angle of the vehicle is calculated based on several GPS and odometry measurements when the car is moving. Therefore, after starting FusionHub, while the car is static, the filter will not deliver a correct yaw angle. The angle will be adjusted to the correct direction after a few seconds of driving the vehicle.

High-Dynamics Filter (IMU + GPS)

Node name: gnssImuFusion

Configuration block example (in sinks section)

"gnssImuFusion": {
    "echoFusedPose": false,
    "endpoint": "tcp://*:8803",
    "fuser": {
        "fitModel": "ModelGnssImu",
        "accelError": 0.01,
        "omegaError": 0.02,
        "measurementError": 0.05,
        "imuToCarRotation": {
            "w": 1,
            "x": 0,
            "y": 0,
            "z": 0
        }
    }
}

Setting up the ImuToCarRotation parameter

The used car frame is VW coordinate frame,

VW frame
x: back
y: right
z: up

The IMU sensor can be mounted in any way but the ImuToCarRotation quaternion need to be provided to transform the IMU data into VW frame.

Example

If the IMU is mounted like follows,

IMU mounting
x: forward
y: left
z: up

To match the VW frame, we need a 180° rotation around the z axis (clockwise). Therefore, the rotation matrix would be,

[ -1,  0,  0;
   0, -1,  0;
   0,  0,  1 ] 

And the orientation quaternion woud be [x, y, z, w] = [ 0, 0, 1, 0 ] which can be specified in the configuration like below,

"imuToCarRotation": {
            "w": 0,
            "x": 0,
            "y": 0,
            "z": 1
        }  

This filter needs as input:

• LPMS-IG1P data source for IMU and GPS data

"imuP": {
    "type": "DualRtk",
    "settings": {
        "sensor1": {
            // If specification needed, insert first IG1 sensor name here
            //"name": "ig1p232800650050",
            "autodetectType": "ig1p"
        },
        "rtcm": true,
        "imuEndpoint": "tcp://*:8802"
    }
}

• Alternatively for case with separate IMU and RTK GPS sources (with NTRIP Caster for RTK correction)

"imu": {
  "type": "OpenZen",
  "settings": {
    "autodetectType": "ig1"
  }
},

"RTCM": {
    "type": "NTRIP",
    "settings": {
        "host": "some-host-name",
        "port": "2101",
        "mountpoint": "some-mount-point",
        "user": "some-user",
        "password": "some-password",
        "userAgent": "LPVR",
        "initialLatitude": 35.65736,
        "initialLongitude": 139.73239,
        "forwardGnss":  true
    }
},

"gnss": {
    "type": "NMEA",
    "settings": {
        "port": "/dev/ttyUSB0",
        "baudrate": 115200,
        "rtcm": true
    }
},

RTCM Source

GNSS Source

CAN bus and vehicle decoder source

"vehicle": {
    "type": "Automotive",
    "vehicleStateEndpoint": "tcp://*:8999",
    "settings": {
        "canInterface": "PeakCAN",
        "vehicleType": "R56"
    }
}

This Filter outputs:

• fusedVehiclePose (2D pose): Output equivalent to the LD filter output. Includes position in meters relative to starting point, global position (lon, lat) and heading.

• fusedPose (3D pose): relative to starting point, x, y (in meters) + z (height) + 3D orientation quaternion

• globalFusedPose: globally referenced 3D position (longitude, latitude, height) + 3D orientation quaternion in ENU frame

Output data format

FusedVehiclePose

{
    "fusedVehiclePose": {
        "acceleration": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "globalPosition": {
            "x": 0.0,
            "y": 0.0
        },
        "lastDataTime": {
            "timestamp": 0
        },
        "position": {
            "x": 0.0,
            "y": 0.0
        },
        "timestamp": {
            "timestamp": 0
        },
        "utmZone": "31T",
        "yaw": 0.0
    }
}

FusedPose

{
	"fusedPose": {
		"lastDataTime": {
			"timestamp": 0
		},
		"orientation": {
			"w": 1.0,
			"x": 0.0,
			"y": 0.0,
			"z": 0.0
		},
		"position": {
			"x": 0.0,
			"y": 0.0,
			"z": 0.0
		},
		"timestamp": {
			"timestamp": 0
		}
    }
}

GlobalFusedPose

{
    "globalFusedPose" {
        "orientation": {
			"w": 1.0,
			"x": 0.0,
			"y": 0.0,
			"z": 0.0
		},
        "timestamp": {
			"timestamp": 0
		},
        "position": {
            "longitude": 0.0,
            "latitude": 0.0,
            "height": 0.0
		}
	}

Information regarding the ENU coordinate system is here: https://en.wikipedia.org/wiki/Local_tangent_plane_coordinates

Example Configuration

Playback and fusion of prerecorded data: gpsImuFusionPlayback.json

Data Playback [DEPRECATED, switch over to ReplayExecutable]

Data from a log file can played back and forwarded to a fusion filter using the fileReader block. An example of how to use this node we are showing below:

"sources": {
    "filereader": {
        "filename": "sampleDriveData.json",
        "playbackInterval": 0.001
    }
},

Replay Node [DEPRECATED, switch over to ReplayExecutable]

Replay data from disk file.

  "sources": {
    "replay": {
      "filename": "./MiniRide10.json",
      "replaySpeed": 1,          // replay speed is adjustable
      "readMultipleLines": 10    // disk reader would read in multiple lines every time.
    }
  }

Replay Executable

This is a separate executable that can be built from FusionHub project. In Visual Studio build target dropdown there will be an option to build ReplayExecutable.exe.

The replay executable will read in from file, push data to replay queue and send them to the network (tcp://localhost:9921 by default). To run the ReplayExecutable,

ReplayExecutable.exe -r <path/to/file.json> [--replay-speed 1] [--queue-size 100] [--echo-data] [--verbose]

A normal FusionHub program can then receive the file data by having an endpoints source defined in the configuration file:

{
  ...,
  "sources": {
    "endpoints": ["tcp://localhost:9921"]
  }
}

Graphical User Interface

Map View

Data Playback and Recording

Data playback and recording works in the same way for all FusionHub versions. It has been described in the previous chapters, but I’ll add a recap here to give it its dedicated chapter, as it’s a very important feature for data analysis and serialization.

Data Recording

Record node

You can record the output from FusionHub to a file by adding

"record": {
    "filename": "driveData.json",
    "format": "json"
}

to the sink section of config.json.

File Logger

"logger": {
    // "endpoints": ["inproc://file_reader_1"],  // from file reader
    "endpoints": ["tcp://localhost:9921"],  // from Replay executable
    "format": "json",
    "filename": "recTest" // this will become the postfix of the filename
}

Data Playback

Data from a log file can played back and forwarded to a fusion filter using the fileReader block. An example of how to use this node we are showing below:

"sources": {
    "filereader": {
        "filename": "driveData.json",
        "playbackInterval": 0.001
    }
},

Communication with External Applications

WebSocket APIs

Apart from manual editing the config.json configuration script or modifying it through the GUI, FusionHub also offers a WebSocket API for external application to change its configuratuion. In fact the GUI uses this interface to access FusionHub’s settings.

Note that the websocket communication is currently not encrypted, it is not secure. Please take your own precautions to make sure network traffic for the configuration isn’t intercepted in some way. We might add an option for secure communication in future releases.

The WebSocket server can be accessed via 19358 port on the machine hosting the FusionHub service. To accelerate development download the Simple WebSocket Client Chrome plugin. This allows you to manually enter API commands and check the replies from the server.

{
  "command": "getConfig"
}

{
  "command": "getSavedConfig"
}

{
  "command": "saveConfig"
}

{
  "command": "setConfig",
  "data": {
    "sources": {
      "optical": {
        "settings": {
          "port": 5005
        }
      }
    }
  }
}

{
  "command": "overwriteConfig",
  "data": {
    "settings": { ... },
    "sources": { ... },
    "sinks": { ... }
  }
}

{
  "command": "getIntercalibrationStatus"
}

{
  "command": "applyIntercalibrationResults"
}

{
  "command": "restartBackend"
}

{
    "command": "startRecording",
    "data": {
        "endpoints": ["inproc://imu_data_source_1"],
        "format": "json",
        "filename": "subaruDrive"
    }
}

{
    "command": "stopRecording",
}

{
    "command": "listRecording",
}

{
  "command": "getVersion"
}

{
  "command": "getVersion",
  "status": "ok",
  "data": {
    "version": "1.0.0"
  }
}

Sending FusionHub Data to External Applications via the ZeroMQ Interface

FusionHub emits data resulting from the sensor fusion through the local network interface.

Output Ports

The network port that this information is output to can be configured in the JSON parameter file config.json of FusionHub.

Data Format

As low level protocol to emit the output data we use ZeroMQ (publisher / subscriber). The data itself is in JSON format and is encoded as Protocol Buffers. Protocol Buffers are documented here. Message are defined in the Protobuf (.protoc) format as defined in the file stream_data.proto. This file is contained in the installation folder of FusionHub.

Python Resources

Download a Python example that shows how to decode messaged from FusionHub from this repository.

Prerequisites can be installed in your Python 3 environment with this:

pip install zmq
pip install protobuf

Make sure to set the input port in FusionHubPythonExample.py correctly. For example for the Antilatency source definition like below, the port needs to be set to 8899.

"optical": {
    "type": "Antilatency",
    "settings": {
        // Use this for access from an external process eg. ALVR
        "endpoint": "tcp://*:8899",
        
        "environmentLink": "AntilatencyAltEnvironmentHorizontalGrid~AgAEBLhTiT_cRqA-r45jvZqZmT4AAAAAAAAAAACamRk_AQQCAwICAgICAQICAAI",
        "placementLink": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
    }
}

C# Resources

On parsing Protobuf files: https://github.com/5argon/protobuf-unity

How to subscribe to ZeroMQ messages: https://github.com/gench23/unity-zeromq-client and https://tech.uqido.com/2020/09/29/zeromq-in-unity/

VRPN Output

VRPN output is set in the following part in the sinks section of config.json. The device name will be referenced by the plugin for Unreal engine.

"VRPN": {
  "settings": {
    "deviceName": "Fusion Hub"
  }
}	  

Please see below how we achieve data input via VRPN in the Unreal engine. First, install the VRPN LiveLink plugin:

Configure the VRPN source with the correct device and subject name:

Apply the output from the fusion hub to an Unreal object eg. a cine camera actor.

Hardware Preparation

Inertial Measurement Units

General documentation for LPMS IMUs is here.

Switching LPMS-IG1(P) to USBxpress Mode

Note: These instructions work for LPMS-IG1 (IMU only) and LPMS-IG1P (IMU + GPS).

First, download LpmsControl 2 from here and install it.

Connect LPMS-IG1(P) to your computer and start LpmsControl 2.

In LpmsControl 2 select one of the LPMS-IG1(P) sensors and connect to it.

In case the sensor is in VCP (virtual COM port) mode as shown below, click on Convert to switch the sensor to USBxpress mode. This is required for communication with FusionHub.

After converting the sensor to USBxpress mode it should be displayed as such.

The image below shows typical output from LPMS-IG1(P) after connecting.

Close LpmsControl 2 to disconnect from the sensor. You are now ready to use LPMS-IG1(P) in FusionHub.