Versions Compared

Version Old Version 8 New Version 9
Changes made by

Klaus Petersen

Klaus Petersen

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Please install the graphical user interface by running lp-fusionhub-dashboard_0.1.0_x64_en-US.msi. It installs lp-fusionhub-dashboard in your start menu, launch the application from there. Press the Connect button after starting FusionHub.exe to connect client and server. In case you are running FusionHub on a separate machine make sure to enter the correct IP address.

Communication with External Applications

Sending FusionHub Data to External Applications via the ZeroMQ Interface

FusionHub emits data resulting from the sensor fusion through the local network interface.

Output Ports

The network port that this information is output to can be configured in the JSON parameter file config.json of FusionHub.

Data Format

As low level protocol to emit the output data we use ZeroMQ (publisher / subscriber). The data itself is in JSON format and is encoded as Protocol Buffers. Protocol Buffers are documented here. Message are defined in the Protobuf (.protoc) format as defined in the file stream_data.proto. This file is contained in the installation folder of FusionHub.

Python Resources

Download a Python example that shows how to decode messaged from FusionHub from this repository.

Prerequisites can be installed in your Python 3 environment with this:

Code Block
pip install zmq
pip install protobuf

Make sure to set the input port in FusionHubPythonExample.py correctly. For example for the Antilatency source definition like below, the port needs to be set to 8899.

...

BASE Filter Configuration

Optical Tracking Source Options

ART

Code Block
"type": "DTrack",
"settings": {
    "port": 5005,
    "bodyID": 3,
    "endpoint": "inproc://optical_data_source_1"
}

Optitrack

Code Block
"type": "Optitrack",
"settings": {
    "host": "localhost",
    "connectionType": "Multicast",
    "settingsbodyID": {
   444
}

VICON

Code Block
"type": "Vicon",
    // Use this for access from an external process eg. ALVR
        "endpoint"settings": {
    "host": "tcp://*:8899localhost",
    "subject": "VCam"
}

Antilatency

Code Block
"type": "Antilatency",
"settings": {
    // Use  "environmentLink": "AntilatencyAltEnvironmentHorizontalGrid~AgAEBLhTiT_cRqA-r45jvZqZmT4AAAAAAAAAAACamRk_AQQCAwICAgICAQICAAI",
   this for access from an external process eg. ALVR
    //"placementLinkendpoint": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"tcp://*:8899",

   } }

C# Resources

On parsing Protobuf files: https://github.com/5argon/protobuf-unity

How to subscribe to ZeroMQ messages: https://github.com/gench23/unity-zeromq-client and https://tech.uqido.com/2020/09/29/zeromq-in-unity/

VRPN Output

VRPN output is set in the following part in the sinks section of config.json. The device name will be referenced by the plugin for Unreal engine.

Code Block
"VRPN": {
  "settings// Use this for internal access eg. sensor fusion
    "endpoint": "inproc://optical_data_source_1",

    "environmentLink": "AntilatencyAltEnvironmentHorizontalGrid~AgAEBLhTiT_cRqA-r45jvZqZmT4AAAAAAAAAAACamRk_AQQCAwICAgICAQICAAI",
    "placementLink": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
}

FLOW Filter Configuration

The FLOW filter has two operation modes with different configuration blocks in config.json and different output formats. The two modes are:

  • Low-dynamics filter (LD)

  • High-dynamics filter (HD)

Low-dynamics Filter (Odometry + GPS + (some) IMU)

Configuration block example (in sinks section)

Node name: vehicularFusion

Code Block
// Sensor fusion config
"vehicularFusion": {
    "deviceNameechoFusedPose": "Fusion Hub"
  }
}

Please see below how we achieve data input via VRPN in the Unreal engine. First, install the VRPN LiveLink plugin:

...

Configure the VRPN source with the correct device and subject name:

...

Apply the output from the fusion hub to an Unreal object eg. a cine camera actor.

...

FLOW Filter Configuration

The FLOW filter has two operation modes with different configuration blocks in config.json and different output formats. The two modes are:

  • Low-dynamics filter (LD)

  • High-dynamics filter (HD)

Low-dynamics Filter (Odometry + GPS + (some) IMU)

Configuration block example (in sinks section)

Node name: vehicularFusion

Code Block
// Sensor fusion config
"vehicularFusion": {
    "echoFusedPose": false,
    "endpoint": "tcp://*:8801",
    "fuser": {
        "fitModel": "SimpleCarModel",
        "driveModel": "Differential",
        "velError": 0.277777778,
        "omegaError": 0.5,
        "measurementError": 0.1,
        "smoothFit": true
    }
}

...

Parameter name

...

Description

...

Default

...

echoFusedPose

...

fusedVehiclePose output is printed to command line

...

false

...

endpoint

...

Output port for the fusion result

...

8801

...

fitModel

...

Model to use for fusion. At the moment only SimpleCarModel is supported.

...

SimpleCarModel

...

driveModel

...

Model to use to calculate car trajectory from CAN bus data. At the moment only Differential is supported.

...

Differential

...

velError

...

Velocity error for Kalman filter. Keep default value.

...

0.277777778

...

omegaError

...

Omega error for Kalman filter. Keep default value.

...

0.5

...

measurementError

...

Measurement error for Kalman filter. Keep default value.

...

0.1

...

smoothFit

...

Enable this option to prevent filter output from jumping between odometry data and GPS measurement. Keep enabled.

...

true

This filter needs as input:

  • LPMS-IG1P data source for IMU and GPS data

Code Block
"imuP": {
    "type": "DualRtk",
    "settings": {false,
    "endpoint": "tcp://*:8801",
    "fuser": {
        "fitModel": "SimpleCarModel",
        "driveModel": "Differential",
        "velError": 0.277777778,
        "omegaError": 0.5,
        "measurementError": 0.1,
        "smoothFit": true
    }
}

Parameter name

Description

Default

echoFusedPose

fusedVehiclePose output is printed to command line

false

endpoint

Output port for the fusion result

8801

fitModel

Model to use for fusion. At the moment only SimpleCarModel is supported.

SimpleCarModel

driveModel

Model to use to calculate car trajectory from CAN bus data. At the moment only Differential is supported.

Differential

velError

Velocity error for Kalman filter. Keep default value.

0.277777778

omegaError

Omega error for Kalman filter. Keep default value.

0.5

measurementError

Measurement error for Kalman filter. Keep default value.

0.1

smoothFit

Enable this option to prevent filter output from jumping between odometry data and GPS measurement. Keep enabled.

true

This filter needs as input:

  • LPMS-IG1P data source for IMU and GPS data

Code Block
"imuP": {
    "type": "DualRtk",
    "settings": {
        "sensor1": {
            // If specification needed, insert first IG1 sensor name here
            //"name": "ig1p232800650050",
            "autodetectType": "ig1p"
        "sensor1": {},
        "rtcm": true,
   // If specification needed, insert first IG1 sensor name here
   "imuEndpoint": "tcp://*:8802"
    }
}

  • CAN bus and vehicle decoder source

Code Block
"vehicle": {
        //"name": "ig1p232800650050",
  "type": "Automotive",
         "autodetectTypevehicleStateEndpoint": "ig1p"tcp://*:8999",
    "settings":  {
 },         "rtcmcanInterface": true"PeakCAN",
        "imuEndpointvehicleType": "tcp://*:8802R56"
    }
}

...

This Filter outputs:

  • fusedVehiclePose

Output data format

Code Block
"vehicle": {
    "typefusedVehiclePose": "Automotive",{
    "vehicleStateEndpoint": "tcp://*:8999",     "settingsacceleration": {
            "canInterfacex": "PeakCAN",-0.4263402493894084,
            "vehicleTypey": "R56"-0.14872631710022688,
      } }

This Filter outputs:

  • fusedVehiclePose

Output data format

Code Block
{     "fusedVehiclePosez": {9.790632347106932
        },
        "accelerationglobalPosition": {
            "x": -01.42634024938940848985360999771979,
            "y": -041.14872631710022688,50585830111033
        },
   "z     "lastDataTime": {
            "timestamp": 9.7906323471069320
        },
        "globalPositionposition": {
            "x": 1.89853609997719790,
            "y": 41.505858301110330
        },
        "lastDataTimetimestamp": {
            "timestamp": 048347424440200
        },
        "positionutmZone": {
 "31T",
          "xyaw": 0,
            "y": 0
        },
        "timestamp": {
            "timestamp": 48347424440200
        },
   }
}

Parameter name

Description

Unit

acceleration

3D acceleration vector as measured by IMU. Describes the orientation of the vehicle.

m/s^2

globalPosition

Longitude and latitude in degrees

degrees

lastDataTime

Ignore

s

position

Position within UTM zone

m

timestamp

Timestamp of data acquisition

ns

utmZone

UTM zone

UTM string

yaw

Globally referenced yaw angle

rad

High-Dynamics Filter (IMU + GPS)

Node name: gnssImuFusion

Configuration block example (in sinks section)

Code Block
"gnssImuFusion": {
    "echoFusedPose": false,
    "utmZoneendpoint": "31Ttcp://*:8803",
    "fuser": {
        "fitModel": "yawModelGnssImu",
        "accelError": 0.01,
        }
}

...

Parameter name

...

Description

...

Unit

...

acceleration

...

3D acceleration vector as measured by IMU. Describes the orientation of the vehicle.

...

m/s^2

...

globalPosition

...

Longitude and latitude in degrees

...

degrees

...

lastDataTime

...

Ignore

...

s

...

position

...

Position within UTM zone

...

m

...

timestamp

...

Timestamp of data acquisition

...

ns

...

utmZone

...

UTM zone

...

UTM string

...

yaw

...

Globally referenced yaw angle

...

rad

High-Dynamics Filter (IMU + GPS)

Node name: gnssImuFusion

Configuration block example (in sinks section)

Code Block
"gnssImuFusion": {
    "echoFusedPose": false,"omegaError": 0.02,
        "measurementError": 0.05,
        "imuToCarRotation": {
            "w": 1,
            "x": 0,
            "endpointy": "tcp://*:8803"-1,
    "fuser": {
         "fitModelz": "ModelGnssImu",         "accelError": 0.01,
        "omegaError": 0.02,}
         "measurementError": 0.05,
        "imuToCarRotation": {
            "w": 1,
            "x": 0,
            "y": -1,
   }
}

Parameter name

Description

Default

echoFusedPose

fusedVehiclePose output is printed to command line

false

endpoint

Output port for the fusion result

8801

fitModel

Model to use for fusion. At the moment only SimpleCarModel is supported.

SimpleCarModel

accelError

Model to use to calculate car trajectory from CAN bus data. At the moment only Differential is supported.

Differential

omegaError

Omega error for Kalman filter. Keep default value.

0.5

measurementError

Measurement error for Kalman filter. Keep default value.

0.1

imuToCarRotation

Orientation quaternion of IMU relative to car frame

1, 0, -1, 0

This filter needs as input:

  • LPMS-IG1P data source for IMU and GPS data

Code Block
"imuP": {
    "type": "DualRtk",
    "settings": {
        "zsensor1": {
0         }   // If }
}

...

Parameter name

...

Description

...

Default

...

echoFusedPose

...

fusedVehiclePose output is printed to command line

...

false

...

endpoint

...

Output port for the fusion result

...

8801

...

fitModel

...

Model to use for fusion. At the moment only SimpleCarModel is supported.

...

SimpleCarModel

...

accelError

...

Model to use to calculate car trajectory from CAN bus data. At the moment only Differential is supported.

...

Differential

...

omegaError

...

Omega error for Kalman filter. Keep default value.

...

0.5

...

measurementError

...

Measurement error for Kalman filter. Keep default value.

...

0.1

...

imuToCarRotation

...

Orientation quaternion of IMU relative to car frame

...

1, 0, -1, 0

This filter needs as input:

  • LPMS-IG1P data source for IMU and GPS data

Code Block
"imuP": {
    "type": "DualRtk",
    "settings": {specification needed, insert first IG1 sensor name here
            //"name": "ig1p232800650050",
            "autodetectType": "ig1p"
        },
        "rtcm": true,
        "imuEndpoint": "tcp://*:8802"
    }
}

  • CAN bus and vehicle decoder source

Code Block
"vehicle": {
    "type": "Automotive",
    "vehicleStateEndpoint": "tcp://*:8999",
    "settings": {
        "canInterface": "PeakCAN",
        "sensor1vehicleType": {"R56"
    }
}

This Filter outputs:

  • fusedVehiclePose

  • fusedPose

Output data format

Code Block
{
    "fusedVehiclePose": {
// If specification needed, insert first IG1 sensor name here"acceleration": {
            //"namex": 0.0,
 "ig1p232800650050"           "y": 0.0,
            "autodetectTypez": "ig1p"0.0
        },
        "rtcmglobalPosition": {
   true,         "imuEndpointx": "tcp://*:8802"1.8982356601544925,
     } }

  • CAN bus and vehicle decoder source

Code Block
"vehicle": {     "typey": "Automotive",41.50544434418204
       "vehicleStateEndpoint": "tcp://*:8999", },
        "settingslastDataTime": {
            "canInterfacetimestamp": "PeakCAN",
    0
   "vehicleType": "R56"     }
}

This Filter outputs:

  • fusedVehiclePose

  • fusedPose

Output data format

Code Block
{
    "fusedVehiclePose": {
,
        "accelerationposition": {
            "x": 0-25.038332083641826,
            "y": 0-36.0,403733501197635
            "z": 0.0
        },
        "globalPositiontimestamp": {
            "xtimestamp": 1.8982356601544925,48910226723400
        },
   "y": 41.50544434418204         }"utmZone": "31T",
        "lastDataTimeyaw": {0.1555754684457767
            "timestamp": 0
        },
        "position": {
            "x": -25.38332083641826,
            }
}

Parameter name

Description

Unit

acceleration

3D acceleration vector as measured by IMU. Describes the orientation of the vehicle.

m/s^2

globalPosition

Longitude and latitude in degrees

degrees

lastDataTime

Unused

s

position

Position within UTM zone

m

timestamp

Timestamp of data acquisition

ns

utmZone

UTM zone

UTM string

yaw

Globally referenced yaw angle

rad

Code Block
{
	"fusedPose": {
		"lastDataTime": {
			"timestamp": 0
		},
		"orientation": {
			"w": 0.4437907292666558,
			"x": 0.5659687502206026,
			"y": -360.403733501197635
        },
        "timestamp4749652416904733,
			"z": 0.5070869566224411
		},
		"position": {
			"x":            "timestamp": 48910226723400
        },
        "utmZone": "31T",
        "yaw": 0.1555754684457767
    -25.383320836418306,
			"y": -36.403733501197166,
			"z": 163.98272320756405
		},
		"timestamp": {
			"timestamp": 48910226723400
		}
	}
}

Parameter name

Description

Unit

acceleration

3D acceleration vector as measured by IMU. Describes the orientation of the vehicle.

m/s^2

globalPosition

Longitude and latitude in degrees

degrees

lastDataTime

Unused

s

position

Position within UTM zone

lastDataTime

Unused

s

orientation

Orientation quaternion

n/a

position

Unused

m

timestamp

Timestamp

Time of data

acquisition

acqusition

ns

utmZone

UTM zone

UTM string

yaw

Globally referenced yaw angle

rad

Code Block
{
	"fusedPose": {
		"lastDataTime": {
			"timestamp": 0
		},
		"orientation": {
			"w": 0.4437907292666558,
			"x": 0.5659687502206026,
			"y": -0.4749652416904733,
			"z": 0.5070869566224411
		},
		"position": {
			"x": -25.383320836418306,
			"y": -36.403733501197166,
			"z": 163.98272320756405
		},
		"timestamp": {
			"timestamp": 48910226723400
		}
	}
}

...

Parameter name

...

Description

...

Unit

...

lastDataTime

...

Unused

...

s

...

orientation

...

Orientation quaternion

...

n/a

...

position

...

Unused

...

m

...

timestamp

...

Time of data acqusition

...

Communication with External Applications

Sending FusionHub Data to External Applications via the ZeroMQ Interface

FusionHub emits data resulting from the sensor fusion through the local network interface.

Output Ports

The network port that this information is output to can be configured in the JSON parameter file config.json of FusionHub.

Data Format

As low level protocol to emit the output data we use ZeroMQ (publisher / subscriber). The data itself is in JSON format and is encoded as Protocol Buffers. Protocol Buffers are documented here. Message are defined in the Protobuf (.protoc) format as defined in the file stream_data.proto. This file is contained in the installation folder of FusionHub.

Python Resources

Download a Python example that shows how to decode messaged from FusionHub from this repository.

Prerequisites can be installed in your Python 3 environment with this:

Code Block
pip install zmq
pip install protobuf

Make sure to set the input port in FusionHubPythonExample.py correctly. For example for the Antilatency source definition like below, the port needs to be set to 8899.

Code Block
"optical": {
    "type": "Antilatency",
    "settings": {
        // Use this for access from an external process eg. ALVR
        "endpoint": "tcp://*:8899",
        
        "environmentLink": "AntilatencyAltEnvironmentHorizontalGrid~AgAEBLhTiT_cRqA-r45jvZqZmT4AAAAAAAAAAACamRk_AQQCAwICAgICAQICAAI",
        "placementLink": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
    }
}

C# Resources

On parsing Protobuf files: https://github.com/5argon/protobuf-unity

How to subscribe to ZeroMQ messages: https://github.com/gench23/unity-zeromq-client and https://tech.uqido.com/2020/09/29/zeromq-in-unity/

VRPN Output

VRPN output is set in the following part in the sinks section of config.json. The device name will be referenced by the plugin for Unreal engine.

Code Block
"VRPN": {
  "settings": {
    "deviceName": "Fusion Hub"
  }
}

Please see below how we achieve data input via VRPN in the Unreal engine. First, install the VRPN LiveLink plugin:

...

Configure the VRPN source with the correct device and subject name:

...

Apply the output from the fusion hub to an Unreal object eg. a cine camera actor.

...

Release Notes

Version 1.1

Release date: 2022/11/21

...