Introduction

FusionHub is a software application that has the purpose of combining various sensor data inputs to create a higher level data output. There are 3 basic versions of FusionHub:

• FusionHub BASE combines data from an outside-in tracking system with inertial measurements via an IMU. Typical applications: Head-mounted display tracking for VR/AR applications, camera tracking for virtual production

• FusionHub MOVE adds an additional platform IMU to the BASE configuration. It combines the data from both IMUs to calculate poses relative to a moving platform. Typical applications: AR/VR in a vehicle, aircraft, or on a simulator platform

• FusionHub FLOW combines odometry, GPS and IMU data from a vehicle to calculate high-accuracy and low-latency global localization information. Typical applications: Automobile localization, robot localization

General

Running the fusion hub

FusionHub consists of two components:

• The main application

• A graphical user interface application

The screenshot below shows a typical output when starting FusionHub.exe.

...

When initializing fusion hub does the following:

• Discover all connected LPMS-IG1P

• Connect to two LPMS-IG1P

• Connect to the NTRIP caster relaying the RTK signal

• Initialize the VRPN output module

• Initialize all other sources and sinks

Next, fusion hub will check the license key in the config.json configuration file.

...

Copy and paste the response key into the ResponseKey section of config.json. The response key is not absolutely required as long as the computer is connected to the internet. Inserting it in the configuration file will allow the system to work without internet and speed up the initialization process.

...

After the system is completely initialized it will

• Stream fused poses (position and orientation) via VRPN server port 3883 to Unreal or Unity

• Show the reception of RTCM data from the base station

• Show the RTK-GPS connection status of sensor 0 and 1. This status should ideally be FIXED.

• Show the fused pose output via VRPN

Low-Dynamics filter (Odometry + GPS + (some)IMU) data format

{
  "fusedVehiclePose": {
    "acceleration": {
      "x": -0.4263402493894084,
      "y": -0.14872631710022688,
      "z": 9.790632347106932
    },
    "globalPosition": {
      "x": 1.8985360999771979,
      "y": 41.50585830111033
    },
    "lastDataTime": {
      "timestamp": 0
    },
    "position": {
      "x": 0,
      "y": 0
    },
    "timestamp": {
      "timestamp": 48347424440200
    },
    "utmZone": "31T",
    "yaw": 0
  }
}

Output example

...

High-Dynamics filter (IMU + GPS) data format examples

{
    "fusedVehiclePose": {
        "acceleration": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "globalPosition": {
            "x": 1.8982356601544925,
            "y": 41.50544434418204
        },
        "lastDataTime": {
            "timestamp": 0
        },
        "position": {
            "x": -25.38332083641826,
            "y": -36.403733501197635
        },
        "timestamp": {
            "timestamp": 48910226723400
        },
        "utmZone": "31T",
        "yaw": 0.1555754684457767
    }
}

{
	"fusedPose": {
		"lastDataTime": {
			"timestamp": 0
		},
		"orientation": {
			"w": 0.4437907292666558,
			"x": 0.5659687502206026,
			"y": -0.4749652416904733,
			"z": 0.5070869566224411
		},
		"position": {
			"x": -25.383320836418306,
			"y": -36.403733501197166,
			"z": 163.98272320756405
		},
		"timestamp": {
			"timestamp": 48910226723400
		}
	}
}

Output example

...

Application Setup

1. Download and install LpmsControl 2 from here.

2. Open the FusionHub ZIP file and copy the contents to a local folder.

3. Edit config.json and adjust parameters as needed. The default config.json looks like below:

...

5. Check if the application is receiving optical data. nOptical should be steadily increasing.

Intercalibration

The intercalibration calculates the orientation offset between the optical tracing body and the IMU attached to it. This calibration is needed for drift free fusion of these two data sources.

1. Place the sensor incl. the optical target in a motion-less state. Wait for the autocalibration to fininsh (autocalibrationStatus -> true).

2. Slowly rotate the sensor (with the optical target attached) within the tracking volume until finished in Intercalibration status becomes true.

3. Copy and paste the output quaternion to intercalibration section in config.json.

4. Restart the application to apply the intercalibration values.

Normal Operation

For normal operation the autocalibration needs to be completed after starting the application. The intercalibration only needs to be run once for a fixed alignment of the optical target and IMU.

1. Start the application via the batch file or the command line

2. Wait for the autocalibration to finish

3. Apply fused data to Unreal scene

Output to Unreal Configuration

1. VRPN output is set in the following part in the sinks section of config.json. The device name will be referenced by the plugin for Unreal engine.

...

4. Apply the output from the fusion hub to an Unreal object eg. a cine camera actor.

...

Config.json Sample

{
  // Definition of data sources
  "sources": {

    // Settings for optical source or proxy
    "optical": {
    
      // Uncomment the lines below to enable Optitrack support
      // Make sure to comment the ART tracking related lines
      //"type": "Optitrack",
      //"settings": {
      //  "host": "localhost",
      // "connectionType": "Multicast",
      // "bodyID": 0
      // }
	  
	  "type": "DTrack",
	  "settings": {
		"port": 5000,
		"bodyID": 1
	  }
    }

    // IMU source address, localhost if run on the same node
    ,
    // "endpoints": [ "tcp://localhost:8765" ]
	"imu": {
	  "type": "OpenZen",
	  "settings": {
	    "autodetectType": "ig1"
      }
	}

    // File reader node to replay sensor data
    //, "filereader": {
    //   "filename": "sensorDataWithTimecode.json"
    // }
  },

  // Definition of data sinks
  "sinks": {
	"VRPN": {
      "settings": {
		"deviceName": "LPVP"
      }
    }	  

    // Sensor fusion node with result quaternion of intercalibration inserted
    , "fusion": {
    
      // Fill in your license key here
      "LicenseInfo": {
        "LicenseKey": "",
        "ResponseKey": ""
      }

      // Fill in the result from the intercalibration here
      , "intercalibration": {
		"w":1.0,
		"x":0.0,
		"y":0.0,
		"z":0.0		
      }
    }

    // Uncomment the following line to activate intercalibration
    , "intercalibration": {}

    // Uncomment the following line to output data to command line
    //, "echo": {}

    // Uncomment the following block to replay data
    //, "replay": {
    //   "settings": {
    //     "bufferSize": 500e-3,
    //     "stepDuration": 0,
    //     "writerEndpoint": "tcp://*:9921",
    //     "verbose": true,
    //     "timecodeInput": true
    //   }
    // }

    // Uncomment following block to record result data to file in JSON format
    //, "record": {
    //  "filename": "log.a",
    //  "format": "json"
    //}
  }
}

Sending FusionHub Data to External Applications via Native ZeroMQ Interface

FusionHub emits the following data through the local network interface:

• Optical pose as acquired from Antilatency tracking

• The fused pose of the HMD ie. the combined data from HMD IMU and ALT tracking

Output Ports

The ports that this information is output to can be configured in the JSON parameter file config.json of FusionHub. The following lines defines these ports:

Antilatency optical poses

...
      // Antilatency
      "type": "Antilatency",
      "settings": {
        // Use this for access from an external process eg. ALVR
        "endpoint": "tcp://*:8899",
...

Fused HMD Poses

...
    // Sensor fusion node with result quaternion of intercalibration inserted
    "fusion": {
      // Use this for access from an external process eg. ALVR
      "endpoint": "tcp://*:8799",
...

Data Format

The low level protocol that this data is output by is ZeroMQ (publisher / subscriber). The data itself is in JSON format and is encoded as Protocol Buffers as is described here.

Message are defined in Protobuf (.protoc) format:

Antilatency optical poses

message OpticalData {
  string object_name = 11;
  int64 timecode = 1;
  bool fake_timecode = 10;
  int64 recorded_time = 2;
  double latency = 9;
  Vector position = 3;
  Quaternion orientation = 4;
  // Errors should be encoded here as well, but they actually are specific to the source
  // something for errors = 5;
  double quality = 6;
  double frame_rate = 7; // in seconds, necessary?
  int32 frame_number = 8; // internal to opticalsystem
  Vector angular_velocity = 12;
}

Note that object_name corresponds to the custom attribute trackerName that we set for each Antilatency tracker. You can filter the objects that you would like to follow using this identifier.

Fused HMD Poses

message FusedPose {
  string object_name = 6;
  int64 timecode = 5; // Optional: if 0 not set.
  int64 timestamp = 1;
  Vector position = 2;
  Quaternion orientation = 3;
  Vector angular_velocity = 4;
}

Python Resources

Download a Python example from this repository.

...

"optical": {
      "type": "Antilatency",
      "settings": {
        // Use this for access from an external process eg. ALVR
        "endpoint": "tcp://*:8899",

        // Use this for internal access eg. sensor fusion
        //"endpoint": "inproc://optical_data_source_1",

        "environmentLink": "AntilatencyAltEnvironmentHorizontalGrid~AgAEBLhTiT_cRqA-r45jvZqZmT4AAAAAAAAAAACamRk_AQQCAwICAgICAQICAAI",
        "placementLink": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
      }
    },

C# Resources

On parsing Protobuf files: https://github.com/5argon/protobuf-unity

How to subscribe to ZeroMQ messages: https://github.com/gench23/unity-zeromq-client and https://tech.uqido.com/2020/09/29/zeromq-in-unity/

Outputting FusionHub Data via VRPN

In config.json make sure to add the following block to the sinks section:

...