Versions Compared

Old Version 31

changes.mady.by.user Klaus Petersen

Saved on

compared with

New Version 32

changes.mady.by.user Klaus Petersen

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Info

Note that this documentation is for the client-side version of LPVR-AIR, where FusionHub runs on the headset. The client-side version has been superseded by the server-side version that runs FusionHub on the server and is easier to use. Check the server-side documentation here.

Table of contents

Table of Contents

Introduction

Info

LPVR-AIR is based on FusionHub. This is a condensed version of the FusionHub manual that focuses on the VR use-case. Find the full FusionHub documentation here.

...

See the illustration below for a block diagram of the overall system:

...

System components

Applications

The following applications need to be started on the head mounted display and the host computer. They should all the included in the installation package that you received from us. We will discuss the order of starting these applications and what their status output should be below.

...

Application

Purpose

Name

FusionHub GUI

  • Connects to FusionHub server on HMD

  • Configures FusionHub

  • Authenticates system

lp-fusionhub-dashboard.exe

ALVR server

  • Receives pose information from client

  • Sends pose information to SteamVR

  • Sends image data to ALVR client

  • Receives image information from SteamVR

ALVR Launcher.exe

Authentication

FusionHub authenticates itself via the GUI client application. In order to run the client application make sure to insert the LPVR USB dongle into the host computer. After the GUI client is connected to FusionHub on the HMD, FusionHub will start streaming pose data to ALVR. Alternatively we offer the possibility to use use an online licensing system that unlocks HMD nodes via the internet. By default we deliver the dongled version.

Running LPVR-AIR

Installation

Install the FusionHub APK and ALVR client APK on the headset using a side-loading tool like Sidequest. In case of a Meta Quest HMD this will require you to put the HMD into developer mode. See here the steps for putting the HMD into developer mode: https://developer.oculus.com/documentation/native/android/mobile-device-setup/

...

Info

Make sure you turn the guardian (i.e. the automatic tracking boundary detection for the Quest’s internal tracking) is turned off in the developer settings of the HMD. For VIVE Focus use the equivalent setting in the Focus' configuration.

Start-up

FusionHub

  • Start FusionHub on the HMD. A window showing the FusionHub console output should open.

  • Start the FusionHub GUI client on the host computer

  • Connect the GUI client to FusionHub on the HMD. Make sure HMD and host are in the same subnet. Enter the the correct IP of the HMD in the client before pressing connect.

...

Info

Press the buttons ‘Set’ and ‘Save’ after changing the configuration script to make your changes active. It might take 1-2 seconds for FusionHub to reset.

In case you happen to enter an invalid configuration, FusionHub might not restart correctly. If you would like to reset your settings, just re-install the FusionHub APK.

Once the configuration is correct, you’ll most likely not have to touch the script again in the foreseeable future.

ALVR

  • Start the ALVR server on the host. While the ALVR server starts up, it will automatically run SteamVR.

  • Start the ALVR client on the HMD. The HMD should be shown in the list of ALVR clients in the ALVR server application. In some cases you need to click the Trust button in the application to start streaming.

  • You can now use and calibrate the system as described in the FusionHub BASE section.

...

Info

Once streaming starts, you should see the SteamVR default environment through the headset. Check if the nIMU counter in the FusionHub GUI is increasing. If both nOptical and nIMU are increasing then the communication between ALVR, optical tracking and FusionHub is working.

Optical tracking systems

Configuration options

Advanced Realtime Tracking (ART) DTrack

FusionHub works with all ART tracking systems, based on their DTrack tracking software.

...

See this guide for further information about how to configure DTrack. Also, refer to this page for more information on ART.

Optitrack

FusionHub works with all Optitrack tracking systems based on their Motive tracking software.

...

Refer to this page for Optitrack setup from the LPVR documentation.

Advanced troubleshooting

The best way to diagnose what is going wrong with the system if something doesn’t work as expected is to look at the log output of FusionHubLauncher. The log can be recorded by connecting your HMD to a PC using a USB (most likely USB-C) cable.

...

Enter adb logcat | findstr fusionhub to stream log data from the device to your command line. Only data from the FusionHubLauncher application will be displayed. The initialization log will be displayed when FusionHubLauncher is first started and every time you restart it using the Restart button in the GUI.

Network setup

Router

In order to establish high bandwidth communication between the host and Quest 2 / Pro we recommend setting up a 5GHz or for optimum performance a 6GHz (WIFI 6e) WIFI router with 2 channels per HMD. We recommend the Eero WIFI 6E router.

Network topography

We recommend to set up a simple network structure, to minimize potential error sources in the installation process as shown in the image below.

...

Network performance

  • A WIFI 6E connection is recommended to achieve optimum performance:

...

  • In case of a sufficient rendering performance the advanced frame timing window should look like the output below:

...

Challenges and limitations of using Meta Quest HMDs

External pose input / internal late latching conflict

Note

This issue has been mostly resolved. Please report to us if you have issues with bad reprojection.

...

Therefore we tried to find a compromise between image stability and late latching capability. If this compromise works well, strongly depends on user requirements and quality of the WIFI connection established to the HMD. A perfect solution is unfortunately not possible without support from the HMD manufacturer Meta.

WIFI environment quality

LPVR-AIR transmits images from the server to the HMD through a regular WIFI connection. Usually a 5G band is used, in the optimum case we switch to WIFI 6E. In environments without much WIFI interference, i.e. other devices using the same WIFI bands, this works very well. Crowded WIFI environments limit the bandwidth of the used WIFI transmission. This can lead to unpredictable loss of image and tracking quality. Examples of crowded spaces are public locations such as exhibition. Beware!

...

In an ideal setup, with several HMDs being used, each HMD uses a separate WIFI 6e channel, with the selected channels as far apart as possible.

Optical tracking parsing latency

Due to limited WIFI bandwidth and computing power limitations on the HMD pose information streamed from the the optical tracking system is parsed on the HMD with a significant delay. So far we have not found a way to reduce this delay. As described in the sensor fusion section, we added input from native inside-out tracking to the fusion in order to compensate for this latency.

HMD built-in CPU performance limitation

The Meta Quest series headsets are stand-alone HMDs that integrate all computing hardware in one device. Due to size and heat emission restrictions, the amount of computing power they provide is limited. For the operation of LPVR, significant compute is required to decode the image stream coming from the rendering PC to the HMD. For high resolution images with a high framerate, performance bottlenecks can happen, resulting in reduced image quality or tracking artefacts. we discovered during experiments that turning the internal inside-out tracking of the HMD off, can improve the performance of the rest of the system. Note that in this case, inside-out tracking can’t be used to supplement pose calculation by FusionHub.

Inside-out / outside-in tracking misalignment

When fusing inside-out and outside-in tracking it is important that the coordinate systems of both pose calculation methods are congruent. If the pivot point of both coordinate systems is not identical, the resulting 3D image might show objects to be shifting with head motion. We provide a calibration tool for ART and Optitrack tracking to calibrate the outside-in pivot point to the inside-out pivot point.

Inside-out fusion (for advanced users)

Info

Adding inside-out tracking data to the fusion mix is optional. It only works if the optical marker target doesn’t cover the HMD’s cameras and you operate in a well-lit room with sufficient features to track.

...

Parameter

Purpose

ioHpWeight

Weight of low-pass filter applied to inside-out tracking after high-pass filter (0 = unmodified inside-out signal, 1 = remove inside-out signal)

ioLpWeight

Weight of high-pass filter applied to inside-out tracking (0 = remove inside-out signal, 1 = full inside-out signal applied to result)

optLpWeight

Weight of low-pass filter applied to outside-in tracking (0 = unmodified outside-in signal, 1 = outside-in signal is removed)

useIOHeight

Set this to true to take height (y-axis) value directly from inside-out tracking

useIOHorizontal

Set this to true to include inside-out tracking in pose calculation. If true, filter parameters described above will be used.

Example configurations

Example configurations for the outside-in / inside-out filter:

Characteristic

Parameter setting

Mainly inside-out, outside-in for global referencing: Tracking is based mainly on the built-in inside-out tracking. Outside-in tracking is used for global referencing.

Note: Turn headset tracking to ON in Quest settings. Room needs to be suitable for inside-out tracking.

Code Block
"ioHpWeight": 0,
"ioLpWeight": 0.9995,
"optLpWeight": 0.9995,
"useIOHeight": false,
"useIOHorizontal": true

Outside-in only + height from inside-out: Horizontal tracking is based on outside-in tracking. Only height (global y-axis) values are taken directly from inside-out tracking.

Note: Turn headset tracking to ON in Quest settings. Room needs to be suitable for inside-out tracking.

Code Block
"ioHpWeight": 1,
"ioLpWeight": 0,
"optLpWeight": 0,
"useIOHeight": true,
"useIOHorizontal": false

Outside-in tracking only: Tracking is based on outside-in tracking only.

Note: To maximize performance, make sure to turn headset tracking to OFF in Quest settings.

Code Block
"ioHpWeight": 1,
"ioLpWeight": 0,
"optLpWeight": 0,
"useIOHeight": false,
"useIOHorizontal": false

Advanced intercalibration (only on Meta Quest)

Note

Note that if you have a pre-configured target from ART or Optitrack, you don’t need to do the intercalibration. You only need to do this if you have a custom target attached to your HMD.

Example videos

Motive

Intercalibratiotool: https://www.dropbox.com/scl/fi/1nkt3acbq69omehzz0j28/Intercalibration-tool-cryptolens.mp4?rlkey=qtg49h7kzbhcs4cz32yrnx7vn&dl=0

DTrack

PC screen recording: https://www.dropbox.com/scl/fi/6ouhtkac6409iekn51hbc/setupLPVRAIR-PC.mp4?rlkey=7ib34as0bfz4veqw69u0s6tga&st=yzvh9g4g&dl=0

Quest screen recording: https://www.dropbox.com/scl/fi/8fpsoybeb67sfn6x35ijt/setupLPVRAIR-Quest2.mp4?rlkey=xy2s5irxaguy5mw7ci8ep8tks&st=118r4rw8&dl=0

Software Required

PC

  1. Optical system software: Motive / DTrack software

  2. Oculus desktop software

  3. SteamVR (download Steam Client from official site, and then follow instructions here)

  4. lh_opt_calib.exe
    Dongled ← requires dongle hardware
    Cryptolens ← requires Internet connection on the 1st activation call

Procedure Overview

There are 5 sections in total for the setup:

  1. Create a new marker body file for the headset

  2. Connect PC to the headset through SteamVR

  3. Generate a calibrated body with lh_opt_calib

  4. Load the calibrated body file to the optical tracking software

  5. Start the FusionHub

I - Create a new marker body file for the headset

Motive

  1. In the perspective view, select the markers that you wish to group into the new rigid body.

  2. Go to View → Builder Pane, give the new rigid body a new name.

  3. Go to View → Assets Pane, select the newly created rigid body, right click on it and export it.

DTrack

  1. Go to Tracking → Body Administration, increase the count by 1, and give your new tracking body a name. Then click “Calibrate”.

  2. Click “calibrate” again and gently move and roll your headset in front of the tracking camera.

  3. Save the body file by clicking “save file(s)”, select your headset body and save it to an empty folder.

II - Connect PC to the headset through SteamVR

  1. On the headset, go to settings → system → Quest Link. Pairing with the PC and click the launch icon. This would start the Oculus software on the PC.

  2. Then on the headset, start SteamVR. This will start the SteamVR on your PC, also in your headset you will enter into SteamVR’s default scene automatically.

III - Generate a calibrated body with lh_opt_calib

Motive

  1. In the command line please go to the directory of lh_opt_calib and execute the following:

    Code Block
    lh_opt_calib.exe --opticalSystem motive --bodyId 6 --localAddress 192.168.1.18 --remoteAddress 192.168.1.18 --bodyFile "C:\Users\lpresearch\Desktop\Quest2TestBodyFile\standard b06.motive"

    bodyId to your headset’s tracking ID,
    localAddress and remoteAddress is the setting for NatNet client
    bodyFile to the file path (from section 1 step 3)

DTrack

  1. Start the ART tracking by clicking “start” in DTrack software.

  2. In the command line please go to the directory of lh_opt_calib and execute the following:

    Code Block
    lh_opt_calib.exe --opticalSystem dtrack --port 5012 --bodyId 6 --bodyFile "C:\Users\lpresearch\Desktop\Quest2TestBodyFile\standard b06.txt"

    set port to DTrack’s port (can be found in DTrack’s Tracking → Output)
    bodyId to your headset’s tracking ID in DTrack
    bodyFile to the file path (from section 1 step 3)

  3. Rotate the headset in front of the tracking camera to collect 10 poses. If in the end all 5 outputs looks close, this section is considered finished. Otherwise please retry step 4 and 5.

  4. Exit the SteamVR on PC. Exit the Oculus software on your headset by clicking “Quit” in the menu for Quest AirLink.

IV - Load the calibrated body file to the optical tracking software

Motive

  1. Remove the rigid body you created just now. Otherwise Motive does NOT overwrite it with the calibrated one.

  2. Drag and drop the newly generated body file xxx_HORAUD.motive into the Assets Pane in the Motive software.

DTrack

  1. In the DTrack software on PC, click “stop”.

  2. Go to Tracking → Body Calibration, select your headset body in the dropdown menu, and click “Load file(s)” to load the calibrated body file xxx_HORAUD.txt under the same directory as your exported body file in section 1.

How to inter-calibration without precalibrated targets

  • We highly recommend purchasing precalibrated targets. If you don’t have precalibrated targets, please follow the instructions in the videos below.

Dtrack calibration : https://www.dropbox.com/scl/fi/cjmaykw5rt5263euqo0z1/Dtrack-Calibration-LPVR-AIR.MP4?rlkey=5y829p6wsi5z6dfpz8vtcpbze&st=3kkd916n&dl=0

Procedure Overview

  1. Open the ART DTrack3 software and connect to the controller

  2. Open "Body Administration" under Tracking > Body Administration

  3. Select the body you wish to calibrate (or create a new one by increasing the "number of bodies" counter)

...

12. Click on "Apply", then "OK". All done!

FusionHub intercalibration : https://www.dropbox.com/scl/fi/di0dzqcdvhmm4m67wrtsf/FusionHub-Intercalibration-LPVR-AIR.mp4?rlkey=yrxlgfok7ghazf6kgemetd00k&st=uorf365v&dl=0

Procedure Overview

Pre-requisites

  • Make sure FusionHub is installed and configured to receive IMU and optical data.

    • You will also need a computer connected to the same network as the headset.

    • For this demo, we will use LP-ALVR to provide IMU data to FusionHub.

 Intercalibration

  1. Launch FusionHub on the headset

  2. Connect FusionHub to the GUI application using the internal IP address of the headset

...

  1. From there click on "Apply Intercalibration Results". Then, on the top of the "Configuration section", click on "Set", then "Save". This will cause FusionHub to restart. Hopefully, everything works now!

References

...