LPVR-AIR Manual

Table of contents

Introduction

The purpose of LPVR-AIR is to wirelessly stream image data from a SteamVR application such as Autodesk VRED to a wireless HMD like the Meta Quest or VIVE Focus. LP-Research's FusionHub software in combination with the open-source application ALVR fulfill this purpose well.

ALVR by default uses the internal inside-out tracking of Meta Quest for pose calculation. LPVR-AIR exchanges the Quest’s native inside-out tracking with combined IMU and ART / Optitrack outside-in tracking to allow simultaneous, spatially synchronized operation of several HMDs in large tracking volumes.

To make the tracking functionality of FusionHub available to standalone augmented and virtual reality headsets, it can be integrated with Android-compatible OpenXR HMDs. This works via a customized version of the ALVR open source project. ALVR allows streaming image data wirelessly from a host computer and interfaces to 3D content engines through SteamVR. While the original ALVR client was built to work on Meta Quest HMDs, ALVR works in principle on any OpenXR compatible headset.

We use a thin client library to receive IMU data from the HMD API, pass it to FusionHub, process it there and then re-inject the information into the video pipeline of the headset. Depending on the type of HMD this happens within the ALVR client’s standard interface or in a separate hardware-specific API layer.

The overall system consists of several applications running at the same time. As the development of this application is still work-in-progress, starting and configuring this solution can be a bit cumbersome. We are working on making the process easier as we move along.

See the illustration below for a block diagram of the overall system:

System components

Applications

The following applications need to be started on the head mounted display and the host computer. They should all the included in the installation package that you received from us. We will discuss the order of starting these applications and what their status output should be below.

On the headset:

On the host computer:

Authentication

FusionHub authenticates itself via the GUI client application. In order to run the client application make sure to insert the LPVR USB dongle into the host computer. After the GUI client is connected to FusionHub on the HMD, FusionHub will start streaming pose data to ALVR. Alternatively we offer the possibility to use use an online licensing system that unlocks HMD nodes via the internet. By default we deliver the dongled version.

Running LPVR-AIR

Installation

Install the FusionHub APK and ALVR client APK on the headset using a side-loading tool like Sidequest. In case of a Meta Quest HMD this will require you to put the HMD into developer mode. See here the steps for putting the HMD into developer mode: https://developer.oculus.com/documentation/native/android/mobile-device-setup/

In case you’re using a VIVE Focus 3 headset you need to do something similar as described here: https://developer.vive.com/resources/hardware-guides/vive-focus-specs-user-guide/how-do-i-put-focus-developer-mode/

The FusionHub GUI client and ALVR server can be started on the host PC without further installation, they can be run right out of the deployment folder.

ALVR requires SteamVR to be set up on the host computer. If you haven’t installed it on your computer yet, please refer to the instructions here: https://store.steampowered.com/app/250820/SteamVR/

Start-up

FusionHub

• Start FusionHub on the HMD. A window showing the FusionHub console output should open.

• Start the FusionHub GUI client on the host computer

• Connect the GUI client to FusionHub on the HMD. Make sure HMD and host are in the same subnet. Enter the the correct IP of the HMD in the client before pressing connect.

• Once connected the FusionHub on the headset select base configuration to see the current configuration of FusionHub:

• Adjust parameter blocks as needed. Refer to the description of FusionHub BASE for configuration options:

• Note the following input and output ports that are hard-coded in the ALVR FusionHub API layer. These are already correctly set in the default configuration file installed with the FusionHub APK, so usually there is no need to change them.

• If it’s not running yet make sure to start and configure your optical tracking system. Once optical data is streamed to FusionHub, the nOptical counter in the GUI should be increasing.

• Default configuration script with optical input defined for ART DTrack:

{
  "LicenseInfo": {
    "LicenseKey": "",
    "ResponseKey": ""
  },
  "settings": {
    "websocketDataOutputRate": 20
  },
  "sinks": {
    "VRPN": {
      "settings": {
        "inputEndpoints": [
          "inproc://optical_data_source_1"
        ],
        "settings": {
          "deviceName": "FusionHub",
          "port": 3883,
          "tracker0": "HMD"
        }
      }
    },
    "fusion": {
      "dataEndpoint": "tcp://*:8799",
      "inputEndpoints": [
        "inproc://optical_data_source_1",
        "tcp://localhost:8898"
      ],
      "settings": {
        "Autocalibration": {
          "minAgeS": 60,
          "nSamplesForAutocalibration": 1500,
          "nSamplesForSteady": 256,
          "noiseRmsLimit": 0.02,
          "steadyThresholdAverage": 0.2,
          "steadyThresholdRms": 1
        },
        "Intercalibration": {},
        "MotionDetection": {
          "omegaLimit": 3,
          "positionSampleInterval": 1000,
          "rotationFilterAlpha": 0.9,
          "timeToUnknown": 500
        },
        "SensorFusion": {
          "alignment": {
            "w": 0.990892966476337,
            "x": 0.13458639604387848,
            "y": 0.0005637732357904688,
            "z": 0.004160907038605602
          },
          "orientationWeight": 0.005,
          "predictionIntervalMs": 0,
          "sggPointsEachSide": 5,
          "sggPolynomialOrder": 5,
          "tiltCorrection": null,
          "yawWeight": 0.01
        },
        "runIntercalibration": false
      },
      "type": "ImuOpticalFusion"
    }
  },
  "sources": {
    "optical": {
      "settings": {
        "bodyIDs": [
          1
        ],
        "endpoints": [
          "inproc://optical_data_source_1"
        ],
        "port": 5000
      },
      "type": "DTrack"
    }
  }
}

ALVR

• Start the ALVR server on the host. While the ALVR server starts up, it will automatically run SteamVR.

• Start the ALVR client on the HMD. The HMD should be shown in the list of ALVR clients in the ALVR server application. In some cases you need to click the Trust button in the application to start streaming.

• You can now use and calibrate the system as described in the FusionHub BASE section.

Here is a video tutorial how to perform the initial startup procedure:

https://vimeo.com/1019970197

https://vimeo.com/1024128202?share=copy

Optical tracking systems

Configuration options

Advanced Realtime Tracking (ART) DTrack

FusionHub works with all ART tracking systems, based on their DTrack tracking software.

"optical": {
    "type": "DTrack",
    
    "settings": {
        "port": 5000,
        
        "bodyIDs": [
            1,
            2,
            3
        ],
        "endpoints": [
            "inproc://optical_data_source_1",
            "tcp://*:8500",
            "tcp://*:8501"
        ],
        "objectNameMapping": {
            "1": "QuestPro",
            "2": "leftController",
            "3": "rightController"
        }
    }
}
}

Adjust the body IDs as configured in DTrack. If you don’t plan to use hand controllers ignore the body IDs for the hand controllers (2 and 3 in the example above). Leave the endpoints and objectNameMapping blocks as shown above.

See this guide for further information about how to configure DTrack. Also, refer to this page for more information on ART.

Optitrack

FusionHub works with all Optitrack tracking systems based on their Motive tracking software.

"optical": {
    "type": "Optitrack",
    
    "settings": {
        "connectionType": "Multicast",
        "localAddress": "192.168.0.99", 
        "remoteAddress": "192.168.0.100",
            
        "bodyIDs": [
            1,
            2,
            3
        ],
        "endpoints": [
            "inproc://optical_data_source_1",
            "tcp://*:8500",
            "tcp://*:8501"
        ],
        "objectNameMapping": {
            "1": "QuestPro",
            "2": "leftController",
            "3": "rightController"
        }
    }
}

Adjust the body IDs as configured in Motive. If you don’t plan to use hand controllers ignore the body IDs for the hand controllers (2 and 3 in the example above). Leave the endpoints and objectNameMapping blocks as shown above. Local address needs to be set to the address of your headset, remote address is the address of the Motive machine.

Refer to this page for Optitrack setup from the LPVR documentation.

Advanced troubleshooting

The best way to diagnose what is going wrong with the system if something doesn’t work as expected is to look at the log output of FusionHubLauncher. The log can be recorded by connecting your HMD to a PC using a USB (most likely USB-C) cable.

For this purpose download the Android platform tools for Windows from here: https://developer.android.com/studio/releases/platform-tools

Copy the platform tools files to a folder and open a command prompt in that folder.

Check if the HMD is detected by your computer by entering adb devices. If your devices is not detected or marked as unauthorized make sure you have the correct USB driver for your HMD installed and acknowledged USB access from the host computer in the Android GUI on the HMD.

Enter adb logcat | findstr fusionhub to stream log data from the device to your command line. Only data from the FusionHubLauncher application will be displayed. The initialization log will be displayed when FusionHubLauncher is first started and every time you restart it using the Restart button in the GUI.

Network setup

Router

In order to establish high bandwidth communication between the host and Quest 2 / Pro we recommend setting up a 5GHz or for optimum performance a 6GHz (WIFI 6e) WIFI router with 2 channels per HMD. We recommend the Eero WIFI 6E router.

Network topography

We recommend to set up a simple network structure, to minimize potential error sources in the installation process as shown in the image below.

Network performance

• A WIFI 6E connection is recommended to achieve optimum performance:

• Transmission speeds are expected to be around 2 Gbps for a stable 6 GHz connection.

• A typical ALVR performance graph is shown below. Overall latencies in good environments should be between 70 and 90ms.

• In SteamVR check Advanced Frame Timing for performance problems:

• In case of a sufficient rendering performance the advanced frame timing window should look like the output below:

Challenges and limitations of using Meta Quest HMDs

External pose input / internal late latching conflict

By default Meta Quest HMDs use their built-in inside-out tracking for measuring the user's head pose. For this project we needed to exchange the built-in tracking with our customized pose estimation. Generally the Quest HMD's software is not made to allow this application. Correct VR rendering is optimized to work with the built-in tracking and alternative ways of pose input are not supported.

We were able to create a workaround for this problem to some degree, by exchanging the pose calculated by the internal system with the pose calculated by FusionHub, before it is sent to the server for rendering. On the rendering side this works well, but unfortunately creates a conflict between the pose reported by the HMD firmware and the pose used for rendering that affects the quality of the HMD's pose prediction, late latching / asynchronous time/space warp. This conflict is perceived as uncomfortable, slight jittering of the image and overcompensation of head motions.

A way to reduce this jittering effect is to reduce the prediction time of the internal pose calculation. This doesn't fully solve the problem though, as decreased prediction time also reduces the responsiveness of the internal late latching. This in turn leads to blurring at edges of the image when the user turns his head rapidly.

Therefore we tried to find a compromise between image stability and late latching capability. If this compromise works well, strongly depends on user requirements and quality of the WIFI connection established to the HMD. A perfect solution is unfortunately not possible without support from the HMD manufacturer Meta.

WIFI environment quality

LPVR-AIR transmits images from the server to the HMD through a regular WIFI connection. Usually a 5G band is used, in the optimum case we switch to WIFI 6E. In environments without much WIFI interference, i.e. other devices using the same WIFI bands, this works very well. Crowded WIFI environments limit the bandwidth of the used WIFI transmission. This can lead to unpredictable loss of image and tracking quality. Examples of crowded spaces are public locations such as exhibition. Beware!

The Meta Quest firmware doesn't allow using a wired ethernet connection to mitigate the issue to provide a quick fix to this problem in urgent situations. Again, a clean execution of this project is prevented by inflexibility of Meta's software. Unfortunately there are no alternative wireless HMDs on the market that allow the modifications we need for an optimum implementation.

In an ideal setup, with several HMDs being used, each HMD uses a separate WIFI 6e channel, with the selected channels as far apart as possible.

Optical tracking parsing latency

Due to limited WIFI bandwidth and computing power limitations on the HMD pose information streamed from the the optical tracking system is parsed on the HMD with a significant delay. So far we have not found a way to reduce this delay. As described in the sensor fusion section, we added input from native inside-out tracking to the fusion in order to compensate for this latency.

HMD built-in CPU performance limitation

The Meta Quest series headsets are stand-alone HMDs that integrate all computing hardware in one device. Due to size and heat emission restrictions, the amount of computing power they provide is limited. For the operation of LPVR, significant compute is required to decode the image stream coming from the rendering PC to the HMD. For high resolution images with a high framerate, performance bottlenecks can happen, resulting in reduced image quality or tracking artefacts. we discovered during experiments that turning the internal inside-out tracking of the HMD off, can improve the performance of the rest of the system. Note that in this case, inside-out tracking can’t be used to supplement pose calculation by FusionHub.

Inside-out / outside-in tracking misalignment

When fusing inside-out and outside-in tracking it is important that the coordinate systems of both pose calculation methods are congruent. If the pivot point of both coordinate systems is not identical, the resulting 3D image might show objects to be shifting with head motion. We provide a calibration tool for ART and Optitrack tracking to calibrate the outside-in pivot point to the inside-out pivot point.

Inside-out fusion (for advanced users)

In order to further improve the smoothness of the head pose tracking we extended our standard sensor fusion (outside-in with IMU fusion) by incorporating inside-out tracking data into the position calculation.

Our purpose is to combine the advantages of the outside-in tracking of providing a global reference, with the high responsiveness of the inside-out tracking.

This is done by high-pass filtering the inside-out position information and adding it to low-pass filtered outside-in tracking data. As a result, fast motions of the headset are tracked using the inside-out tracking, while the output is "pulled" towards the slower globally referenced position provided by the outside-in tracking.

Glossary:

• Inside-out tracking: the built-in tracking of Meta Quest. Advantage of this tracking method: smooth and responsive. Disadvantage: Doesn't work in some environments, no global reference shared with other HMDs.

• Outside-in tracking: External camera-based tracking such as ART or Optitrack. Advantages: Very reliable, provides global reference to all HMDs. Disadvantage: Might introduce latency when sending data from tracking server to HMDs.

• IMU: Inertial measurement unit containing a gyroscope to measure angular velocity and an accelerometer to measure acceleration of the HMD. IMU data is very responsive and is therefore fused with optical tracking data to improve latency and allow for pose prediction.

To activate inside-out fusion add the following configuration block:

    "insideOutFusion": {
      "dataEndpoint": "tcp://*:8799",
      "inputEndpoints": [
        "inproc://fused_pose_source",
        "tcp://localhost:8899"
      ],
      "settings": {
        "ioHpWeight": 1,
        "ioLpWeight": 0,
        "optLpWeight": 0,
        "useIOHeight": false,
        "useIOHorizontal": false
      }
    }

Make sure to set the data endpoint of the main fusion block to inproc://fused_pose_source.

The FusionHub configuration contains several parameters that allow the tuning of the inside-out fusion:

Example configurations

Example configurations for the outside-in / inside-out filter:

"ioHpWeight": 0,
"ioLpWeight": 0.9995,
"optLpWeight": 0.9995,
"useIOHeight": false,
"useIOHorizontal": true

"ioHpWeight": 1,
"ioLpWeight": 0,
"optLpWeight": 0,
"useIOHeight": true,
"useIOHorizontal": false

"ioHpWeight": 1,
"ioLpWeight": 0,
"optLpWeight": 0,
"useIOHeight": false,
"useIOHorizontal": false

Advanced intercalibration (only on Meta Quest)

Example videos

Motive

Intercalibratiotool: https://www.dropbox.com/scl/fi/1nkt3acbq69omehzz0j28/Intercalibration-tool-cryptolens.mp4?rlkey=qtg49h7kzbhcs4cz32yrnx7vn&dl=0

DTrack

PC screen recording: https://www.dropbox.com/scl/fi/6ouhtkac6409iekn51hbc/setupLPVRAIR-PC.mp4?rlkey=7ib34as0bfz4veqw69u0s6tga&st=yzvh9g4g&dl=0

Quest screen recording: https://www.dropbox.com/scl/fi/8fpsoybeb67sfn6x35ijt/setupLPVRAIR-Quest2.mp4?rlkey=xy2s5irxaguy5mw7ci8ep8tks&st=118r4rw8&dl=0

Software Required

PC

1. Optical system software: Motive / DTrack software

2. Oculus desktop software

3. SteamVR (download Steam Client from official site, and then follow instructions here)

4. lh_opt_calib.exe
   Dongled ← requires dongle hardware
   Cryptolens ← requires Internet connection on the 1st activation call

Procedure Overview

There are 5 sections in total for the setup:

1. Create a new marker body file for the headset

2. Connect PC to the headset through SteamVR

3. Generate a calibrated body with lh_opt_calib

4. Load the calibrated body file to the optical tracking software

5. Start the FusionHub

I - Create a new marker body file for the headset

Motive

1. In the perspective view, select the markers that you wish to group into the new rigid body.

2. Go to View → Builder Pane, give the new rigid body a new name.

3. Go to View → Assets Pane, select the newly created rigid body, right click on it and export it.

DTrack

1. Go to Tracking → Body Administration, increase the count by 1, and give your new tracking body a name. Then click “Calibrate”.

2. Click “calibrate” again and gently move and roll your headset in front of the tracking camera.

3. Save the body file by clicking “save file(s)”, select your headset body and save it to an empty folder.

II - Connect PC to the headset through SteamVR

1. On the headset, go to settings → system → Quest Link. Pairing with the PC and click the launch icon. This would start the Oculus software on the PC.

   

2. Then on the headset, start SteamVR. This will start the SteamVR on your PC, also in your headset you will enter into SteamVR’s default scene automatically.

III - Generate a calibrated body with lh_opt_calib

Motive

1. In the command line please go to the directory of lh_opt_calib and execute the following:

   lh_opt_calib.exe --opticalSystem motive --bodyId 6 --localAddress 192.168.1.18 --remoteAddress 192.168.1.18 --bodyFile "C:\Users\lpresearch\Desktop\Quest2TestBodyFile\standard b06.motive"

   bodyId to your headset’s tracking ID,
   localAddress and remoteAddress is the setting for NatNet client
   bodyFile to the file path (from section 1 step 3)

DTrack

1. Start the ART tracking by clicking “start” in DTrack software.

2. In the command line please go to the directory of lh_opt_calib and execute the following:

   lh_opt_calib.exe --opticalSystem dtrack --port 5012 --bodyId 6 --bodyFile "C:\Users\lpresearch\Desktop\Quest2TestBodyFile\standard b06.txt"

   set port to DTrack’s port (can be found in DTrack’s Tracking → Output)
   bodyId to your headset’s tracking ID in DTrack
   bodyFile to the file path (from section 1 step 3)

3. Rotate the headset in front of the tracking camera to collect 10 poses. If in the end all 5 outputs looks close, this section is considered finished. Otherwise please retry step 4 and 5.

4. Exit the SteamVR on PC. Exit the Oculus software on your headset by clicking “Quit” in the menu for Quest AirLink.

   

IV - Load the calibrated body file to the optical tracking software

Motive

1. Remove the rigid body you created just now. Otherwise Motive does NOT overwrite it with the calibrated one.

2. Drag and drop the newly generated body file xxx_HORAUD.motive into the Assets Pane in the Motive software.

DTrack

1. In the DTrack software on PC, click “stop”.

2. Go to Tracking → Body Calibration, select your headset body in the dropdown menu, and click “Load file(s)” to load the calibrated body file xxx_HORAUD.txt under the same directory as your exported body file in section 1.

How to inter-calibration without precalibrated targets

• We highly recommend purchasing precalibrated targets. If you don’t have precalibrated targets, please follow the instructions in the videos below.

Dtrack calibration : https://www.dropbox.com/scl/fi/cjmaykw5rt5263euqo0z1/Dtrack-Calibration-LPVR-AIR.MP4?rlkey=5y829p6wsi5z6dfpz8vtcpbze&st=3kkd916n&dl=0

Procedure Overview

1. Open the ART DTrack3 software and connect to the controller
   

2. Open "Body Administration" under Tracking > Body Administration
   

3. Select the body you wish to calibrate (or create a new one by increasing the "number of bodies" counter)

• Give it a name if desired

• Once it is selected, click on "Custom calibration" at the bottom of the window
  

4. In the "Custom Body Calibration" window that appears, make sure that the `standard` body type is selected
   

5. Click on Calibrate.

• After a short countdown, a new window will appear, with a progress bar and a counter. Simply move the headset with the markers attached as long as the progress bar is not complete

• Make sure the counter displays the number of markers attached to the headset
  

6. Close the calibration windows and open the "Body Adjustment" window from the main window under Tracking > Body adjustment
   

7. Select the newly calibrated body from the drop-down list at the top of that window
   

8. Place the headset in view of the ART cameras, aligned the same way that the room was calibrated.
   

9. Toggle on the "use tracked orientation" box at the bottom of the window.
   

10. Click on "Set origin to COG" and "Adjust axes to room" from the panel on the right side. You can now uncheck "use tracked orientation"

11. If needed, adjust manually the center of the headset relative to the marker's positions using the "Body position" and "Body orientation" sections of the right panel.

12. Click on "Apply", then "OK". All done!

FusionHub intercalibration : https://www.dropbox.com/scl/fi/di0dzqcdvhmm4m67wrtsf/FusionHub-Intercalibration-LPVR-AIR.mp4?rlkey=yrxlgfok7ghazf6kgemetd00k&st=uorf365v&dl=0

Procedure Overview

• Pre-requisites

  • Make sure FusionHub is installed and configured to receive IMU and optical data.

  • You will also need a computer connected to the same network as the headset.

  • For this demo, we will use LP-ALVR to provide IMU data to FusionHub.
    

•  Intercalibration

1. Launch FusionHub on the headset
   

2. Connect FusionHub to the GUI application using the internal IP address of the headset

• To do so, input the headset IP address into the "Server URL" field, at the top of the GUI. If the GUI window is not big enough, you might have to click the three-dot icon at the top right

• There should be a default port pre-specified, change it if needed. If you do not know FusionHub's port, leave it to the default value of `19358`

• You will know when you are connected when the status color next to the "Server URL" field turns green
  

3. Move to the "Base Configurations" panel, either from the appropriate icon on the Dashboard or through the side menu (You might have to click the hamburger icon on the top left if your window is not big enough)
   

4. Turn on optical tracking and make sure that the `nOptical` counter (displayed under the "Auto Calibration" section of the "Base Configurations" panel) is increasing
   

5. Turn on IMU data and make sure that the `nIMU` counter is increasing

• For this demo, we use LP-ALVR. the first step is to launch the alvr client on the headset

• Once the client is awaiting a connection, we launch the streamer from the PC we wish to stream. This will prompt us to launch SteamVR if it is not already launched.

• After SteamVR is operational (this might involve an automatic restart), the streamer should detect the headset and ask if we want to trust it. Clicking on "trust" should start the connection and FusionHub should start receiving IMU data.
  

6. At the bottom of the "Intercalibration" section, check the "Recalibrate on next restart" checkbox, and click on "Restart" right underneath
   

7. When FusionHub boots again, rotate the headset around not too quickly, but with big movements, marking with regular pauses to let the tracking catch up. If you see the `nUsedPoses` counter under the "Intercalibration" section go up, that means you are probably doing it correctly
   

8. When the `nUsedPoses` counter reaches 50 (it might take a few more poses), the computed calibration value should appear under the `quaternion` field

• If the `nUsedPoses` counter goes back to 0, it means that the computation did not converge to an appropriate calibration. Usually, this means you rotated the headset too quickly
  

9. From there click on "Apply Intercalibration Results". Then, on the top of the "Configuration section", click on "Set", then "Save". This will cause FusionHub to restart. Hopefully, everything works now!

References

• ALVR open source project: https://github.com/alvr-org/ALVR

• FusionHub documentation: FusionHub Manual