LPVR-AIR Manual

Table of contents

Introduction

The purpose of LPVR-AIR is to wirelessly stream image data from a SteamVR application such as Autodesk VRED to the Meta Quest 2 or Pro. LP-Research's FusionHub software in combination with the open-source application ALVR fulfill this purpose well.

ALVR by default uses the internal inside-out tracking of Meta Quest for pose calculation. LPVR-AIR exchanges the Quest’s native inside-out tracking with combined IMU and ART / Optitrack outside-in tracking to allow simulatenous, spatially synchronized operation of several HMDs in large tracking volumes.

To make the tracking functionality of FusionHub available to standalone augmented and virtual reality headsets, it can be integrated with Android-compatible OpenXR HMDs. This works via a customized version of the ALVR open source project. ALVR allows streaming image data wirelessly from a host computer and interfaces to 3D content engines through SteamVR. While the original ALVR client was built to work on Meta Quest HMDs, ALVR works in principle on any OpenXR compatible headset.

We use a thin client library to receive IMU data from the HMD API, pass it to FusionHub, process it there and then re-inject the information into the video pipeline of the headset. Depending on the type of HMD this happens within the ALVR client’s standard interface or in a separate hardware-specific API layer.

The overall system consists of several applications running at the same time. As the development of this application is still work-in-progress, starting and configuring this solution can be a bit cumbersome. We are working on making the process easier as we move along.

See the illustration below for a block diagram of the overall system:

Open

System components

Applications

The following applications need to be started on the head mounted display and the host computer. They should all the included in the installation package that you received from us. We will discuss the order of starting these applications and what their status output should be below.

On the headset:

On the host computer:

Authentification

FusionHub authenticates itself via the GUI client application. In order to run the client application make sure to insert the LPVR USB dongle into the host computer. After the GUI client is connected to FusionHub on the HMD, FusionHub will start streaming pose data to ALVR. Alternatively we offer the possibility to use use an online licensing system that unlocks HMD nodes via the internet.

Running LPVR-AIR

Installation

Install the FusionHub APK and ALVR client APK on the headset using a side-loading tool like Sidequest. In case of a Meta Quest HMD this will require you to put the HMD into developer mode. See here the steps for putting the HMD into developer mode: https://developer.oculus.com/documentation/native/android/mobile-device-setup/

The FusionHub GUI client and ALVR server can be started on the host PC without further installation, they can be run right out of the deployment folder.

Preparation of Headset

There are a few steps that we need to consider when starting the system. Some of these are due to limitations of our overall system implementation, some are necessary because of issues introduced by the Quest’s firmware.

• Make sure you turn the guardian (ie. the automatic tracking boundary detection for the Quest’s internal tracking) is turned off in the developer settings of the HMD.

• If inside-out tracking is not being used, we recommend to switch headset tracking off ie. switch to 3DOF tracking in the Quest system settings. This will free additional CPU performance for the decoding of the video stream and therefore improve image quality.

• When powering-up or re-starting the HMD, it will automatically turn headset tracking to ON. If not needed, turn headset tracking to OFF to save computing resources.

• When powering-up or re-starting the HMD, it will not automatically connect to a previously saved network, if the network is not connected to the internet. In this case manually connect to the designated access point after start-up. Internet is not needed for operation, this is a bug in Meta’s firmware. This is an issue that might be solved by a software update from Meta.

• To reduce impact of latency and jitter from optical tracking, turn on some smoothing and prediction in the ART DTrack or Optitrack Motive settings.

• If needed, cover the proximity sensor in the center between the lenses of the HMD with a piece of dark tape to prevent the HMD from going to sleep. Generally LPVR-AIR is able to deal with the HMD going to sleep and waking up, but in case you discover any issues the tape will help.

Start-up

FusionHub

• Start FusionHub on the HMD. A window showing the FusionHub console output should open.

• Start the FusionHub GUI client on the host computer

• Connect the GUI client to FusionHub on the HMD. Make sure HMD and host are in the same subnet. Enter the the correct IP of the HMD in the client before pressing connect.

• Adjust parameter blocks as needed. Refer to the description of FusionHub BASE for configuration options. Note the following input and output ports that are hard-coded in the ALVR FusionHub API layer. These are already correctly set in the default configuration file installed with the FusionHub APK, so usually there is no need to change them.

• If it’s not running yet make sure to start and configure your optical tracking system. Once optical data is streamed to FusionHub, the nOptical counter in the GUI should be increasing.

ALVR

• Start the ALVR server on the host. While the ALVR server starts up, it will automatically run SteamVR.

• Start the ALVR client on the HMD. The HMD should be shown in the list of ALVR clients in the ALVR server application. In some cases you need to click the Trust button in the application to start streaming.

• Once streaming starts, you should see the SteamVR default envinronment through the headset. Check if the nIMU counter in the FusionHub GUI is increasing. If both nOptical and nIMU are increasing then the communication between ALVR, optical tracking and FusionHub is working.

• You can now use and calibrate the system as described in the FusioHub BASE section.

Optical tracking systems

Configuration options

Advanced Realtime Tracking (ART):

FusionHub works with all ART tracking systems, based on their DTrack tracking software.

Optitrack:

FusionHub works with all Optitrack tracking systems based on their Motive tracking software.

VICON:

FusionHub consumes VICON’s DataStream protocol. Communication has been tested with their Shogun software.

Antilatency:

FusionHub connects directly to Antilatency’s USB or wireless trackers.

Marker adjustment

The optical system is the tracking reference, its pose is what is received by the visualization backend. The orientation of the IMU sensor is calibrated relative to the optical markers on the HMD. Therefore it is important to set up the tracking body or rigid body in the optical tracking software (DTrack, Motive etc.) in a way that its axes align with the optical axes of the head mounted display.

Open SteamVR coordinate system

We will add to this section soon. In the meantime refer to this page for ART setups and this page for OptiTrack setups from the LPVR documentation.

Optitrack notes

In order to avoid excessive buffering and data loss, make sure to reduce the amount of data being streamed from Motive. We recommend the streaming settings below. Make sure to set the local interface IP to the IP of the network connection that is being used for communicating with the HMD. It is a common mistake to not set this IP correctly. If the correct IP doesn’t show up, restart Motive.

Intercalibration

Example videos

Motive

Intercalibration tool: https://www.dropbox.com/scl/fi/1nkt3acbq69omehzz0j28/Intercalibration-tool-cryptolens.mp4?rlkey=qtg49h7kzbhcs4cz32yrnx7vn&dl=0

DTrack

PC screen recording: https://www.dropbox.com/s/oqt3blgb18qwxdc/setupLPVRAIR-PC.mp4?dl=0

Quest screen recording:

Software Required

PC

1. Optical system software: Motive / DTrack software

2. Oculus desktop software

3. SteamVR (download Steam Client from official site, and then follow instructions here)

4. lh_opt_calib.exe
   Dongled ← requires dongle hardware
   Cryptolens ← requires Internet connection on the 1st activation call

Procedure Overview

There are 5 sections in total for the setup:

1. Create a new marker body file for the headset

2. Connect PC to the headset through SteamVR

3. Generate a calibrated body with lh_opt_calib

4. Load the calibrated body file to the optical tracking software

5. Start the FusionHub

I - Create a new marker body file for the headset

Motive

1. In the perspective view, select the markers that you wish to group into the new rigid body.

2. Go to View → Builder Pane, give the new rigid body a new name.

3. Go to View → Assets Pane, select the newly created rigid body, right click on it and export it.

DTrack

1. Go to Tracking → Body Administration, increase the count by 1, and give your new tracking body a name. Then click “Calibrate”.

2. Click “calibrate” again and gently move and roll your headset in front of the tracking camera.

3. Save the body file by clicking “save file(s)”, select your headset body and save it to an empty folder.

II - Connect PC to the headset through SteamVR

1. On the headset, go to settings → system → Quest Link. Pairing with the PC and click the launch icon. This would start the Oculus software on the PC.

2. Then on the headset, start SteamVR. This will start the SteamVR on your PC, also in your headset you will enter into SteamVR’s default scene automatically.

III - Generate a calibrated body with lh_opt_calib

Motive

1. In the command line please go to the directory of lh_opt_calib and execute the following:

   bodyId to your headset’s tracking ID,
   localAddress and remoteAddress is the setting for NatNet client
   bodyFile to the file path (from section 1 step 3)

DTrack

1. Start the ART tracking by clicking “start” in DTrack software.

2. In the command line please go to the directory of lh_opt_calib and execute the following:

   set port to DTrack’s port (can be found in DTrack’s Tracking → Output)
   bodyId to your headset’s tracking ID in DTrack
   bodyFile to the file path (from section 1 step 3)

3. Rotate the headset in front of the tracking camera to collect 10 poses. If in the end all 5 outputs looks close, this section is considered finished. Otherwise please retry step 4 and 5.

4. Exit the SteamVR on PC. Exit the Oculus software on your headset by clicking “Quit” in the menu for Quest AirLink.

IV - Load the calibrated body file to the optical tracking software

Motive

1. Remove the rigid body you created just now.

2. Drag and drop the newly generated body file xxx_HORAUD.motive into the Assets Pane in the Motive software.

DTrack

1. In the DTrack software on PC, click “stop”.

2. Go to Tracking → Body Calibration, select your headset body in the dropdown menu, and click “Load file(s)” to load the calibrated body file xxx_HORAUD.txt under the same directory as your exported body file in section 1.

Troubleshooting

The best way to diagnose what is going wrong with the system if something doesn’t work as expected is to look at the log output of FusionHubLauncher. The log can be recorded by connecting your HMD to a PC using a USB (most likely USB-C) cable.

For this purpose download the Android platform tools for Windows from here:

Copy the platform tools files to a folder and open a command prompt in that folder.

Check if the HMD is detected by your computer by entering adb devices. If your devices is not detected or marked as unauthorized make sure you have the correct USB driver for your HMD installed and acknowledged USB access from the host computer in the Android GUI on the HMD.

Enter adb logcat | findstr fusionhub to stream log data from the device to your command line. Only data from the FusionHubLauncher application will be displayed. The initialization log will be displayed when FusionHubLauncher is first started and everytime you restart it using the Restart button in the GUI.

Challenges and limitations of using Meta Quest HMDs

External pose input / internal late latching conflict

By default Meta Quest HMDs use their built-in inside-out tracking for measuring the user's head pose. For this project we needed to exchange the built-in tracking with our customized pose estimation. Generally the Quest HMD's software is not made to allow this application. Correct VR rendering is optimized to work with the built-in tracking and alternative ways of pose input are not supported.

We were able to create a workaround for this problem to some degree, by exchanging the pose calculated by the internal system with the pose calculated by FusionHub, before it is sent to the server for rendering. On the rendering side this works well, but unfortunately creates a conflict between the pose reported by the HMD firmware and the pose used for rendering that affects the quality of the HMD's pose prediction, late latching / asynchronous time/space warp. This conflict is perceived as uncomfortable, slight jittering of the image and overcompensation of head motions.

A way to reduce this jittering effect is to reduce the prediction time of the internal pose calculation. This doesn't fully solve the problem though, as decreased prediction time also reduces the responsiveness of the internal late latching. This in turn leads to blurring at edges of the image when the user turns his head rapidly.

Therefore we tried to find a compromise between image stability and late latching capability. If this compromise works well, strongly depends on user requirements and quality of the wifi connection established to the HMD. A perfect solution is unfortunatly not possible without support from the HMD manufacturer Meta.

Wifi environment quality

LPVR-AIR transmits images from the server to the HMD through a regular wifi connection. Usually a 5G band is used, in the optimum case we switch to wifi 6e. In environments without much wifi interference, ie. other devices using the same wifi bands, this works very well. Crowded wifi environments limit the bandwidth of the used wifi transmission. This can lead to unpredictable loss of image and tracking quality. Examples of crowded spaces are public locations such as exhibition. Beware!

The Meta Quest firmware doesn't allow using a wired ethernet connection to mitigate the issue to provide a quick fix to this problem in urgent situations. Again, a clean execution of this project is prevented by inflexibility of Meta's software. Unfortunately there are no alternative wireless HMDs on the market that allow the modifications we need for an optimum implementation.

In an ideal setup, with several HMDs being used, each HMD uses a separate wifi 6e channel, with the selected channels as far apart as possible.

Optical tracking parsing latency

Due to limited wifi bandwidth and computing power limitations on the HMD pose information streamed from the the optical tracking system is parsed on the HMD with a significant delay. So far we have not found a way to reduce this delay. As described in the sensor fusion section, we added input from native inside-out tracking to the fusion in order to compensate for this latency.

HMD built-in CPU performance limitation

The Meta Quest series headsets are stand-alone HMDs that integrate all computing hardware in one device. Due to size and heat emission restrictions, the amount of computing power they provide is limited. For the operation of LPVR, significant compute is required to decode the image stream coming from the rendering PC to the HMD. For high resolution images with a high framerate, performance bottlenecks can happen, resulting in reduced image quality or tracking artefacts. we discovered during experiments that turning the internal inside-out tracking of the HMD off, can improve the performance of the rest of the system. Note that in this case, inside-out tracking can’t be used to supplement pose calculation by FusionHub.

Inside-out / outside-in tracking misalignment

When fusing inside-out and outside-in tracking it is important that the coordinate systems of both pose calculation methods are congruent. If the pivot point of both coordinate systems is not identical, the resulting 3D image might show objects to be shifting with head motion. We provide a calibration tool for ART and Optitrack tracking to calibrate the outside-in pivot point to the inside-out pivot point.

IMU - outside-in - inside-out fusion

In order to further improve the smoothness of the head pose tracking we extended our standard sensor fusion (outside-in with IMU fusion) by incorporating inside-out tracking data into the position calculation.

Our purpose is to combine the advantages of the outside-in tracking of providing a global reference, with the high responsiveness of the inside-out tracking.

This is done by high-pass filtering the inside-out position information and adding it to low-pass filtered outside-in tracking data. As a result, fast motions of the headset are tracked using the inside-out tracking, while the output is "pulled" towards the slower globally referenced position provided by the outside-in tracking.

Glossary:

• Inside-out tracking: the built-in tracking of Meta Quest. Advantage of this tracking method: smooth and responsive. Disadvantage: Doesn't work in some environments, no global reference shared with other HMDs.

• Outside-in tracking: External camera-based tracking such as ART or Optitrack. Advantages: Very reliable, provides global reference to all HMDs. Disadvantage: Might introduce latency when sending data from tracking server to HMDs.

• IMU: Inertial measurement unit containing a gyroscope to measure angular velocity and an accelerometer to measure acceleration of the HMD. IMU data is very responsive and is therefore fused with optical tracking data to improve latency and allow for pose prediction.

The FusionHub configuration contains several parameters that allow the tuning of the inside-out fusion:

Example configurations for the outside-in / inside-out filter:

Network setup

Router

In order to establish high bandwidth communication between the host and Quest 2 / Pro we recommend setting up a 5GHz or for optimum performance a 6GHz (wifi 6e) wifi router with 2 channels per HMD. We recommend the tp-link AXE16000 quad-band 16-stream wi-fi 6e router.

Network topography

We recommend to set up a simple network structure, to minimze potential error sources in the installation process as shown in the image below.

References

• ALVR open source project:

• FusionHub documentation: