LPVR-AIR is based on FusionHub. This is a condensed version of the FusionHub manual that focuses on the VR use-case. Find the full FusionHub documentation here.
The purpose of LPVR-AIR is to wirelessly stream image data from a SteamVR application such as Autodesk VRED to the Meta Quest 2 or Pro. LP-Research's FusionHub software in combination with the open-source application ALVR fulfill this purpose well.
ALVR by default uses the internal inside-out tracking of Meta Quest for pose calculation. LPVR-AIR exchanges the Quest’s native inside-out tracking with combined IMU and ART / Optitrack outside-in tracking to allow simulatenous, spatially synchronized operation of several HMDs in large tracking volumes.
To make the tracking functionality of FusionHub available to standalone augmented and virtual reality headsets, it can be integrated with Android-compatible OpenXR HMDs. This works via a customized version of the ALVR open source project. ALVR allows streaming image data wirelessly from a host computer and interfaces to 3D content engines through SteamVR. While the original ALVR client was built to work on Meta Quest HMDs, ALVR works in principle on any OpenXR compatible headset.
We use a thin client library to receive IMU data from the HMD API, pass it to FusionHub, process it there and then re-inject the information into the video pipeline of the headset. Depending on the type of HMD this happens within the ALVR client’s standard interface or in a separate hardware-specific API layer.
The overall system consists of several applications running at the same time. As the development of this application is still work-in-progress, starting and configuring this solution can be a bit cumbersome. We are working on making the process easier as we move along.
See the illustration below for a block diagram of the overall system:
The following applications need to be started on the head mounted display and the host computer. They should all the included in the installation package that you received from us. We will discuss the order of starting these applications and what their status output should be below.
On the headset:
Receives IMU data from ALVR client
Receives optical tracking data
Sends sensor fusion result to ALVR client
Receives image data from ALVR server
Connects to FusionHub server
Sends pose information to ALVR server
On the host computer:
Connects to FusionHub server on HMD
Receives pose information from client
Sends pose information to SteamVR
Sends image data to ALVR client
Receives image information from SteamVR
FusionHub authenticates itself via the GUI client application. In order to run the client application make sure to insert the LPVR USB dongle into the host computer. After the GUI client is connected to FusionHub on the HMD, FusionHub will start streaming pose data to ALVR. Alternatively we offer the possibility to use use an online licensing system that unlocks HMD nodes via the internet.
There are a few steps that we need to consider when starting the system. Some of these are due to limitations of our overall system implementation, some are necessary because of issues introduced by the Quest’s firmware.
If inside-out tracking is not being used, we recommend to switch headset tracking off ie. switch to 3DOF tracking in the Quest system settings. This will free additional CPU performance for the decoding of the video stream and therefore improve image quality.
When powering-up or re-starting the HMD, it will automatically turn headset tracking to ON. If not needed, turn headset tracking to OFF to save computing resources.
When powering-up or re-starting the HMD, it will not automatically connect to a previously saved network, if the network is not connected to the internet. In this case manually connect to the designated access point after start-up. Internet is not needed for operation, this is a bug in Meta’s firmware.
To reduce impact of latency and jitter from optical tracking, turn on some smoothing and prediction in the ART DTrack or Optitrack Motive settings.
The FusionHub GUI client and ALVR server can be started on the host PC without further installation, they can be run right out of the deployment folder.
Start FusionHub on the HMD. A window showing the FusionHub console output should open.
Start the FusionHub GUI client on the host computer
Connect the GUI client to FusionHub on the HMD. Make sure HMD and host are in the same subnet. Enter the the correct IP of the HMD in the client before pressing connect.
Adjust parameter blocks as needed. Refer to the description of FusionHub BASE for configuration options. Note the following input and output ports that are hard-coded in the ALVR FusionHub API layer. These are already correctly set in the default configuration file installed with the FusionHub APK, so usually there is no need to change them.
Fused pose data
If it’s not running yet make sure to start and configure your optical tracking system. Once optical data is streamed to FusionHub, the nOptical counter in the GUI should be increasing.
Start the ALVR server on the host. While the ALVR server starts up, it will automatically run SteamVR.
Start the ALVR client on the HMD. The HMD should be shown in the list of ALVR clients in the ALVR server application. In some cases you need to click the Trust button in the application to start streaming.
Once streaming starts, you should see the SteamVR default envinronment through the headset. Check if the nIMU counter in the FusionHub GUI is increasing. If both nOptical and nIMU are increasing then the communication between ALVR, optical tracking and FusionHub is working.
You can now use and calibrate the system as described in the FusioHub BASE section.
Optical tracking systems
Advanced Realtime Tracking (ART):
FusionHub works with all ART tracking systems, based on their DTrack tracking software.
The optical system is the tracking reference, its pose is what is received by the visualization backend. The orientation of the IMU sensor is calibrated relative to the optical markers on the HMD. Therefore it is important to set up the tracking body or rigid body in the optical tracking software (DTrack, Motive etc.) in a way that its axes align with the optical axes of the head mounted display.
SteamVR coordinate system
Possible marker arrangement on HMD
Markers as arranged in optical tracking tool (ART DTrack)
In order to avoid excessive buffering and data loss, make sure to reduce the amount of data being streamed from Motive. We recommend the streaming settings below. Make sure to set the local interface IP to the IP of the network connection that is being used for communicating with the HMD. It is a common mistake to not set this IP correctly. If the correct IP doesn’t show up, restart Motive.
set port to DTrack’s port (can be found in DTrack’s Tracking → Output) bodyId to your headset’s tracking ID in DTrack bodyFile to the file path (from section 1 step 3)
Rotate the headset in front of the tracking camera to collect 10 poses. If in the end all 5 outputs looks close, this section is considered finished. Otherwise please retry step 4 and 5.
Exit the SteamVR on PC. Exit the Oculus software on your headset by clicking “Quit” in the menu for Quest AirLink.
IV - Load the calibrated body file to the optical tracking software
Remove the rigid body you created just now.
Drag and drop the newly generated body file xxx_HORAUD.motive into the Assets Pane in the Motive software.
In the DTrack software on PC, click “stop”.
Go to Tracking → Body Calibration, select your headset body in the dropdown menu, and click “Load file(s)” to load the calibrated body file xxx_HORAUD.txt under the same directory as your exported body file in section 1.
The best way to diagnose what is going wrong with the system if something doesn’t work as expected is to look at the log output of FusionHubLauncher. The log can be recorded by connecting your HMD to a PC using a USB (most likely USB-C) cable.
Copy the platform tools files to a folder and open a command prompt in that folder.
Check if the HMD is detected by your computer by entering adb devices. If your devices is not detected or marked as unauthorized make sure you have the correct USB driver for your HMD installed and acknowledged USB access from the host computer in the Android GUI on the HMD.
Enter adb logcat | findstr fusionhub to stream log data from the device to your command line. Only data from the FusionHubLauncher application will be displayed. The initialization log will be displayed when FusionHubLauncher is first started and everytime you restart it using the Restart button in the GUI.
Challenges and limitations of using Meta Quest HMDs
External pose input / internal late latching conflict
By default Meta Quest HMDs use their built-in inside-out tracking for measuring the user's head pose. For this project we needed to exchange the built-in tracking with our customized pose estimation. Generally the Quest HMD's software is not made to allow this application. Correct VR rendering is optimized to work with the built-in tracking and alternative ways of pose input are not supported.
We were able to create a workaround for this problem to some degree, by exchanging the pose calculated by the internal system with the pose calculated by FusionHub, before it is sent to the server for rendering. On the rendering side this works well, but unfortunately creates a conflict between the pose reported by the HMD firmware and the pose used for rendering that affects the quality of the HMD's pose prediction, late latching / asynchronous time/space warp. This conflict is perceived as uncomfortable, slight jittering of the image and overcompensation of head motions.
A way to reduce this jittering effect is to reduce the prediction time of the internal pose calculation. This doesn't fully solve the problem though, as decreased prediction time also reduces the responsiveness of the internal late latching. This in turn leads to blurring at edges of the image when the user turns his head rapidly.
Therefore we tried to find a compromise between image stability and late latching capability. If this compromise works well, strongly depends on user requirements and quality of the wifi connection established to the HMD. A perfect solution is unfortunatly not possible without support from the HMD manufacturer Meta.
Wifi environment quality
LPVR-AIR transmits images from the server to the HMD through a regular wifi connection. Usually a 5G band is used, in the optimum case we switch to wifi 6e. In environments without much wifi interference, ie. other devices using the same wifi bands, this works very well. Crowded wifi environments limit the bandwidth of the used wifi transmission. This can lead to unpredictable loss of image and tracking quality. Examples of crowded spaces are public locations such as exhibition. Beware!
The Meta Quest firmware doesn't allow using a wired ethernet connection to mitigate the issue to provide a quick fix to this problem in urgent situations. Again, a clean execution of this project is prevented by inflexibility of Meta's software. Unfortunately there are no alternative wireless HMDs on the market that allow the modifications we need for an optimum implementation.
In an ideal setup, with several HMDs being used, each HMD uses a separate wifi 6e channel, with the selected channels as far apart as possible.
Optical tracking parsing latency
Due to limited wifi bandwidth and computing power limitations on the HMD pose information streamed from the the optical tracking system is parsed on the HMD with a significant delay. So far we have not found a way to reduce this delay. As described in the sensor fusion section, we added input from native inside-out tracking to the fusion in order to compensate for this latency.
HMD built-in CPU performance limitation
The Meta Quest series headsets are stand-alone HMDs that integrate all computing hardware in one device. Due to size and heat emission restrictions, the amount of computing power they provide is limited. For the operation of LPVR, significant compute is required to decode the image stream coming from the rendering PC to the HMD. For high resolution images with a high framerate, performance bottlenecks can happen, resulting in reduced image quality or tracking artefacts. we discovered during experiments that turning the internal inside-out tracking of the HMD off, can improve the performance of the rest of the system. Note that in this case, inside-out tracking can’t be used to supplement pose calculation by FusionHub.
Inside-out / outside-in tracking misalignment
When fusing inside-out and outside-in tracking it is important that the coordinate systems of both pose calculation methods are congruent. If the pivot point of both coordinate systems is not identical, the resulting 3D image might show objects to be shifting with head motion. We provide a calibration tool for ART and Optitrack tracking to calibrate the outside-in pivot point to the inside-out pivot point.
IMU - outside-in - inside-out fusion
In order to further improve the smoothness of the head pose tracking we extended our standard sensor fusion (outside-in with IMU fusion) by incorporating inside-out tracking data into the position calculation.
Our purpose is to combine the advantages of the outside-in tracking of providing a global reference, with the high responsiveness of the inside-out tracking.
This is done by high-pass filtering the inside-out position information and adding it to low-pass filtered outside-in tracking data. As a result, fast motions of the headset are tracked using the inside-out tracking, while the output is "pulled" towards the slower globally referenced position provided by the outside-in tracking.
Inside-out tracking: the built-in tracking of Meta Quest. Advantage of this tracking method: smooth and responsive. Disadvantage: Doesn't work in some environments, no global reference shared with other HMDs.
Outside-in tracking: External camera-based tracking such as ART or Optitrack. Advantages: Very reliable, provides global reference to all HMDs. Disadvantage: Might introduce latency when sending data from tracking server to HMDs.
IMU: Inertial measurement unit containing a gyroscope to measure angular velocity and an accelerometer to measure acceleration of the HMD. IMU data is very responsive and is therefore fused with optical tracking data to improve latency and allow for pose prediction.
The FusionHub configuration contains several parameters that allow the tuning of the inside-out fusion:
Weight of low-pass filter applied to inside-out tracking after high-pass filter (0 = unmodified inside-out signal, 1 = remove inside-out signal)
Weight of high-pass filter applied to inside-out tracking (0 = remove inside-out signal, 1 = full inside-out signal applied to result)
Weight of low-pass filter applied to outside-in tracking (0 = unmodified outside-in signal, 1 = outside-in signal is removed)
Set this to true to take height (y-axis) value directly from inside-out tracking
Set this to true to include inside-out tracking in pose calculation. If true, filter parameters described above will be used.
Example configurations for the outside-in / inside-out filter:
Mainly inside-out, outside-in for global referencing: Tracking is based mainly on the built-in inside-out tracking. Outside-in tracking is used for global referencing.
Note: Turn headset tracking to ON in Quest settings. Room needs to be suitable for inside-out tracking.
In order to establish high bandwidth communication between the host and Quest 2 / Pro we recommend setting up a 5GHz or for optimum performance a 6GHz (wifi 6e) wifi router with 2 channels per HMD. We recommend the tp-link AXE16000 quad-band 16-stream wi-fi 6e router.
We recommend to set up a simple network structure, to minimze potential error sources in the installation process as shown in the image below.