LPVR-AIR Manual
Table of contents
- 1 Table of contents
- 2 Introduction
- 3 System components
- 3.1 Applications
- 3.2 Authentication
- 4 Running LPVR-AIR
- 4.1 Installation
- 4.2 Start-up
- 5 Optical tracking systems
- 6 Advanced troubleshooting
- 7 Network setup
- 7.1 Router
- 7.2 Network topography
- 7.3 Network performance
- 8 Challenges and limitations of using Meta Quest HMDs
- 9 Inside-out fusion (for advanced users)
- 10 Advanced intercalibration (only on Meta Quest)
- 10.1 Example videos
- 10.2 Software Required
- 10.2.1 PC
- 10.3 Procedure Overview
- 11 How to inter-calibration without precalibrated targets
- 11.1 Dtrack calibration :
- 11.1.1 Procedure Overview
- 11.2 FusionHub intercalibration :
- 11.2.1 Procedure Overview
- 11.2.1.1 Pre-requisites
- 11.2.1.2 Intercalibration
- 11.2.1 Procedure Overview
- 11.1 Dtrack calibration :
- 12 References
Introduction
LPVR-AIR is based on FusionHub. This is a condensed version of the FusionHub manual that focuses on the VR use-case. Find the full FusionHub documentation here.
The purpose of LPVR-AIR is to wirelessly stream image data from a SteamVR application such as Autodesk VRED to a wireless HMD like the Meta Quest or VIVE Focus. LP-Research's FusionHub software in combination with the open-source application ALVR fulfill this purpose well.
ALVR by default uses the internal inside-out tracking of Meta Quest for pose calculation. LPVR-AIR exchanges the Quest’s native inside-out tracking with combined IMU and ART / Optitrack outside-in tracking to allow simultaneous, spatially synchronized operation of several HMDs in large tracking volumes.
To make the tracking functionality of FusionHub available to standalone augmented and virtual reality headsets, it can be integrated with Android-compatible OpenXR HMDs. This works via a customized version of the ALVR open source project. ALVR allows streaming image data wirelessly from a host computer and interfaces to 3D content engines through SteamVR. While the original ALVR client was built to work on Meta Quest HMDs, ALVR works in principle on any OpenXR compatible headset.
We use a thin client library to receive IMU data from the HMD API, pass it to FusionHub, process it there and then re-inject the information into the video pipeline of the headset. Depending on the type of HMD this happens within the ALVR client’s standard interface or in a separate hardware-specific API layer.
The overall system consists of several applications running at the same time. As the development of this application is still work-in-progress, starting and configuring this solution can be a bit cumbersome. We are working on making the process easier as we move along.
See the illustration below for a block diagram of the overall system:
System components
Applications
The following applications need to be started on the head mounted display and the host computer. They should all the included in the installation package that you received from us. We will discuss the order of starting these applications and what their status output should be below.
On the headset:
Application | Purpose | Name (can vary by release) |
---|---|---|
FusionHub server |
|
|
ALVR client |
|
|
On the host computer:
Application | Purpose | Name |
---|---|---|
FusionHub GUI |
|
|
ALVR server |
|
|
Authentication
FusionHub authenticates itself via the GUI client application. In order to run the client application make sure to insert the LPVR USB dongle into the host computer. After the GUI client is connected to FusionHub on the HMD, FusionHub will start streaming pose data to ALVR. Alternatively we offer the possibility to use use an online licensing system that unlocks HMD nodes via the internet. By default we deliver the dongled version.
Running LPVR-AIR
Installation
Install the FusionHub APK and ALVR client APK on the headset using a side-loading tool like Sidequest. In case of a Meta Quest HMD this will require you to put the HMD into developer mode. See here the steps for putting the HMD into developer mode: https://developer.oculus.com/documentation/native/android/mobile-device-setup/
In case you’re using a VIVE Focus 3 headset you need to do something similar as described here: https://developer.vive.com/resources/hardware-guides/vive-focus-specs-user-guide/how-do-i-put-focus-developer-mode/
The FusionHub GUI client and ALVR server can be started on the host PC without further installation, they can be run right out of the deployment folder.
ALVR requires SteamVR to be set up on the host computer. If you haven’t installed it on your computer yet, please refer to the instructions here: https://store.steampowered.com/app/250820/SteamVR/
Make sure you turn the guardian (i.e. the automatic tracking boundary detection for the Quest’s internal tracking) is turned off in the developer settings of the HMD. For VIVE Focus use the equivalent setting in the Focus' configuration.
Start-up
FusionHub
Start FusionHub on the HMD. A window showing the FusionHub console output should open.
Start the FusionHub GUI client on the host computer
Connect the GUI client to FusionHub on the HMD. Make sure HMD and host are in the same subnet. Enter the the correct IP of the HMD in the client before pressing connect.
Once connected the FusionHub on the headset select base configuration to see the current configuration of FusionHub:
Adjust parameter blocks as needed. Refer to the description of FusionHub BASE for configuration options:
Note the following input and output ports that are hard-coded in the ALVR FusionHub API layer. These are already correctly set in the default configuration file installed with the FusionHub APK, so usually there is no need to change them.
Endpoint | Direction | Purpose |
---|---|---|
| Output | Fused pose data |
| Input | IMU data |
If it’s not running yet make sure to start and configure your optical tracking system. Once optical data is streamed to FusionHub, the
nOptical
counter in the GUI should be increasing.Default configuration script with optical input defined for ART DTrack:
{
"LicenseInfo": {
"LicenseKey": "",
"ResponseKey": ""
},
"settings": {
"websocketDataOutputRate": 20
},
"sinks": {
"VRPN": {
"settings": {
"inputEndpoints": [
"inproc://optical_data_source_1"
],
"settings": {
"deviceName": "FusionHub",
"port": 3883,
"tracker0": "HMD"
}
}
},
"fusion": {
"dataEndpoint": "tcp://*:8799",
"inputEndpoints": [
"inproc://optical_data_source_1",
"tcp://localhost:8898"
],
"settings": {
"Autocalibration": {
"minAgeS": 60,
"nSamplesForAutocalibration": 1500,
"nSamplesForSteady": 256,
"noiseRmsLimit": 0.02,
"steadyThresholdAverage": 0.2,
"steadyThresholdRms": 1
},
"Intercalibration": {},
"MotionDetection": {
"omegaLimit": 3,
"positionSampleInterval": 1000,
"rotationFilterAlpha": 0.9,
"timeToUnknown": 500
},
"SensorFusion": {
"alignment": {
"w": 0.990892966476337,
"x": 0.13458639604387848,
"y": 0.0005637732357904688,
"z": 0.004160907038605602
},
"orientationWeight": 0.005,
"predictionIntervalMs": 0,
"sggPointsEachSide": 5,
"sggPolynomialOrder": 5,
"tiltCorrection": null,
"yawWeight": 0.01
},
"runIntercalibration": false
},
"type": "ImuOpticalFusion"
}
},
"sources": {
"optical": {
"settings": {
"bodyIDs": [
1
],
"endpoints": [
"inproc://optical_data_source_1"
],
"port": 5000
},
"type": "DTrack"
}
}
}
Press the buttons ‘Set’ and ‘Save’ after changing the configuration script to make your changes active. It might take 1-2 seconds for FusionHub to reset.
In case you happen to enter an invalid configuration, FusionHub might not restart correctly. If you would like to reset your settings, just re-install the FusionHub APK.
Once the configuration is correct, you’ll most likely not have to touch the script again in the foreseeable future.
ALVR
Start the ALVR server on the host. While the ALVR server starts up, it will automatically run SteamVR.
Start the ALVR client on the HMD. The HMD should be shown in the list of ALVR clients in the ALVR server application. In some cases you need to click the
Trust
button in the application to start streaming.You can now use and calibrate the system as described in the FusionHub BASE section.
Here is a video tutorial how to perform the initial startup procedure:
https://vimeo.com/1024128202?share=copy
Optical tracking systems
Configuration options
Advanced Realtime Tracking (ART) DTrack
FusionHub works with all ART tracking systems, based on their DTrack tracking software.
"optical": {
"type": "DTrack",
"settings": {
"port": 5000,
"bodyIDs": [
1,
2,
3
],
"endpoints": [
"inproc://optical_data_source_1",
"tcp://*:8500",
"tcp://*:8501"
],
"objectNameMapping": {
"1": "QuestPro",
"2": "leftController",
"3": "rightController"
}
}
}
}
Adjust the body IDs as configured in DTrack. If you don’t plan to use hand controllers ignore the body IDs for the hand controllers (2 and 3 in the example above). Leave the endpoints and objectNameMapping blocks as shown above.
See this guide for further information about how to configure DTrack. Also, refer to this page for more information on ART.
Optitrack
FusionHub works with all Optitrack tracking systems based on their Motive tracking software.
"optical": {
"type": "Optitrack",
"settings": {
"connectionType": "Multicast",
"localAddress": "192.168.0.99",
"remoteAddress": "192.168.0.100",
"bodyIDs": [
1,
2,
3
],
"endpoints": [
"inproc://optical_data_source_1",
"tcp://*:8500",
"tcp://*:8501"
],
"objectNameMapping": {
"1": "QuestPro",
"2": "leftController",
"3": "rightController"
}
}
}
Adjust the body IDs as configured in Motive. If you don’t plan to use hand controllers ignore the body IDs for the hand controllers (2 and 3 in the example above). Leave the endpoints and objectNameMapping blocks as shown above. Local address needs to be set to the address of your headset, remote address is the address of the Motive machine.
Refer to this page for Optitrack setup from the LPVR documentation.
Advanced troubleshooting
The best way to diagnose what is going wrong with the system if something doesn’t work as expected is to look at the log output of FusionHubLauncher. The log can be recorded by connecting your HMD to a PC using a USB (most likely USB-C) cable.
For this purpose download the Android platform tools for Windows from here: https://developer.android.com/studio/releases/platform-tools
Copy the platform tools files to a folder and open a command prompt in that folder.
Check if the HMD is detected by your computer by entering adb devices
. If your devices is not detected or marked as unauthorized make sure you have the correct USB driver for your HMD installed and acknowledged USB access from the host computer in the Android GUI on the HMD.
Enter adb logcat | findstr fusionhub
to stream log data from the device to your command line. Only data from the FusionHubLauncher application will be displayed. The initialization log will be displayed when FusionHubLauncher is first started and every time you restart it using the Restart
button in the GUI.
Network setup
Router
In order to establish high bandwidth communication between the host and Quest 2 / Pro we recommend setting up a 5GHz or for optimum performance a 6GHz (WIFI 6e) WIFI router with 2 channels per HMD. We recommend the Eero WIFI 6E router.
Network topography
We recommend to set up a simple network structure, to minimize potential error sources in the installation process as shown in the image below.
Network performance
A WIFI 6E connection is recommended to achieve optimum performance:
Transmission speeds are expected to be around 2 Gbps for a stable 6 GHz connection.
A typical ALVR performance graph is shown below. Overall latencies in good environments should be between 70 and 90ms.
In SteamVR check Advanced Frame Timing for performance problems:
In case of a sufficient rendering performance the advanced frame timing window should look like the output below:
Challenges and limitations of using Meta Quest HMDs
External pose input / internal late latching conflict
By default Meta Quest HMDs use their built-in inside-out tracking for measuring the user's head pose. For this project we needed to exchange the built-in tracking with our customized pose estimation. Generally the Quest HMD's software is not made to allow this application. Correct VR rendering is optimized to work with the built-in tracking and alternative ways of pose input are not supported.
We were able to create a workaround for this problem to some degree, by exchanging the pose calculated by the internal system with the pose calculated by FusionHub, before it is sent to the server for rendering. On the rendering side this works well, but unfortunately creates a conflict between the pose reported by the HMD firmware and the pose used for rendering that affects the quality of the HMD's pose prediction, late latching / asynchronous time/space warp. This conflict is perceived as uncomfortable, slight jittering of the image and overcompensation of head motions.
A way to reduce this jittering effect is to reduce the prediction time of the internal pose calculation. This doesn't fully solve the problem though, as decreased prediction time also reduces the responsiveness of the internal late latching. This in turn leads to blurring at edges of the image when the user turns his head rapidly.
Therefore we tried to find a compromise between image stability and late latching capability. If this compromise works well, strongly depends on user requirements and quality of the WIFI connection established to the HMD. A perfect solution is unfortunately not possible without support from the HMD manufacturer Meta.
WIFI environment quality
LPVR-AIR transmits images from the server to the HMD through a regular WIFI connection. Usually a 5G band is used, in the optimum case we switch to WIFI 6E. In environments without much WIFI interference, i.e. other devices using the same WIFI bands, this works very well. Crowded WIFI environments limit the bandwidth of the used WIFI transmission. This can lead to unpredictable loss of image and tracking quality. Examples of crowded spaces are public locations such as exhibition. Beware!
The Meta Quest firmware doesn't allow using a wired ethernet connection to mitigate the issue to provide a quick fix to this problem in urgent situations. Again, a clean execution of this project is prevented by inflexibility of Meta's software. Unfortunately there are no alternative wireless HMDs on the market that allow the modifications we need for an optimum implementation.
In an ideal setup, with several HMDs being used, each HMD uses a separate WIFI 6e channel, with the selected channels as far apart as possible.
Optical tracking parsing latency
Due to limited WIFI bandwidth and computing power limitations on the HMD pose information streamed from the the optical tracking system is parsed on the HMD with a significant delay. So far we have not found a way to reduce this delay. As described in the sensor fusion section, we added input from native inside-out tracking to the fusion in order to compensate for this latency.
HMD built-in CPU performance limitation
The Meta Quest series headsets are stand-alone HMDs that integrate all computing hardware in one device. Due to size and heat emission restrictions, the amount of computing power they provide is limited. For the operation of LPVR, significant compute is required to decode the image stream coming from the rendering PC to the HMD. For high resolution images with a high framerate, performance bottlenecks can happen, resulting in reduced image quality or tracking artefacts. we discovered during experiments that turning the internal inside-out tracking of the HMD off, can improve the performance of the rest of the system. Note that in this case, inside-out tracking can’t be used to supplement pose calculation by FusionHub.
Inside-out / outside-in tracking misalignment
When fusing inside-out and outside-in tracking it is important that the coordinate systems of both pose calculation methods are congruent. If the pivot point of both coordinate systems is not identical, the resulting 3D image might show objects to be shifting with head motion. We provide a calibration tool for ART and Optitrack tracking to calibrate the outside-in pivot point to the inside-out pivot point.
Inside-out fusion (for advanced users)
In order to further improve the smoothness of the head pose tracking we extended our standard sensor fusion (outside-in with IMU fusion) by incorporating inside-out tracking data into the position calculation.
Our purpose is to combine the advantages of the outside-in tracking of providing a global reference, with the high responsiveness of the inside-out tracking.
This is done by high-pass filtering the inside-out position information and adding it to low-pass filtered outside-in tracking data. As a result, fast motions of the headset are tracked using the inside-out tracking, while the output is "pulled" towards the slower globally referenced position provided by the outside-in tracking.
Glossary:
Inside-out tracking: the built-in tracking of Meta Quest. Advantage of this tracking method: smooth and responsive. Disadvantage: Doesn't work in some environments, no global reference shared with other HMDs.
Outside-in tracking: External camera-based tracking such as ART or Optitrack. Advantages: Very reliable, provides global reference to all HMDs. Disadvantage: Might introduce latency when sending data from tracking server to HMDs.
IMU: Inertial measurement unit containing a gyroscope to measure angular velocity and an accelerometer to measure acceleration of the HMD. IMU data is very responsive and is therefore fused with optical tracking data to improve latency and allow for pose prediction.
To activate inside-out fusion add the following configuration block:
Make sure to set the data endpoint of the main fusion block to inproc://fused_pose_source
.
The FusionHub configuration contains several parameters that allow the tuning of the inside-out fusion:
Parameter | Purpose |
---|---|
| Weight of low-pass filter applied to inside-out tracking after high-pass filter (0 = unmodified inside-out signal, 1 = remove inside-out signal) |
| Weight of high-pass filter applied to inside-out tracking (0 = remove inside-out signal, 1 = full inside-out signal applied to result) |
| Weight of low-pass filter applied to outside-in tracking (0 = unmodified outside-in signal, 1 = outside-in signal is removed) |
| Set this to true to take height (y-axis) value directly from inside-out tracking |
| Set this to true to include inside-out tracking in pose calculation. If true, filter parameters described above will be used. |
Example configurations
Example configurations for the outside-in / inside-out filter:
Characteristic | Parameter setting |
---|---|
Mainly inside-out, outside-in for global referencing: Tracking is based mainly on the built-in inside-out tracking. Outside-in tracking is used for global referencing. Note: Turn headset tracking to ON in Quest settings. Room needs to be suitable for inside-out tracking. | |
Outside-in only + height from inside-out: Horizontal tracking is based on outside-in tracking. Only height (global y-axis) values are taken directly from inside-out tracking. Note: Turn headset tracking to ON in Quest settings. Room needs to be suitable for inside-out tracking. | |
Outside-in tracking only: Tracking is based on outside-in tracking only. Note: To maximize performance, make sure to turn headset tracking to OFF in Quest settings. |
Advanced intercalibration (only on Meta Quest)
Example videos
Motive
Intercalibratiotool: https://www.dropbox.com/scl/fi/1nkt3acbq69omehzz0j28/Intercalibration-tool-cryptolens.mp4?rlkey=qtg49h7kzbhcs4cz32yrnx7vn&dl=0
DTrack
PC screen recording: https://www.dropbox.com/scl/fi/6ouhtkac6409iekn51hbc/setupLPVRAIR-PC.mp4?rlkey=7ib34as0bfz4veqw69u0s6tga&st=yzvh9g4g&dl=0
Quest screen recording: https://www.dropbox.com/scl/fi/8fpsoybeb67sfn6x35ijt/setupLPVRAIR-Quest2.mp4?rlkey=xy2s5irxaguy5mw7ci8ep8tks&st=118r4rw8&dl=0
Software Required
PC
Optical system software: Motive / DTrack software
SteamVR (download Steam Client from official site, and then follow instructions here)
lh_opt_calib.exe
Dongled ← requires dongle hardware
Cryptolens ← requires Internet connection on the 1st activation call
Procedure Overview
There are 5 sections in total for the setup:
Create a new marker body file for the headset
Connect PC to the headset through SteamVR
Generate a calibrated body with
lh_opt_calib
Load the calibrated body file to the optical tracking software
Start the FusionHub
I - Create a new marker body file for the headset
Motive
In the perspective view, select the markers that you wish to group into the new rigid body.
Go to View → Builder Pane, give the new rigid body a new name.
Go to View → Assets Pane, select the newly created rigid body, right click on it and export it.
DTrack
Go to Tracking → Body Administration, increase the count by 1, and give your new tracking body a name. Then click “Calibrate”.
Click “calibrate” again and gently move and roll your headset in front of the tracking camera.
Save the body file by clicking “save file(s)”, select your headset body and save it to an empty folder.
II - Connect PC to the headset through SteamVR
On the headset, go to settings → system → Quest Link. Pairing with the PC and click the launch icon. This would start the Oculus software on the PC.
Then on the headset, start SteamVR. This will start the SteamVR on your PC, also in your headset you will enter into SteamVR’s default scene automatically.
III - Generate a calibrated body with lh_opt_calib
Motive
In the command line please go to the directory of
lh_opt_calib
and execute the following:bodyId to your headset’s tracking ID,
localAddress and remoteAddress is the setting for NatNet client
bodyFile to the file path (from section 1 step 3)
DTrack
Start the ART tracking by clicking “start” in DTrack software.
In the command line please go to the directory of
lh_opt_calib
and execute the following:set port to DTrack’s port (can be found in DTrack’s Tracking → Output)
bodyId to your headset’s tracking ID in DTrack
bodyFile to the file path (from section 1 step 3)Rotate the headset in front of the tracking camera to collect 10 poses. If in the end all 5 outputs looks close, this section is considered finished. Otherwise please retry step 4 and 5.
Exit the SteamVR on PC. Exit the Oculus software on your headset by clicking “Quit” in the menu for Quest AirLink.
IV - Load the calibrated body file to the optical tracking software
Motive
Remove the rigid body you created just now. Otherwise Motive does NOT overwrite it with the calibrated one.
Drag and drop the newly generated body file
xxx_HORAUD.motive
into the Assets Pane in the Motive software.
DTrack
In the DTrack software on PC, click “stop”.
Go to Tracking → Body Calibration, select your headset body in the dropdown menu, and click “Load file(s)” to load the calibrated body file
xxx_HORAUD.txt
under the same directory as your exported body file in section 1.
How to inter-calibration without precalibrated targets
We highly recommend purchasing precalibrated targets. If you don’t have precalibrated targets, please follow the instructions in the videos below.
Dtrack calibration : https://www.dropbox.com/scl/fi/cjmaykw5rt5263euqo0z1/Dtrack-Calibration-LPVR-AIR.MP4?rlkey=5y829p6wsi5z6dfpz8vtcpbze&st=3kkd916n&dl=0
Procedure Overview
Open the ART DTrack3 software and connect to the controller
Open "Body Administration" under Tracking > Body Administration
Select the body you wish to calibrate (or create a new one by increasing the "number of bodies" counter)
Give it a name if desired
Once it is selected, click on "Custom calibration" at the bottom of the window
In the "Custom Body Calibration" window that appears, make sure that the `standard` body type is selected
Click on Calibrate.
After a short countdown, a new window will appear, with a progress bar and a counter. Simply move the headset with the markers attached as long as the progress bar is not complete
Make sure the counter displays the number of markers attached to the headset
Close the calibration windows and open the "Body Adjustment" window from the main window under Tracking > Body adjustment
Select the newly calibrated body from the drop-down list at the top of that window
Place the headset in view of the ART cameras, aligned the same way that the room was calibrated.
Toggle on the "use tracked orientation" box at the bottom of the window.
10. Click on "Set origin to COG" and "Adjust axes to room" from the panel on the right side. You can now uncheck "use tracked orientation"
11. If needed, adjust manually the center of the headset relative to the marker's positions using the "Body position" and "Body orientation" sections of the right panel.
12. Click on "Apply", then "OK". All done!
FusionHub intercalibration : https://www.dropbox.com/scl/fi/di0dzqcdvhmm4m67wrtsf/FusionHub-Intercalibration-LPVR-AIR.mp4?rlkey=yrxlgfok7ghazf6kgemetd00k&st=uorf365v&dl=0
Procedure Overview
Pre-requisites
Make sure FusionHub is installed and configured to receive IMU and optical data.
You will also need a computer connected to the same network as the headset.
For this demo, we will use LP-ALVR to provide IMU data to FusionHub.
Intercalibration
Launch FusionHub on the headset
Connect FusionHub to the GUI application using the internal IP address of the headset
To do so, input the headset IP address into the "Server URL" field, at the top of the GUI. If the GUI window is not big enough, you might have to click the three-dot icon at the top right
There should be a default port pre-specified, change it if needed. If you do not know FusionHub's port, leave it to the default value of `19358`
You will know when you are connected when the status color next to the "Server URL" field turns green
Move to the "Base Configurations" panel, either from the appropriate icon on the Dashboard or through the side menu (You might have to click the hamburger icon on the top left if your window is not big enough)
Turn on optical tracking and make sure that the `nOptical` counter (displayed under the "Auto Calibration" section of the "Base Configurations" panel) is increasing
Turn on IMU data and make sure that the `nIMU` counter is increasing
For this demo, we use LP-ALVR. the first step is to launch the alvr client on the headset
Once the client is awaiting a connection, we launch the streamer from the PC we wish to stream. This will prompt us to launch SteamVR if it is not already launched.
After SteamVR is operational (this might involve an automatic restart), the streamer should detect the headset and ask if we want to trust it. Clicking on "trust" should start the connection and FusionHub should start receiving IMU data.
At the bottom of the "Intercalibration" section, check the "Recalibrate on next restart" checkbox, and click on "Restart" right underneath
When FusionHub boots again, rotate the headset around not too quickly, but with big movements, marking with regular pauses to let the tracking catch up. If you see the `nUsedPoses` counter under the "Intercalibration" section go up, that means you are probably doing it correctly
When the `nUsedPoses` counter reaches 50 (it might take a few more poses), the computed calibration value should appear under the `quaternion` field
If the `nUsedPoses` counter goes back to 0, it means that the computation did not converge to an appropriate calibration. Usually, this means you rotated the headset too quickly
From there click on "Apply Intercalibration Results". Then, on the top of the "Configuration section", click on "Set", then "Save". This will cause FusionHub to restart. Hopefully, everything works now!
References
ALVR open source project: https://github.com/alvr-org/ALVR
FusionHub documentation: FusionHub Manual