Inside-Out tracking with LPSLAM and LPVR-DUO

Owned by Klaus Petersen
Last updated: Dec 03, 2024 by Luis Peralta
8 min read

LPSLAM can function as the optical tracking system for LPVR-DUO. To fully integrate it, the communication between the two systems must be configured, and a fiducial marker in the platform’s IMU must be set up to perform the IMU inter-calibration.

1) Varjo XR-3 camera adjustment

For better tracking performance with LPSLAM, it is recommended that the camera’s autoexposure be deactivated, and the Exposure and ISO values be set manually via Varjo Base. To allow the cameras to see inside the vehicle clearly:

  1. Reduce the impact of brightness coming from outside the windows and prioritize indoor lighting by underexposing the image slightly.

  2. Set a low ISO setting to reduce the sensor’s sensitivity to light and prevent bright areas, such as windows, from overexposing the image. If the lighting is very dim, you can increase the ISO. However, keep in mind that higher ISO settings may introduce noise into the image.

Screenshot 2024-05-17 155127.png
Manual setting of exposure and ISO in Varjo Base

 

2) Setting up a fiducial marker

First, choose a fiducial marker from the Chilitags library of pre-made markers (available here) and print it out. Attach the marker to the reference/platform IMU and ensure it’s placed in a location visible to the HMD cameras.

The choice of Chilitags marker does not matter, as long as it is the correct physical size for the configuration. However, keep in mind that the markers in the downloaded folder are already categorized as mini or big, and with or without borders. If you choose a marker with a border, ensure that it is printed with the border.

IMG_20241127_140909.jpg
A fiducial marker of 4.5cm attached to the IMU

In your LPSLAM configuration file, set a tracker of type Marker and configure the corresponding parameters. Adjust these parameters as needed to ensure quick detection of the fiducial marker while maintaining a stable tracked pose. For more information on the Marker tracker parameters, refer to the configuration file documentation.

â‹® "trackers": [ â‹® "type": "Marker", "configuration": { "marker_size": 0.045, "process_noise": 0.8, "observation_noise": 0.2, "show_tags": true, "detect_only": 90 } ] â‹®

The detect_only parameter is optional, but it is recommended to set it to the ID of the chosen fiducial marker. This helps prevent the tracker from mistakenly identifying image features as markers that are not actually present in the scene or when using other fiducial markers to enhance feature-poor surfaces.

 

3) Setting up communication in LPSLAM

For a faster and easier setup, the configs folder has the pre-configured file lpvr-duo.jsonto copy-paste the contents or upload it to the driver.

Proxy ports

LPSLAM generates two types of absolute 6DoF poses within the Visual-SLAM global reference frame:

  • Visual-SLAM poses: The estimated poses of the headset.

  • Marker poses: The estimated poses of the fiducial markers visible in the scene.

Each pose type is sent to a different proxy port, configured separately for each tracker using the proxy_port parameter, as shown below:

â‹® "trackers": [ { "type": "VSLAMStereo", "configuration": { â‹® "proxy_port": 5002 } }, { "type": "Marker", "configuration" : { â‹® "proxy_port": 5003 } } â‹®

Tracking IDs

Each tracked object has to have a tracking ID. For a tracker of type VSLAMStereo or VSLAMMono, the tracked object is the system to which the camera is attached, in this case, the HMD. For the Marker tracker, each fiducial marker in the scene represents a tracked object.

  • The tracking ID for VSLAMStereo or VSLAMMono trackers is configured using the tracking_id parameter.

â‹® "trackers": [ { "type": "VSLAMStereo", "configuration": { â‹® "tracking_id": 1 } }, â‹®

  • The tracking IDs for fiducial markers detected by the Marker tracker are directly defined by the markers chosen from Chilitags' library of pre-made tags, as each marker comes with a pre-defined ID. Therefore, if marker 90 is chosen, the tracking ID will also be 90.

 

4) Running LPSLAM + LPVR-DUO

After setting the fiducial marker and configuring the proxy ports and tracking ID, launch Varjo Base and let it run in the background. Then, start LPSLAM tracking using GUI controller.

Next, in the System tab of Varjo Base, select LPVR-DUO-Varjo as the tracking method from the drop-down menu. Then, open the web interface (http://localhost:7119/) to configure LPVR-DUO.

 

5) Setting up communication in LPVR

Navigate to the Expert Mode page in LPVR-DUO’s web interface.

Proxy ports

In the JSON editor, set an absolute pose source for each type of pose coming from LPSLAM. Assign a freely chosen identifying name and a port under their settings with the type DTrack, as shown below.

Tracking IDs

To receive the HMD poses from LPSLAM, got to the trackedObjects section add an entry with the corresponding absoluteSource. Then, set the tracking ID defined earlier for the VSLAM tracker and apply the configuration.

 

6) Calibration of Platform IMU

To perform the IMU calibration using LPSLAM as the optical tracking system, you first need to add the reference/platform IMU under the imuSources section in LPVR’s configuration.

In the LPVR-DUO’s web interface, navigate to the Calibration page from the bottom navigation bar. Click on the second setup option (Is your IMU tracked by the optical tracking system?) to display the corresponding calibration process.

1) LPMS-IG1 tracking setup

Select the absolute source corresponding to the Marker tracker, and set the Tracking ID to the ID of the chosen fiducial marker.

2) Calibrate optical tracking of LPMS-IG1

Use the HMD cameras too view the fiducial marker, and the Start Calibration button will be enabled. Then, press the Start Calibration button and move the IMU with the marker in front of the cameras. The interface will display a progress bar that fills up as you change the IMU’s orientation.

Once all the required poses are obtained, the Apply Result button will appear, allowing you to apply the estimated calibration quaternion to the driver.

3) Extract IMU orientation w.r.t. space

Place the IMU back in its fixed position and ensure the marker is visible to the HMD cameras. While the marker is in view and being tracked, press the Use Current Orientation button.

 

7) Full LPVR-DUO integration

Once you have successfully performed the IMU calibration, you will have a configuration similar to the example shown below.

 

You can first map your environment and then disable the mapping function in LPSLAM’s Map Viewer. This is particularly useful in in-car scenarios where you only want the HMD to track its pose using only in-car visual features. However, it is not fully necessary if you are using the processor type ThresholdImage.

Adjust Coordinates

Due to the Visual-SLAM nature of LPSLAM, the origin of its coordinate system is defined by the initial tracked pose, as shown in the image below.

As a result, if the HMD was misaligned with the world frame at startup, the XR content will also be misaligned. To correct this, navigate to the Expert Mode page of the LPVR-DUO web interface and click the Show additional functions button below the JSON editor.

Within the additional functions, navigate to Adjust Coordinates (Development Only). While the HMD positioned in the desired alignment, press the Set Current Pose as Origin button. The scene or XR content will align automatically.

Â