LPSLAM can function as the optical tracking system for LPVR-DUO. To fully integrate it, the communication between the two systems must be configured, and a fiducial marker in the platform’s IMU must be set up to perform the IMU inter-calibration.

1) Varjo XR-3 camera adjustment

For better tracking performance with LPSLAM, it is recommended that the camera’s autoexposure be deactivated, and the Exposure and ISO values be set manually via Varjo Base. To allow the cameras to see inside the vehicle clearly:

1. Reduce the impact of brightness coming from outside the windows and prioritize indoor lighting by underexposing the image slightly.

2. Set a low ISO setting to reduce the sensor’s sensitivity to light and prevent bright areas, such as windows, from overexposing the image. If the lighting is very dim, you can increase the ISO. However, keep in mind that higher ISO settings may introduce noise into the image.

2) Setting up a fiducial marker

First, choose a fiducial marker from the Chilitags library of pre-made markers (available here) and print it out. Attach the marker to the reference/platform IMU and ensure it’s placed in a location visible to the HMD cameras.

In your LPSLAM configuration file, set a tracker of type Marker and configure the corresponding parameters. Adjust these parameters as needed to ensure quick detection of the fiducial marker while maintaining a stable tracked pose. For more information on the Marker tracker parameters, refer to the configuration file documentation.

  ⋮
  "trackers": [
    ⋮
    "type": "Marker",
      "configuration": {
        "marker_size": 0.045,
        "process_noise": 0.8,
        "observation_noise": 0.2,
        "show_tags": true,
        "detect_only": 90
      }
  ]
  ⋮

3) Setting up communication in LPSLAM

Proxy ports

LPSLAM generates two types of absolute 6DoF poses within the Visual-SLAM global reference frame:

• Visual-SLAM poses: The estimated poses of the headset.

• Marker poses: The estimated poses of the fiducial markers visible in the scene.

Each pose type is sent to a different proxy port, configured separately for each tracker using the proxy_port parameter, as shown below:

  ⋮
  "trackers": [
        {
            "type": "VSLAMStereo",
            "configuration": {
                ⋮
                "proxy_port": 5002
            }
        },
        {
            "type": "Marker",
            "configuration" : {
                ⋮
                "proxy_port": 5003
            }
        }
  ⋮

Tracking IDs

Each tracked object has to have a tracking ID. For a tracker of type VSLAMStereo or VSLAMMono, the tracked object is the system to which the camera is attached, in this case, the HMD. For the Marker tracker, each fiducial marker in the scene represents a tracked object.

• The tracking ID for VSLAMStereo or VSLAMMono trackers is configured using the tracking_id parameter.

  ⋮
  "trackers": [
        {
          "type": "VSLAMStereo",
          "configuration": {
              ⋮
              "tracking_id": 1
            }
  },
  ⋮

• The tracking IDs for fiducial markers detected by the Marker tracker are directly defined by the markers chosen from Chilitags' library of pre-made tags, as each marker comes with a pre-defined ID. Therefore, if marker 90 is chosen, the tracking ID will also be 90.

4) Running LPSLAM + LPVR-DUO

After setting the fiducial marker and configuring the proxy ports and tracking ID, launch Varjo Base and let it run in the background. Then, start LPSLAM tracking using either the standalone app or the GUI controller.

Next, in the System tab of Varjo Base, select LPVR-DUO-Varjo as the tracking method from the drop-down menu. Then, open the web interface (http://localhost:7119/) to configure LPVR-DUO.

5) Setting up communication in LPVR

Proxy ports

In the JSON editor of the Expert Mode tab, set an absolute pose source for each type of pose coming from LPSLAM. Assign a freely chosen identifying name and a port under their settings with the type DTrack, as shown below.

{
 "absoluteSources": {
    "vslam": {
      "settings": {
        "oneBaseCounting": false,
        "port": 5002
        },
      "type": "DTrack"
    },
    "marker": {
      "settings": {
        "oneBaseCounting": false,
        "port": 5003
        },
      "type": "DTrack"
    },
  },
  ⋮

Tracking IDs

To receive the HMD poses from LPSLAM, got to the trackedObjects section add an entry with the corresponding absoluteSource. Then, set the tracking ID defined earlier for the VSLAM tracker and apply the configuration.

  ⋮
  "trackedObjects": [
    {
      "absoluteSource": {
        "name": "vslam",
        "trackingId": 1
      },
      "combinerType": "DifferentalImu",
      "emmiterName": "HMD"
      "enabled": true,
      "imuSource": "HMD_imu",
      "settings": {
        "ignoreGravity": true,
        "opticalWeight": 0.005,
        "referenceImu": "Platform_IMU",
        "referenceOrientationQuat": {
          "w": 1,
          "x": -1,
          "y": 1,
          "z": 1
        },
        "referenceToOpticalQuat": {
          "w": 1,
          "x": 0,
          "y": 0,
          "z": 0
        }
      }
    }
  ]
  ⋮

6) Calibration of Platform IMU

To perform the IMU calibration using LPSLAM as the optical tracking system, you first need to add the reference/platform IMU under the imuSources section in LPVR’s configuration.

  ⋮
  "imuSources": {
    ⋮
    "Platform_IMU": {
      "settings": {
        "autodetectType": "ig1"
      },
      "type": "OpenZen"
    }
  },
  ⋮

In the LPVR-DUO’s web interface, navigate to the Calibration page from the bottom navigation bar. Click on the second setup option (Is your IMU tracked by the optical tracking system?) to display the corresponding calibration process.

1) LPMS-IG1 tracking setup

Select the absolute source corresponding to the Marker tracker, and set the Tracking ID to the ID of the chosen fiducial marker.

2) Calibrate optical tracking of LPMS-IG1

Use the HMD cameras too view the fiducial marker, and the Start Calibration button will be enabled. Then, press the Start Calibration button and move the IMU with the marker in front of the cameras. The interface will display a progress bar that fills up as you change the IMU’s orientation.

When moving the IMU in front of the camera, ensure the marker remains at reasonably visible angles, relatively close, and within the inner square of the Marker Viewer grid. To assist, the Marker Viewer colorizes the reference frame of well-detected poses. As the marker moves further from the camera’s center or plane, measurement uncertainty increases, making the poses less reliable. These poses will appear in black and white and will not be included in the calibration process.

Once all the required poses are obtained, the Apply Result button will appear, allowing you to apply the estimated calibration quaternion to the driver.

3) Extract IMU orientation w.r.t. space

Place the IMU back in its fixed position and ensure the marker is visible to the HMD cameras. While the marker is in view and being tracked, press the Use Current Orientation button.

7) Full LPVR-DUO integration

Once you have successfully performed the IMU calibration, you will have a configuration similar to the example shown below.

 {
 "absoluteSources": {
    "vslam": {
      "settings": {
        "oneBaseCounting": false,
        "port": 5002
        },
      "type": "DTrack"
    },
    "marker": {
      "settings": {
        "oneBaseCounting": false,
        "port": 5003
        },
      "type": "DTrack"
    },
  },
  "emitters": {
    "HMD": {
      "type": "Keeper"
    }
  },
  "imuSources": {
    "HMD_imu": {
      "type": "Varjo"
    },
    "Platform_IMU": {
      "settings": {
        "autodetectType": "ig1"
      },
      "type": "OpenZen"
    }
  },
  "mapper": {
    "dataProxy": {
      "port": 14355
    },
    "recordToFile": false
  },
  "trackedObjects": [
    {
      "absoluteSource": {
        "name": "vslam",
        "trackingId": 1
      },
      "combinerType": "DifferentialImu",
      "emitterName": "HMD",
      "enabled": true,
      "imuSource": "HMD_imu",
      "settings": {
        "ignoreGravity": true,
        "opticalWeight": 0.005,
        "referenceImu": "Platform_IMU",
        "referenceOrientationQuat": {
          "w": 0.9534095147079993,
          "x": -0.15571224039940204,
          "y": 0.2534998537800348,
          "z": -0.050018192565875905
        },
        "referenceToOpticalQuat": {
          "w": 0.9988665344367207,
          "x": -0.018843779478164475,
          "y": 0.04052453991632312,
          "z": -0.016380476852768867
        }
      }
    }
  ]
}

You can first map your environment and then disable the mapping function in LPSLAM’s Map Viewer. This is particularly useful in in-car scenarios where you only want the HMD to track its pose using only in-car visual features. However, it is not fully necessary if you are using the processor type ThresholdImage.

Adjust Coordinates

Due to the Visual-SLAM nature of LPSLAM, the origin of its coordinate system is defined by the initial tracked pose, as shown in the image below.

As a result, if the HMD was misaligned with the world frame at startup, the XR content will also be misaligned. To correct this, navigate to the Expert Mode page of the LPVR-DUO web interface and click the Show additional functions button below the JSON editor.

Within the additional functions, navigate to Adjust Coordinates (Development Only). While the HMD positioned in the desired alignment, press the Set Current Pose as Origin button. The scene or XR content will align automatically.