Create a Tracking Twin - Digital Twin XR

When doing XR, it is essential to perfectly align the physical LED wall with its virtual representation (XR walls). This is the baseline for projecting graphics onto them and achieving a believable parallax and set extension illusion.

Historically this has been done manually measuring the size and location of the LED wall. Having a LIDAR scan of the geometry helped. This process is obviously quite time consuming and hard, but can still be seen as a fallback solution.

Manage Digital Twins

To create more than one Digital Twin version which you can switch between and manage, go to Director > Configure > LED mapping.

Automatic alignment process

Together with Pixotope Tracking we provide a more automated way of alignment, where image processing is used to scan the actual position and orientation of each individual LED panel.

This is a 4-step process

1. Pixotope Tracking generates a marker pattern which is sent to Pixotope Graphics

2. Pixotope Graphics creates mappings and displays the maker pattern on the LED panels

3. Pixotope Tracking generates Pixotope Tracking Twin data based on the position and orientation of each LED panel and sends it to Pixotope Graphics

4. Pixotope Graphics updates the Digital Twin XR Actor based on the Pixotope Tracking Twin data

1. Generate marker pattern using Pixotope Tracking

Create a marker pattern that covers all LED bodies of the XR stage. You can create multiple patterns by dividing parts of the XR stage into bodies. Each wall, floor or ceiling segment can be a new body. An LED body is one continuous LED surface. A curved LED wall with an extra LED floor would be 2 separate bodies.

1.1 Marker Layout

1. Open the Marker Pattern Generator:

2. Click + Add Body on the upper right side to add as many bodies as you need.

   1. The bodies do not have to correspond to the video output routing. In general, all LED panels that form a single shape, should be a single body.

   2. Each LED part disconnected from it, such as ceilings, floors or totems, should become individual bodies.

   3. Each LED portion that uses different panel parameters, like Panel Pixels or Size, than the others will also require their own body setup. Name them to be able to identify them.

3. For a Digital Twin the Shape should always be Arbitrary.

4. Enter the specifications of the LED panels which you are using. That is: Number of Panels, the Panel Size in meters and the Number of Pixels that one panel consists of. This information refers to that specific body and can differ between multiple bodies.

5. With Panels per Marker you can extend one marker over multiple panels. This may be applied when markers on a single panel are very small in relation to the LED wall size, and they might become hard to recognize from distances in which the whole wall can be seen.
   If multiple panels display a single marker, these panels must be completely planar and can not have an angle between them! If the wall is curved, most of the time the panels in one column, above each other, share the same orientation and are available for 2 panels per marker.

6. Marker Spacing determines the space between the markers. As a rule of thumb we recommend the empty space between the markers to be half the size of a marker. If the space between markers is too small there can be interference in the individual marker detection.

7. Rotate Markers rotates the markers by 90 degrees. Since the markers are wider than high, this enables you to expand one marker over two panels vertically. A curved LED wall is a use case for this, as explained in point 5. The markers can be larger that way, requiring less Marker Spacing.

8. Checkered Pattern removes every other marker. This enables you to increase the size of the markers without any interference between them. In general, this setting should be avoided because it relies on extrapolating existing data for missing markers.

9. Start / End Code refers to the code of the first and the last marker of this body. Every Marker as a unique code which is determined by its missing dots in an ascending order, as shown below. This is filled in automatically and will only need to be touched if multiple pattern configurations are being used in the same space, and they need to be able to be told apart.

10. Border Settings can be used to remove a line or column of markers at the edge of the body, in case they can not be visible during calibration. These panels will then be extrapolated from other panels data.

11. Color Settings can be used to change some colors for convenience.

12. A GhosTrack pattern can be created when enabling the GhosTrack Settings. This will create a second pattern version that replaces most of the markers with a custom image. The image should provide feature points to the tracking that are more distinct corners than the repetitive circles, but have markers remain in place to enable Auto-Init.

13. The Export Image button exports the pattern as a .png file.

1.2 Coordinate System

Set up the coordinate system of the pattern. Note that this is a setting that happens in a single body only - all other bodies use that body and it’s coordinate system for reference.

The coordinate system used for the XR scene will be defined in the Marker Calibration process. However, it can be pre-defined in the pattern by selecting markers. These will also be used for calculations, so the selection is mandatory.

The preferred setup for a wall is to have the lower left corner as the origin and have the X-axis going through the lower right corner of the LED wall. The Z axis then goes upwards along the edge of the LED wall.

1. Choose Master, Axis and Plane marker by right-clicking onto the respective markers. These three markers determine origin, axes and scale of the coordinate system:

   1. The Master Marker determines the position of the origin. It is by default in the lower left corner of its panel.

   2. The Axis Marker determines the first axis (Z for LED wall setup or Y for LED floor setup).
      The Axis Marker also determines the scale by its distance to the Master Marker. A blue line drawn into the pattern preview indicates this. Mind that this scale will not be measured but calculated by the number of panels! That means Master and Axis Markers have to be in one plane and not angled towards each other! (e.g. the panels in a straight column of a curved wall)

   3. The Plane Marker determines the second axis (X for LED wall and floor setup)

2. Select Axis Setup. This will determine whether the coordinate system will result in the body with the Origin Markers to be a floor, ceiling or wall.

3. Choose the Master Marker and Master Position. Master Position offsets the position of the origin from the default position. The default position ( Master position = 0,0,0) is the lower left corner of the Master Marker panel

   

1.3 Sending Pattern to Pixotope Graphics

1. Save the configuration file locally (usually /home/tracking/pxVision/VioStudio.cfg) using the “File” menu in the top left corner

2. To save the marker pattern configuration as well as the marker pattern .png-file onto the Pixotope Graphics Engine, click Save to Pixotope Store using the “File” menu in the top left corner

7. Select the IP address of the Pixotope Engine and Connect. Then Click Save to Store

2. Create mappings using Pixotope Graphics

2.1 Create the initial Pixotope Tracking Twin

1. Open your XR level in the Pixotope Editor

2. Add the Digital Twin XR actor to your scene

   • The actor gets automatically placed as a child underneath the Camera Root

3. Click the "Initialize" button in Digital Twin XR > 1. Initialize Twin > Pixotope Tracking Twin

   • For every single LED tile of a body, an XR wall actor is being created

2.2 Create mappings and select output machines

2.3 Display the tracking pattern

1. Expand Display Mode

2. Choose "Pattern" for the XR Wall Texture

3. The colored squares in the corners of the pattern must match the corners of the LED pattern.

3. Generate Pixotope Tracking Twin data using Pixotope Tracking

3.1 Setup Data hub connection

1. Start "Pixotope Vision Marker Calibration" (or "GhostTrack Marker Calibration")

2. Open the Settings and activate “Show expert settings”

3. Go to the DataHub tab and enter the Pixotope Data Hub IP address which can be found in Director > Status panel (lower right corner in the Director)

4. Save and restart "Pixotope Vision Marker Calibration" (or "GhostTrack Marker Calibration") to have the program connect to the Pixotope Data hub

3.2 Marker Calibration: Generate the digital twin data and send it to Pixotope Graphics

1. Start "Pixotope Vision Marker Calibration" (or "GhostTrack Marker Calibration")

2. Configure to use the sensor camera and the correct lens file and set the video settings for good visibility

3. Perform a reconstruction of all LED bodies including each marker from different angles and positions. Make sure to have multiple samples of each marker. Check the expand for more details

4. Open the Marker Calibration menu

   

5. Make sure that the program sees and identifies as many markers as possible (indicated by the yellow marker detection, the green numbers can be ignored).

   1. Find the ideal Threshold value in the Image Segmenter section on the right side. Selecting Binary mode in the dropdown menu and zooming into the image with the mouse wheel can help estimate the value.

      

      
      

   2. Click Apply to all to use the threshold in all keyframes

   3. Move on to scan through the keyframes, checking some of them whether the threshold is a good choice. If a different threshold gives better results for specific perspectives, adjust it and click Apply to all again to add additional data.

6. Optional: determine the origin, orientation and scale manually:

8. Click "Process Markers", wait for the process to finish and check if theoretical position of markers (in red) matches the actual markers and if the coordinate system is correct. a grid will indicate the origin and the three axes as well as the zero plane.

9. Scan through some keyframes and check the result. If content, click Send to Store to send the data to Director.

10. Save the AutoInit file (“AutoInit.cfg”) and the Marker file (“VioMarker.cfg”) in the folder /home/tracking/pxVision.

4. Update the Digital Twin

1. Click "Update Position Data" in the Digital Twin XR actor in the graphics engine. This will apply the Pixotope Tracking Twin data to this actor and thus reshape and relocate the XR wall(s).

To completely start over:

1. Click "Delete Pixotope Tracking Twin”

Check your alignment using an AR machine

Manage XR holdouts

Enable XR holdouts when using a virtual set extension.