Calibrate XR alignment - Digital Twin XR

When doing XR, it is essential to perfectly align the physical LED wall with its virtual representation (XR walls). This is the baseline for projecting graphics onto them and achieving a believable parallax and set extension illusion.

Historically this has been done manually measuring the size and location of the LED wall. Having a LIDAR scan of the geometry helped. This process is obviously quite time consuming and hard, but can still be seen as a fallback solution.

Learn more about how to Create a Manual Twin

Automatic alignment process

Together with Pixotope Tracking we provide a more automated way of alignment, where image processing is used to scan the actual position and orientation of each individual LED panel.

This is a 4-step process

1. Pixotope Tracking generates a marker pattern which is sent to Pixotope Graphics

2. Pixotope Graphics creates mappings and displays the maker pattern on the LED panels

3. Pixotope Tracking generates Digital Twin data based on the position and orientation of each LED panel and sends it to Pixotope Graphics

4. Pixotope Graphics updates the Digital Twin XR Actor based on the digital twin data

1. Generate marker pattern using Pixotope Tracking

Create a marker pattern that covers all LED bodies of the XR stage. You can create multiple patterns by dividing parts of the XR stage into bodies. Each wall, floor or ceiling segment can be a new body. An LED body is one continuous LED surface. A curved LED wall with an extra LED floor would be 2 separate bodies.

Prerequisite: The Pixotope Vision sensor camera needs to be calibrated. Meaning that a lens file has to be made.

1.1 Marker Layout

1. Open the Marker Pattern Generator:

2. Click + Add Body on the upper right side to add as many bodies as you need. Name them to be able to identify them

3. For a digital twin the Shape should always be Arbitrary

4. Enter the specifications of the LED panels which you are using. That is: Number of Panels, the Panel Size in meters and the Number of Pixels that one panel consists of. This information refers to that specific body and can differ between multiple bodies.

5. With Panels per Marker you can extend one marker over multiple panels. If multiple panels display a single marker, these panels must be completely planar

6. Marker Spacing determines the space between the markers. As a rule of thumb we recommend the empty space between the markers to be half the size of a marker. If the space between markers is too small there can be interference in the individual marker detection

7. Rotate Markers rotates the markers by 90 degrees. Since the markers are wider than high, this enables you to expand one marker over two panels vertically. A curved LED wall is a use case for this, because panels are planar for each of its column. The markers can be larger that way

8. Checkered Pattern removes every other marker. This enables you to increase the size of the markers without any interference between them. Larger markers are easier to recognize from greater distance

9. Start / End Code refers to the code of the first and the last marker of this body. Every Marker as a unique code which is determined by its missing dots in an ascending order:

10. You can enable Border Settings to remove a line or column of markers at the edge of the body

11. You can enable Color Settings to change the colors used for background, markers and corners

12. A GhosTrack pattern can be created when enabling the GhosTrack Settings

13. The Export Image button exports the pattern as a .png file

1.2 Coordinate System

The preferred setup for a wall is to have the lower left corner as the origin and have the X-axis going through the lower right corner of the LED wall. The Z axis then goes upwards along the left edge of the LED wall

1. Choose Master, Axis and Plane marker by right-clicking onto the respective markers. These three markers determine origin, axes and scale of the coordinate system:

   1. The Master Marker determines the position of the origin. It is by default in the lower left corner of its panel

   2. The Axis Marker determines the secondary axis (Z for LED wall setup and Y for LED floor setup)

   3. The Axis Marker also determines the scale by its distance to the Master Marker. A blue line drawn into the pattern preview indicates this

   4. The Plane Marker determines the primary axis (X for LED wall and floor setup)

2. Select Axis Setup

3. Choose the Master Marker and Master Position. Master Position offsets the position of the origin from the default position. The default position ( Master position = 0,0,0) is the lower left corner of the Master Marker panel

If an LED wall is above the ground it is necessary to offset the Master position down to the floor directly underneath it. For example, if the LED wall is 15cm above the floor and you have selected the lowest left marker as Master, put 0.15 into the z-field of the Master Position. This will offset the entire X-axis to the floor.

4. Choose the Axis Marker. The scale is determined automatically by the distance between the Axis Marker and the Master Marker. For this to be correct, all LED panels on the line between the Axis Marker and the Master Marker must be planar

Choose the marker that is the furthest away from the Master Marker and that is planar to it, because a larger distance allows more precise calculations.

5. Choose the Plane Marker that will determine the X-axis. If possible, choose the lowest marker on the right. The X-axis will then go through the lower right corner of the LED wall.

The coordinate settings can also be defined “manually” by clicking onto three points in the reconstruction, as described in the Pixotope Tracking manual. This will overwrite the coordinate setup defined in this chapter.

Please be aware that an LED wall might not be perfectly 90 degrees upright. So defining the coordinate system with three points on the wall or three points on the floor can result in different axis angles.

1.3 Sending Pattern to Pixotope Graphics

1. Save the configuration file locally (usually /home/tracking/pxVision/VioStudio.cfg) using the “File” menu in the top left corner

2. To save the marker pattern configuration as well as the marker pattern .png-file onto the Pixotope Graphics Engine, click Save to Pixotope Store using the “File” menu in the top left corner

Please be aware that

• the Pixotope Graphics engine you are sending to must be running as a Server

  • Change in START > Machine Settings > Role

• the encryption has to be disabled for the Pixotope network you are on

  • Change in START > Machine Settings > Encryption

7. Select the IP address of the Pixotope Engine and Connect. Then Click Save to Store

The same marker pattern can be used for calibrating the offset between the sensor camera and the film camera as well as for determining the optical parameters of your zoom lens. Also the AutoInit function can use the same markers. Please see the Pixotope Tracking documentation for more details.

2. Create mappings using Pixotope Graphics

2.1 Create the initial Digital Twin

1. Open your XR level in the Pixotope Editor

2. Add the Digital Twin XR actor to your scene

   • The actor gets automatically placed as a child underneath the Camera Root

3. Click the "Initialize" button in Digital Twin XR > 1. Initialize Twin > Digital twin

   • For every single LED tile of a body, an XR wall actor is being created

At this point these XR walls are displayed in a plane and are not yet positioned correctly.

2.2 Create mappings and select output machines

1. Click "Create Single Mapping"

2. Select the first body

   1. the bodies were already set up with the correct size in mind, so you can keep the settings as they are

3. Click "Create mapping"

4. Repeat this for all bodies

Learn more about how to Create mappings

2.3 Display the tracking pattern

1. Expand Display Mode

2. Choose "Pattern" for the XR Wall Texture

3. The blue squares in the corners of the pattern must match the corners of the LED pattern.

3. Generate Digital Twin data using Pixotope Tracking

3.1 Setup Data hub connection

1. Start "Pixotope Vision Marker Calibration" (or "GhostTrack Marker Calibration")

2. Open the Settings and activate “Show expert settings”

3. Go to the DataHub tab and enter the Pixotope Data Hub IP address which can be found in Director > Status panel (lower right corner in the Director)

4. Save and restart "Pixotope Vision Marker Calibration" (or "GhostTrack Marker Calibration") to have the program connect to the Pixotope Data hub

3.2 Generate the digital twin data and send it to Pixotope Graphics

1. Start "Pixotope Vision Marker Calibration" (or "GhostTrack Marker Calibration")

2. Configure to use the sensor camera and the correct lens file and set the video settings for good visibility

3. Set the distance factor to a relatively low value, for example 0.10

4. Perform a reconstruction of all LED bodies including each marker from different angles and positions of the sensor camera, including being close to the panels

5. Open the Marker Calibration menu

6. Make sure that the program sees and identifies as many markers as possible (indicated by the yellow marker detection). Find the ideal Threshold value in the Image Segmenter on the right side. Master, Axis and Plane Marker must be included in at least two keyframes

7. Optional: determine the origin, orientation and scale manually:

Manual specification

1. Refine the coordinates by clicking in the Keyframes, as described in the Pixotope Tracking documentation. this will overwrite the coordinate system setup done in the Marker Pattern Generator

2. It makes sense to put the zero plane on the floor. For being able to click on points on the floor, place marks like tape underneath the corners of the LED wall.

3. If you also want to set the scale manually, enable Manually specify distance and type in the measured distance between the red and the blue point

4. Remember to set the axes correctly

8. Click "Process Markers" and check if theoretical position of markers (in red) matches the actual markers and if the coordinate system is correct a grid will indicate the origin and the three axes as well as the zero plane. The Digital Twin data is being sent to the Pixotope Store in the moment when the processing of the markers is finished.

9. Save the AutoInit file (“AutoInit.cfg”) and the Marker file (“VioMarker.cfg”) in the folder /home/tracking/pxVision.

The Digital Twin data can later be used to fine tune the precision of the tracking. Please see the Pixotope Tracking documentation for more details.

4. Update the Digital Twin

1. Click "Update Digital Twin" in the Digital Twin XR actor in the graphics engine. This will apply the digital twin data to this actor and thus reshape and relocate the XR wall(s).

If the pixel data of the LED wall has changed, existing mappings have to be recreated.

Enable Auto Update Digital Twin to continuously read the Digital Twin Data sent from Pixotope Tracking. This is mainly used for testing.

To completely start over:

1. Click "Delete Digital Twin”

This will reset the Digital Twin XR actor and delete all related XR walls.

The digital twin data which was sent to Pixotope Graphics is NOT deleted.

Check your alignment using an AR machine

Learn more about how to Check your alignment

Create XR holdouts

1. Click "Create Holdouts" in 3. Create Holdouts

Learn more about how to Create holdouts

Set up for 2D Media playback

Learn more about how to Playback 2D media on the LED wall