Tracking Maintenance Procedure
The tracking engines are designed to run the Pixotope tracking system unattended for long periods of time without having to actively maintain either the operating system or the tracking software. The following practices are meant to keep the systems in best working order.
Operation
Daily
Perform an operations check before going on air (see below)
The tracking software should be terminated when the studio is out of use. This serves both to reduce processor load and extend the lifespan of the infrared light, which will turn off when the software is not running.
Depending on the tracking engine hardware configuration, the tracking computer should be powered down when the studio is out of use. Small form factor machines, such as barebone PCs, laptops or similar, are not designed to run powered on for multiple days. To reduce crew workload when bringing a studio on air, the tracking software can be set to autostart when the engine powers on.
System temperature check: Use the Psensor program to check that essential hardware components like CPU, GPU and Video I/O card have an acceptable temperature.
CPU: < 90°C
GPU: < 80°C
Video I/O: < 80°C
Operations Check
Simple tests can offer a quick analysis of the tracking before going on air:
Ensure graphics are in their usual position when launching the scene
Ensure graphics move in sync with the video when the camera is panned
Determine if the virtual graphics are zoomed in at the same ratio as the real video and zooms in synchronously with the video
Ensure there is no visible jitter in the graphics when the camera is still or when the camera is moving
Do's and Don'ts
Once your system has been commissioned, follow these practices to ensure proper maintenance and care, to keep the same quality as achieved during installation:
Camera Tracking Do’s and Don’ts
The sensor unit have been mounted in place during installation. This unit must not change its relative position nor orientation to the tracked camera. Any changes to this positioning, even by millimeters or tenths of a degree, can potentially cause visible offsets. If the mounting has shifted or changed, a new offset calibration must be performed, which can include fine tuning adjustments.
The system will also have been commissioned to a particular lens. The mounting of the lens on the camera body must not change. If the lens has to be removed and remounted, the center shift must be revised.
All studio staff should be aware of the two previous points. Whenever lens or sensor unit mounting is changed or reconfigured, those in charge of tracking must be informed.
Whenever changes are made to the environment where an optical camera tracking sensor looks into, the current reconstruction should be extended by some keyframes taken of that area.
TalenTrack Do’s and Don’ts
The TalenTrack sensor cameras have been mounted in place during installation. These units must not change their position, orientation or lens settings. Any changes of this kind, even very small ones, can potentially cause visible offsets. A new TalenTrack calibration must then be performed.
Since the detection of persons is solely video based, it must be ensured that the tracking area is bright enough during show light conditions. The brightness can be adjusted in the TalenTrack software or with the iris of the lens
Tracking Engines
Updates
Without official notice, there will be no need to update either the operating system or software. Updates may be needed if there is a required bugfix for the specific system or new features are desired.
Updates may go along with new dependencies such as driver- or kernel-updates. It is recommended to perform any updates only when sufficient time is available to react to further software modifications needed.
Anti-Virus
The use of Linux itself is a sufficient protection against viruses. The Linux operating system used for tracking has furthermore been streamlined to run only essential services needed for tracking and remote support. By reducing the software overhead, the need for anti-virus protection is also reduced.
In case there are requirements to run additional protection, in order to comply with regulations or laws, the following recommendations can be taken into consideration:
Keep the tracking engines air-gapped from the internet during use. Note, that update or remote support sessions may require internet connectivity
ClamAV anti-virus software was qualified in August 2023 to run on a tracking engine. At the time, ClamAV was found not to interfere with engine performance during on-air use. However, manual or scheduled scans should not be performed during on-air use
When using anti-virus software, schedule regular anti-virus scans or perform them manually during a time when the engine is not on air
When using anti-virus software, schedule regular anti-virus database updates or perform them manually during a time when the engine is not on air
Third-Party Software
Certified software comes pre-configured on the operating system. All third-party software has to be tested for interference in correlation with the on-air tracking software, before going on air, and can not be supported by Pixotope support or engineering.
We always recommend a system configuration remain as close to the installation state as possible. Third party software can interfere with system performance, this includes video conferencing and remote desktop software such as: Zoom, Google Hangouts, TeamViewer, Anydesk etc...
Similarly, running software such as Dropbox or Adobe Creative Cloud can interfere both with network and hard drive bandwidth.
Troubleshooting
Once a tracking system is commissioned, there is no reason tracking quality will degrade unless software or hardware changes have been made post-installation. Most malfunctions result from hardware connectivity or manipulation (see “Do's and Don'ts”)
When troubleshooting, first determine whether all signals are properly connected to the tracking system. If anything is out of the ordinary, check all physical connectors and cables: new disconnects or timing issues are not attributable to software unless the original configuration has been manually altered post-commissioning. Use the following list to start troubleshooting your system:
Camera Tracking Signals
In the Chief Video window, ensure a live video of the tracked camera is visible (does not apply to Pixotope Fly setups).
In the Chief, under the lens tab, check the “Dump Parameters” box and ensure the system log prints live lens data values (does not apply to fixed lens setups).
In the Chief, under the IMU tab, check the “dump IMU” box and ensure the system log prints live IMU data.
In the Chief, under the WorkerCam tab, select the “Dump” option “final” to ensure the system log prints live position and orientation data.
In the Worker, ensure the sensor unit video is visible.
TalenTrack Signals
In the TalenTrack Video window, ensure a live video feed in the production frame rate is visible.
In the TalenTrack Client window, ensure that all signals are constantly green.
In the TalenTrack Server window, ensure that all videos are live and the skeletons are detected with a rectangular bounding box around it in the selected, large video on the right side.
If TalenTrack is used together with a Pixotope camera tracking system: In the TalenTrack window in the Painter section, check Dump camera to ensure the system log prints the live camera tracking data it receives. If this is the case and the camera tracking is also working correctly, a skeleton should be painted on top of detected persons in the TalenTrack Video window.
Further checks are:
Camera Tracking Troubleshooting
In the Worker video, dark blue tiles indicate areas that contain tracked feature points
In the Worker video, light blue tiles indicate areas where feature points are expected but currently not visible. If these tiles occupy a large portion of the video, even when nothing is obstructing that part of the view, try reloading the reconstruction.
In the Chief window, in the WorkerCam tab, the final tracking data can be printed in the System Log window. The last column of this dump tells the timing of the tracking data packages. It should be constantly close to the duration of a frame or field.
When zooming in with no camera movement, ensure the graphics do not shift their position in the image center (test for center shift, an indicator of the lens having been removed).
TalenTrack Troubleshooting
If a camera doesn’t connect correctly, it’s view will remain empty in the TalenTrack Server at program start or will freeze when the Server is already running. Try stopping the program in the App Manager and after some seconds starting it again. If the camera remains disconnected, try rebooting the camera itself by disconnecting the Power over Ethernet connection.
The best way to check the TalenTrack function is the skeletons painted on top of tracked persons in the TalenTrack Video window. If no Pixotope camera tracking system is used, the skeletons in the TalenTrack Server and the skeleton Viewer in the TalenTrack window can be checked
If persons in the tracking area do not appear as skeletons in the TalenTrack Front, or the recognition is unstable, check the various views in the TalenTrack Server. Check whether at least two cameras can recognize a decent number of joints of the same person or maybe perspective and person position block too many joints. If multiple cameras detect a skeleton but it does not receive the 3D recognition bounding box, the system may have trouble aligning the joints from multiple views. Check the calibration. Cameras may have been accidentally moved e.g. during works on the lighting, and need re-calibration.
If the joint tracking seems unstable, check the various views in the TalenTrack Server. Check whether the joints can be seen from multiple views in the respective position within the tracking area. Check whether the persons appearance has sufficient contrast towards the surroundings in the camera view. Increasing the brightness may help. If none of the above applies, but recognition seems unreasonably unstable, the camera calibration may be off. This is usually visible by a change in focus and may be caused by accidentally hitting the camera or extreme temperature changes. The cameras should be re-calibrated.
If the tracked joints do not appear in the correct position in the virtual space, make sure the coordinate systems of TalenTrack and the camera tracking systems align correctly. Check the grid in the TalenTrack Server. The coordinate system may have been put in the wrong place in the Coordinate system step or even the camera tracking may be incorrect.