Set up Body Pose Estimation (BPE)
Body Pose Estimation allows an accurate live estimation of the pose and position of a human body directly from a video image, using an AI powered framework from NVIDIA. It provides a real-time set of 3D data points corresponding to a skeleton with bones and body parts.
With it you can:
position the VS compositing plane, allowing the talent on the green screen set to freely move toward and away from the camera
render shadows and reflections more accurately and even render footprints
allow the talent to interact with their surrounding using
physics
trigger boxes
hand action
or clicker control for pick up / place logic
Limitations for the current NVIDIA AR SDK
Only a single person is allowed within the camera field of view (including humanoids like mannequins)
if a full person cannot be seen, it will try to infer and extrapolate bone positions
The resolution of the fingers is poor or erratic, so hand poses cannot be determined
The GPU processing overhead can be high and could affect rendering performance on complex scenes
See Share Body Pose Estimation with other engines below to overcome this
It is likely that the skeleton position is always one frame behind the actual position
Only NVIDIA GeForce RTX 20 and 30 Series or NVIDIA RTX professional GPUs are supported
Use NVIDIA GPUs with Tensor Cores, ideally Tensor 2 Cores
Install NVIDIA AR SDK
Install the AR SDK from
https://www.nvidia.com/en-us/geforce/broadcasting/broadcast-sdk/resources/Restart Pixotope
Only NVIDIA GeForce RTX 20 and 30 Series or NVIDIA RTX professional GPUs are supported
Enable Body Pose Estimation
In Director
Go to SETUP > Configure > Routing
Enable Body Pose Estimation for the video input which should be analyzed

Use Body Pose Estimation in level
In Editor
Add the "BPE & Plane" actor to the level which can be found in Place Actors > Pixotope
Dropping it into the scene creates
a "BPE Compositing Mesh"
and a "VS Internal Compositing Plane" which follows the "BPE Compositing Mesh" in position and scale
Learn more about the BPE Compositing Mesh
The same setup as the BPE & Plane actor can be achieved by adding
a "BPE Compositing Mesh"
a "VS Internal Compositing Plane"
with "Use body pose estimation for position" enabled
and linking "Position Scale" of both actors
Share Body Pose Estimation with other engines
It can be useful to share the Body Pose Estimation data with other engines if:
the rendering overhead is too high to generate Body Pose Estimation on the same machine as the scene is getting rendered
the main camera is moving in too tight on the presenter, and the Body Pose Estimation cannot resolve the skeleton
In a multi-cam environment where there are multiple cameras which need to use the Body Pose Estimation data
Scenario 1 - Single Camera, Second Engine
Use a separate Pixotope engine with a camera input from the main camera to generate the Body Pose Estimation data.
On the source engine
The source engine is used to generate the Body Pose Estimation
Open the "Utilities" panel and go to BPE > Export
Click "Start Export"
this stores the Body Pose Estimation data in the Store and makes it accessible for all connected engines

On the target engines
The target engines use the generated Body Pose Estimation data
Open the "Utilities" panel and go to BPE > Import
Enter the name of the source engine/machine which generates the Body Pose Estimation data
[Source machine name]-Engine
for exampleRenderMachine1-Engine
Click "Start Import"
The Body Pose Estimation data will be 1 frame delayed from real-time.
The target engines do not need to have the same high powered GPU to use the Body Pose Estimation data.
Scenario 2 - Witness Camera, Second Engine
Use a separate camera to provide the images for the Body Pose Estimation source engine.
Set up witness camera and engine
The “witness” camera should
be setup to view the talent on screen wherever they are on the green screen set
be free from occlusions from other camera’s motion or set elements
not capture any people to the edge or off-set
the engine creating the Body Pose Estimation requires a high performing GPU - the target systems do not
Set up Body Pose Estimation sharing as shown in Scenario 1
Advanced
Use Body Pose Estimation in blueprints
All Body Pose Estimation points can be directly accessed in blueprints via the BPE Manager.

In addition, the Procedural Composite Mesh
has multiple “attachment” points that can be used.

