Make performance optimizations

By Optimization we mean the technical and creative steps we can take to adjust, reduce, and streamline our content to help it run faster and hit the desired frame rate target!

Every show and production where Pixotope is being used, needs to run at a given frame rate. Given that Pixotope is a real-time renderer, we need to render each final frame of graphics in the window of time between one frame and the next. In other words - the higher the frame rate of our show, the less time we have to render our graphics.

Render time = 1 / frame rate

The show frame rate allots our rendering budget and, as a consequence, the level of optimization needed!

Note that the engine's output resolution also heavily impacts performance — it is much more demanding to render UHD than HD in real time!

The time it takes to render a frame of graphics in Pixotope depends on multiple factors - primarily how complex your graphics are, your video I/O configuration, and whether your hardware is up to spec.

The moment we fail to maintain a stable (or even workable) frame rate, we experience performance issues. This is most commonly caused by a combination of several compounding issues rather than a singular bottleneck.

Prepare for performance testing

Our optimization workflow is simply to try and identify the current biggest resource drains, alleviating them as much as possible before repeating the process over and over again.

The goal is to arrive at being able to comfortably render the content at our target frame rate, including some extra leeway for the sake of show safety.

Test in the right environment

It is very much not ideal, sometimes bordering on meaningless, to profile performance issues on a workstation different from the engines used in the actual show production.

At the very least, you will want to have the same hardware configuration that the live production will utilize. Ideally, you will also have the same video and tracking inputs and outputs, to ensure you are accounting for the video I/O overhead when profiling your render times.

Verify that you are using the correct resolution (if your broadcast is in 1080 and you are rendering at 4K, it will not give you an accurate representation of the frame rate).

Test from the same camera angle, to make sure you get comparable results.

Prioritize

Keep in mind that everything comes at the cost of something else. When it comes to rendering, we have a FIXED BUDGET of milliseconds available to us. Whatever feature or asset you wish to add to your scene, will at some point come at the cost of something else. Identify which features give you the most bang for your buck, and prioritize. 

Faster is harder

A lot of aspects of the Pixotope rendering pipeline has a flat overhead cost (Postprocessing effects, video I/O transfer waiting, etc), taking a flat chunk out of our render time - meaning that higher frame rates require DRASTICALLY more optimized scenes than low frame rate shows!

Use Live Mode

The Pixotope Editor is fast, but there are still significant overhead costs to rendering the UI, as well as having all the working tools loaded. Even though it is fine to do rough estimates and profiling in Play in Editor mode (disable WYSIWYG), for accurate values, please debug when running in standalone Live mode.

Clean your desk

If your workstation is running a bunch of non-essential garbage applications and processed in the background, this can negatively affect your engine’s performance. Remove and close applications you are not using for your show.

Identify performance issues

The easiest way to identify performance issues, is to visually identify that the output video feed is stuttering (dropping frames), or even outright freezing.

Dropped frames can also be caused by bad genlock and/or bad tracking data. If the stutters are very intermittent, it can be an idea to decouple tracking and enable free run, to reduce possible culprits by process of elimination. Occam’s razor is our most trusted tool.

It is also pertinent to keep in mind that the currently rendered graphics might not be the same graphics that cause performance issues - hero events with fireworks and explosions might be the straw that breaks the camel’s back, causing stutters only when active.

When experiencing intermittent stutters, try to identify if a particular camera angle, triggered effect, or combination of these were what caused the issue. Reproduction helps narrow down possible causes!

Pixotope Utilities

There are tons of convenient utilities to help debug this, and the most useful ones are all readily available within the Pixotope "Utilities" window. 

UnitGraph

The first thing you want to do is enable "UnitGraph", and review your frame timings.

If the green "Frame" timer is a steady green line, and the counter shows your target MS, then you are not currently experiencing performance issues.	If the green "Frame" timer is uneven and/or has vertical spikes, we have been, or are, experiencing frame drops!

If we are experiencing frame drops, note the values for Game, Draw and GPU Time. The total rendering time for the final Frame is typically clamped to the highest of these values.

Game

If we are stuck waiting on Game, we are typically CPU bottlenecked, and there is too much happening in terms of blueprint logic or other CPU bound computations.

Draw

If we are stuck waiting on Draw, we are also CPU bottlenecked, but the issue is more likely to be too many draw calls/culling checks, caused by too many individual actors in our scene.

GPU Time

Lastly, if we are stuck on GPU Time, our issue is the GPU workload. This is both the most frequent problem, as well as the one with most possible culprits. Luckily, we also have access to an excellent tool to break down the GPU render process - ProfileGPU!

ProfileGPU

Access the GPU visualizer by

• clicking ProfileGPU in Pixotope Utilities

• typing ProfileGPU in the command window

• typing Ctrl + Shift + Comma

This also works in Live mode!

As seen in the image above, we are rendering our entire scene in less than 3 ms. We can hover our cursor over the colored sections of the "Scene" bar, or expand the "Scene" section in the lower explorer, to see where those 3 ms are spent. 

The bigger green and cyan elements near the middle of the Scene color bar are ShadowDepths and Lights, informing us that we spend about a third of our rendering time simply rendering lights and shadow. A step we could then logically take to reduce our rendering time, would be to either disable shadowcasting from some of our lights, or remove and reduce our total light count! 

There are tons of other advanced profiling tools in the Utilities window, each useful for certain tasks, but ProfileGPU is the most powerful performance overview utility for the average artist.

The usual suspects, and how to address them

CPU bottleneck

CPU below spec

If the CPU is below spec, simply running Unreal alone can be enough to cause stalls.

• Solved by buying better hardware

Check out our System requirements

Too many draw calls (Individual assets being rendered)

Typical in big, open environments like cityscapes, where they may be hundreds of thousands of instances actors scattered around.

• Using CullDistanceVolumes, or Hierarchical Instanced Static Meshes (HISMA), is a great way to DRASTICALLY reduce the amount of necessary drawcalls

• Depending on the issue, → Instance Tool may be a magic bullet here

• Another solution is to simply go through and delete all the content you are not using. If you simply wish to render a busy city street as a backdrop for a shot - do you really need to render the ENTIRE city?

Too complex, or simply too much blueprint/code logic

Having a single actor doing a few linetraces per tick, is usually not an issue. But having a thousand moving objects doing overlap checks against all nearby actors every tick can immediately stall your engine to single-digit frame rates.

Physics, particles and skeletal meshes

GPU bottleneck

GPU below spec

The GPU is, quite literally, the single most important piece of equipment for performance in Pixotope. As we are using consumer level hardware, it is also a quite affordable upgrade. Never try to cut costs here, and upgrade regularly.

Check out our System requirements

"Too much" content in the scene

This can mean many things, and is rarely an issue by itself, but having hundreds of thousands of individual pieces of foliage, all rendering without culling, will easily start causing GPU performance issues (too many shadow/vertex calculations).

• A good rule of thumb is to only use geometry that you need. If you are doing a simple ocean view shot, do you really need an entire city scape behind you, if it’s never in view? Could that bought Los Angeles city pack be replaced with a few layered 2D mattes?

• Nanite generally removes any need to worry about vertex counts, but at the time of this writing, foliage is one of the exceptions.

  • Issue can be resolved by using less poly heavy foliage actors, turning off foliage shadowcasting, or using bigger and more sparsely scattered foliage actors

  • Another solution is to cull foliage more aggressively

Too complex shaders occupying too much screen space

Too complex shaders occupying too much screen space. Parallax with hundreds of layers can make a single cube stall the engine, for instance.

• Using the shader complexity view (Alt + 8 in the editor) can help identifying the problem area, but might not catch things like parallax. Once you have identified the problem actor(s), inspect the materials, and contemplate using cheaper solutions to achieve similar results, or a new material altogether if you are facing something like a grossly overdimensioned master material when trying to make simple materials.

Overdraw abuse

Overdraw abuse - too many particles, like fog or smoke, causing transparency overdraw. Every additional layer is one more round of processing for each affected pixel, and this quickly adds up. 

• Try to avoid vast amounts of overdraw. Don’t use massive particle cards that are 90% invisible - it increases cost for no added benefit. Look up using BVC for polygonal cutout optimizations, as well as try to use as few particles as possible for the effect you want to achieve. Fewer, bigger, denser particles are better than many faint overlapping ones!

Shadow costs

Shadow costs. This is one of the BIGGER problems, caused by too many light sources with too much attenuation radius.

• Spotlights are relatively cheap, area lights and point lights are more performance intensive, and directional lights can be extremely costly. A good rule of thumb is to only have one shadowcasting directional light

Expensive render features

• Expensive render features - from volumetric fog, to multi-bounce raytracing, to complex post processing, can all take large chunks of time to process.

  • If you can’t afford it, you can’t afford it. Turn off that fancy convolution based bloom. Uncheck "Volumetric" in your exponential height fog. Set your raytracing bounces to 1, or turn it off outright. These are luxuries you treat yourself to, once you’ve trimmed the fat elsewhere in your pipeline.

Video I/O bottleneck

Additional video inputs and outputs

Every additional video input or output in Pixotope is processing that forces the rendering process to wait until the image transfer in the hardware has completed

• There are some hardware implications here where incorrect system configurations can literally double the time it takes for video frames to transfer. Please refer to the hardware specifications. 

Large amount of SDI inputs

Using a large amount of SDI inputs, or UHD feeds where HD’s would suffice, will incur heavy render penalties

• Pixotope is great at many things, but it is NOT a video switcher. As an example, rather than setting up multiple media inputs for virtual video wall needs, consider using a single media input with the video switching happening upstream from Pixotope

Also check out Epic’s Performance and Profiling in the Unreal engine
→ https://docs.unrealengine.com/en-US/Engine/Performance