Virtual production is rapidly transforming the media and film industries, offering new creative possibilities and efficiencies. At BBC R&D, we’re exploring how this technology can be adapted for the television sector and lower cost productions.

In television, green screen virtual studios are already in use, especially in game shows and sports programming like Match of the Day, where multi-camera setups and post-production compositing are used with virtual backgrounds designed by digital artists. A more immersive alternative—LED-based live-action virtual production—uses massive LED walls to display real-time virtual environments, replicating realistic lighting and reflections on set. While visually impressive, these systems can cost tens or even hundreds of thousands of pounds per week, making them inaccessible for most TV productions. Within Emerald, BBC R&D are trying to prototype a scaled-down, cost-effective virtual production system which doesn't require a specialist facility or specialist operators, and can fit into typical TV production workflows.

Gaussian splats for virtual backgrounds

BBC R&D has previously explored the use of Neural Radiance Fields (NeRFs) for novel view synthesis in broadcasting—methods that generate new camera perspectives from a sparse set of photographs. More recently, we’ve been investigating Gaussian Splatting, a newer approach that offers significant advantages in terms of speed, interactivity, and real-time rendering.

Unlike NeRFs, which encode scenes as radiance fields within a neural network, Gaussian Splatting represents environments using millions of 3D Gaussian primitives. This allows for photorealistic 3D environments to be generated quickly from regular photographs, without the need for creating dense meshes or complex geometry pipelines. Gaussian splats as a 3D representation are visually and spatially accurate, editable, and lightweight enough to render in real time using standard game engines. This makes them well-suited for the creation of virtual sets.

For virtual production, this means real-world locations can be quickly captured and transformed into immersive, navigable virtual sets—without the specialist skills of traditional 3D modelling or added complexity of 3D capture techniques. Gaussian Splatting therefore has the potential to make advanced production techniques more accessible to a wider creative and research community at a much lower cost.

Camera tracking

A photorealistic virtual background—like one created using Gaussian Splatting—is only part of what is required of a virtual production workflow. To convincingly integrate real and virtual elements, the camera must be tracked with high precision in six degrees of freedom (position and orientation). This live tracking data is then used to project the correct perspective of the virtual environment onto the LED background in real time.

Professional studios often rely on proprietary tracking systems and hardware—but these are expensive and require specialist operators. In contrast, the robotics community has developed a rich ecosystem of open-source tools for real-time 3D mapping and localisation, such as SLAM (Simultaneous Localization and Mapping). However, these tools are rarely accessible or user-friendly for creative professionals and require some complex and integrations to be useful in a virtual production environment.

The system we’re developing uses AprilTags—a type of fiducial marker with a unique ID in the field of view. These black and white square tags act as a reference positional objects, in order to track the camera’s position. These tags can be easily placed around any filming location, including non-specialist spaces like offices or studios. A SLAM system is then used to scan the environment and create a 3D map that records the precise location of each tag. This map is fed into a camera tracking system based on PhotonVision, an open-source robotics vision platform, which uses the tag data to track the camera’s position and orientation in real time given an AprilTag layout. This live tracking information is then used to render the correct perspective of the virtual background on the LED screen behind the subject, ensuring the correct alignment between the physical and virtual elements.

Towards low-cost and open-source virtual production

By combining Gaussian Splatting with open-source robotics software, BBC R&D is developing a virtual production system that is portable and low-cost, designed to fit into existing TV production workflows and doesn’t require a dedicated virtual studio.  This system could democratise access to advanced production techniques typically only available to large-scale productions with high budgets.