The international research consortium MAX-R, led by Pompeu Fabra University (UPF), has developed some twenty cutting-edge technologies on an international scale to produce and broadcast audiovisual content with extended reality (XR). Some of the tools developed allow designing XR content with body movements, using virtual reality glasses and hand controls. The BBC is one of the members of the consortium and it has already begun to implement some of the technologies resulting from the project, specifically the live broadcast of events on the metaverse, providing an immersive experience from anywhere in the world.

The concept of extended reality (XR) encompasses augmented reality (AR), virtual reality (VR), and mixed reality (MR). VR allows simulating an experience within an artificial environment, while AR enables complementing a real environment with superimposed digital resources. Lastly, MR not only combines digital and virtual elements, but also allows them to interact. Currently, much entertainment and information content already incorporates XR, but MAX-R has positioned itself at the forefront of this technological revolution, with the creation of a wide range of tools to generate top quality XR real time content, more efficiently, and greater potential for interactivity than available to date.

The European project MAX-R (Mixed Augmented and Extended Reality Media Pipeline) has been carried out over the last two and a half years by an integral consortium of 11 leading institutions and companies working in technology, creation and media (full list below). To achieve this, it was endowed with a budget of 11 million euros, nine of which funded by the EU Horizon Europe programme. Innovate UK has also contributed funding.

On behalf of UPF, consortium leader, two research groups from the Department of Engineering have taken part: The Interactive Technologies group (GTI) and the Wireless Networking (WN) group. Josep Blat, director of the GTI, explains: “MAX-R has substantially improved the efficiency and integration of virtual production technologies for cinema, television and events, as it allows integrating digital visual effects into the recording process itself —for example, on LED screens— with the involvement of actors, presenters or singers. In this way, all the elements are integrated more naturally and there are significant post-production savings”.

Boris Bellalta, director of the WN research group, adds: “We have developed a set of tools that are capable of dynamically adjusting the amount of data that is exchanged between the network and the glasses, mainly real-time images and data concerning movement or user actions. Although greater data exchange enables improving both the quality of the projected content and the feeling of immersion, the capacity of the network —which can vary unexpectedly— often limits its performance. Hence, the tools developed make it possible to detect these changes and adapt the volume of data transmitted in order to ensure the highest possible quality”.

Tools to enjoy experiences that combine real and virtual events

The technologies arising from the project can be applied in various fields such as audiovisual creation and graphic design, the media, interactive video games and immersive experiences, for instance. In addition to the above, tools have been created that allow broadcasting online events in which thousands of people can participate and interact at the same time; or mobile applications to generate virtual reality elements from real objects. In addition, tools have been created so that a journalist can present an event without actually being there (teleported).

In the field of interactive video games, optimized streaming technologies have been developed to share real-time augmented reality content among various players in Wi-Fi environments. Other tools render it possible to rationalize and make the production of XR content more effective, keeping the energy and economic resources invested down. Finally, state-of-the-art tools have been devised to improve the real-time light composition and colour grading of audiovisual productions.

Some tools have already been incorporated into reality shows and entertainment programmes

Some of the technologies developed by the project are already being used in information and entertainment programmes. For example, to create the images of the robot for the animated short film SURVIVOR, real actors were recorded in action to digitize their movements. The data captured from their movements was used to animate the robot realistically. The recording process was carried out by the German film school Filmakademie Baden-Württemberg, one of the MAX-R consortium partners. Also, the BBC group’s Radio 3 video game music programme, "The Sound of Gaming", has incorporated groundbreaking virtual experiences in some of its editions.

The wide range of tools developed by the MAX-R consortium is based on open source APIs (software interfaces) that will facilitate their integration into other applications.

11 partners that are leaders in technology, media and audiovisual production

According to their area of expertise, the 11 members of the consortium have focused on different types of tools, as set out below.

• UPF (consortium coordinator, Spain): the GTI research group has specialized in the creation of open source tools for generating augmented reality content. Meanwhile, its WN research group has dedicated itself to perfecting streaming systems to share virtual reality content among various people, especially interactive video game users.
• Disguise (United Kingdom): production of extended reality tools with rationalization and efficiency criteria.
• Foundry (United Kingdom): real time visual effects and metadata.
• FilmLight (UK): real-time colour grading.
• Brainstorm (Spain): integrated virtual production.
• ARRI (Germany): efficient production through reliable metadata and integrated work processes.
• Improbable (UK): massive interactive events.
• BBC (UK):  immersive live event streaming.
• Filmakademie Baden-Württemberg (Germany): toolkit for generating open source extended reality content.
• Hasselt University (Belgium): streaming and tracking of advanced extended reality content.
• CREW (Belgium): live large-area participatory performances with extended reality.

Many of the tools created will continue to evolve and improve, based on the consortium partners’ collaborations which will be pursued beyond the end of the project.

Video showing the process of creating 3D extended reality designs using the Rooms tool created by UPF:

Video that summarises the tools created by the MAX-R Consortium: