UPF will be coordinating the European project, Emerald, to develop artificial intelligence (AI) applications to make the production processes of digital media, the audiovisual sector and digital entertainment more efficient and reduce their level of energy expenditure. Faced with the strong increase in the production of audiovisual and augmented reality content in recent years, the Emerald project seeks to help make these sectors more sustainable.

The interdisciplinary consortium promoting the Emerald project, made up of six European partners besides UPF, is being coordinated by Josep Blat and Coloma Ballester. Blat and Ballester are the coordinators of the GTI (Interactive Technologies Group) and IMVA (Intelligent Multimodal Vision Analysis) research groups, respectively, both at the UPF Department of Information and Communication Technologies (DTIC).

Also members of the Emerald consortium are the BBC, Trinity College Dublin and four companies supplying the necessary technologies to develop media or audiovisual products (Brainstorm Multimedia, Disguise Systems, FilmLight GmbH, and MOG Technologies).

Earlier this March, the European Union’s Horizon programme granted €3.1 million to develop the Emerald project, which will research and develop AI applications, based on Machine Learning technologies for the media and audiovisual sector. The two-and-a-half-year project will begin in October.

The coordinator of the GTI (DTIC-UPF), Josep Blat, summarizes the reasons that have led to the Emerald project: “Film, television, series... consume a lot of energy, both in the creation and in the consumption of audiovisual products, and artificial intelligence, based on processing a lot of data, also accounts for high energy expenditure. This needs rethinking. This is what the project will do: generate more sustainable tools for these sectors, as well as tools to measure energy expenditure in the different processes”.

Meanwhile, Coloma Ballester, the IMVA coordinator, says: “the digital media industries, where huge volumes of digital data are processed and post-processed, will benefit from research and transfer and will incorporate artificial intelligence, automating their tools and processes, while raising the quality of digital content and at the same time increasing the speed of production and reducing the demand for energy and resources”.

Emerald’s lines of action for applying AI to the media and audiovisual sector

To this end, the project will work with various goals. On the one hand, it will promote the automation of production processes, to save time and resources for the media or producers of audiovisual or entertainment content. On the other, it will develop AI-based machine learning applications to facilitate data processing in the industries of the sector or improve and optimize the quality of digital and audiovisual content. Work will also be carried out to promote energy savings and efficiency in these sectors. To monitor the results, a test bench will be created to measure and calculate the energy used by sector media and industries.

What tools will Emerald develop to improve the sustainability of these sectors?

The project will develop algorithms, techniques and tools to automate and improve the sustainability of specific tasks of media and audiovisual production processes. Some of them will be used to improve and optimize video editing and post-production, for example with regard video matting or work to correct the light and colour of films or audiovisual products after their assembly, and to reduce the bandwidth and energy needed to store and process them in the cloud.

Other tools, related rather to the dissemination of journalistic information, will be used to extract data from specific news or to synthesize or modify its contents. In addition, low-power devices for content analysis and new, more sustainable tools for relaying news will be created.

Other technologies will be more linked to sports information. Techniques will be developed that apply AI to the analysis of sport-related information and audiovisual content, as well as improving computer vision techniques that allow estimating the position of people or targets in real time during live audiovisual broadcasts. Also, a web server format will be created that is capable of automatically enriching the so-called metadata (descriptors of other data) in digital and audiovisual content from the world of sport, as well as to improve streaming. Other tools that the project aims to develop are linked to the application of AI in the world of animation and in the field of virtual and augmented reality.

Emerald also plans to develop prototypes of the tools designed to conduct demonstrations, which may encourage their subsequent development by other companies in the sector and their future entry into the market.

The European partners of the Emerald interdisciplinary consortium

UPF, through the GTI and IMVA research groups (DTIC-UPF)

UPF is coordinating the consortium behind the Emerald project, through the GTI (Interactive Technologies Group) and the IMVA (Intelligent Multimodal Vision Analysis Group, formerly IPCV) both at the Department of Information and Communication Technologies (DTIC) of the University. The GTI focuses on the human aspects of technology, especially on encouraging and improving the social use of ICTs, and develops projects related to interactive systems, online interactive 3D graphics, access to augmented reality in the field of information, digital inclusion, the expressiveness of avatars, and didactic games. For its part, the IMVA focuses its research on the analysis and understanding of visual information, often based on different modalities beyond vision, such as audio information, natural language and even ultrasound or magnetic resonance. The group develops both model-based and machine learning-based proposals, leading to technologies that take advantage of the correlations and complementary nature of different modalities, which are applied to improve the accessibility of people with visual, hearing or reading difficulties to multimedia content or to optimize video editing and post-production processes, among others. 

Trinity College, Dublin (Ireland)

Trinity College Dublin, a university with more than four centuries of history, is the best positioned in Ireland in the European and international rankings of teaching quality and innovation. It has three different faculties: Science, Technology, Engineering and Mathematics; Arts, Humanities and Social Sciences; and Health Sciences.

BBC

The British Broadcasting Corporation (BBC) is the UK’s public broadcaster. It is a leading public medium and has traditionally stood out for its model of impartiality and independence and for its production of high quality, educational and entertainment content for millions of people in the UK and around the world.

Brainstorm Multimedia

Brainstorm, based in Valencia and Madrid, specializes in real-time 3D graphics and industry-leading virtual set solutions for broadcasting, feature film production and corporate presentations.

Disguise Systems Limited

Disguise (UK) is a global provider of live event visualization solutions, specializing in providing software, hardware and support services that enable creative production teams to visualize, simulate and deliver their shows in 3D in real time.

FilmLight GmbH

FilmLight develops image processing applications and tools to facilitate audiovisual workflows, which are transforming film and video post-production and setting new standards of quality, reliability and performance.

MOG Technologies

MOG Technologies, a Portuguese company, is a global provider of digital media technology. It offers innovative solutions that enable digitally transforming any industry that needs to create, manage and distribute online content.