When Björk kicked off her Volta Tour in April 2007, the images from the inaugural concert at Coachella, in California, travelled around the world, piquing curiosity and creating all sorts of buzz. What was that on stage with her? Looming behind the Icelandic singer were two enormous screens offering real-time footage of what was happening on a round backlit table on which two musicians were moving strange objects to generate sounds.

Thus, four years after a group of researchers from UPF’s Music Technology Research Group began working on the reactable, the collaborative electronic musical instrument finally made the leap to the general public, with the resulting surge in popularity. However, even before Björk, it had already participated in numerous international festivals, from Ars Electronica, in Austria, to Sonar, in Barcelona, to Transmediale, in Berlin, winning the most prestigious awards.

‘Björk was preparing the visuals for the tour, and Michel Gondry [the film and music video director] told her about the reactable and showed it to her on YouTube. She liked it and shortly after we met in Paris to explain to her how it worked’, said Sergi Jordà, co-creator – along with Marcos Alonso, Günter Geiger and Martin Kaltenbrunner – of the instrument, which allows users to create complex and dynamic soundscapes, as if it were a programming language that could be directly manipulated.

Björk incorporated the reactable, invented by the Music Technology Group (MTG), into her 2007 Volta Tour.  

The reactable is surely one of the best known and most renowned projects, both at home and abroad, to emerge from UPF’s Music Technology Group (MTG), but it’s not the only one. In Japan, Hatsune Miku, a virtual singer whose voice is done with the voice-synthesizing software Vocaloid, has become a genuine mass phenomenon. It is another example of cutting-edge technology developed by the MTG, this time in partnership with Yamaha, one of the world’s most important musical instrument and electronics companies.

Both projects, the reactable and Vocaloid, bear witness to the MTG’s unique idiosyncrasy: pioneering multidisciplinary research focused on sound and music, close collaboration with industry, an international outlook, and knowledge transfer. ‘When we started in 1995, we were unique in Spain and one of just a handful in Europe and the rest of the world. We were doing really bleeding-edge stuff at a time when virtually no one else was doing it’, said Jordà.

The MTG played an undeniably decisive role in inspiring and driving the creation of the DTIC

The DTIC is born

The MTG’s unique and groundbreaking spirit inspired and largely drove the creation a few years later of the Department of Information and Communication Technologies (DTIC) at UPF, which has played a leading role in audiovisual research in Spain and Europe from its inception.

The MTG was part of the Audiovisual University Institute (IUA), an experimental research and digital media production institute that had been promoted by Xavier Berenguer and the then rector Rafel Argullol with the aim of conducting pioneering audiovisual research with an emphasis on technology.

However, ‘There came a point where UPF wanted to open itself up to new fields. Based on our research, they saw the opportunity to launch degree programmes in computer science and audiovisual systems’, recalled Xavier Serra, the driving force behind the MTG’s singularity. In fact, within the IUA, there were already other groups working and doing research in the field of image and video, although the MTG was the most technological one. ‘We played a decisive role in the department’s creation’, said Serra.

Josep Blat was the person tapped by UPF to create a new department dedicated to information technology. At the time, Blat, a mathematician by training, was doing research on computer graphics and creativity algorithms at the University of the Balearic Islands, where he had founded the first European master’s degree programme in computer animation. He thus had a multidisciplinary profile and a clear passion for research that bridged the gap between creativity and technology, which was exactly what UPF was looking for.

The first thing he did upon taking the reins of the department in 1999 was to launch the undergraduate programme in Computer Engineering with a specialization in audiovisual media. That programme was followed by others in Audiovisual Systems Engineering, Telecommunication Networks, and, later, in Biomedicine and in Mathematics in Data Science. As part of its teaching activity, the DTIC also offers six research master’s degree programmes and a doctoral programme in ICT.

‘From the start, it had a clear interdisciplinary outlook and was very open. You have to remember that UPF was a very young university, with young people and a highly internationalized staff, and we’ve always attracted researchers from elite programmes, such as ICREA or Ramón y Cajal’, Serra added.

‘We started partnering with some of the world’s most important companies in this sector from the very beginning. Twenty-five years later, we’re still working with the most innovative things being done in the audiovisual industry, in terms of both music technology and visual effects’, said Blat, who heads the DTIC’s Interactive Technologies Research Group.

Since its launch, the DTIC has consistently played a leading role in audiovisual technology and research both nationally and in Europe. It has 24 research groups, 41 permanent lecturers, including ICREA research lecturers, and over 200 postdoctoral researchers and other types of research collaborators, more than 60% of whom are foreign, in consonance with the department’s international approach.

Unlike other, more traditional ICT departments, the DTIC has had a special relationship with the cultural industries sector from the start. In fact, this relationship is part of its identity and one of its founding traits.

‘People think engineers cannot do art and be creative. We’ve proved that that’s not true, that you can experiment with tools and technologies’, said Vladimir Estivill, head of the department, for whom the DTIC ‘has a more universal and very modern vision’. ‘Our researchers are very open-minded and explore the possibilities of technology for the creative industries. For example, when we created the DTIC, our lecturers were working on issues related to CD-ROMs. Now they’re delving into topics related to the metaverse’, he added.

DTIC researchers thus share the common challenge of developing culturally sensitive information technology that facilitates universal access to cultural resources. Hence, many of the department’s research lines are linked to cultural and creative industries (CCIs), such as the development of technologies for the consumption and creation of music and sound, image and digital content, cultural heritage, or educational purposes. To this end, it has backed the creation of six spin-offs.

‘From the start, it had a clear interdisciplinary outlook and was very open’, said Xavier Serra of the DTIC

In addition to technologies for consuming and creating music and sound, the field in which the MTG operates, the DTIC researches image and digital content technologies through three groups, including the Image Processing Research Group (IPG), led by Glòria Haro and Coloma Ballester. The latter was one of the first lecturers and researchers to join the DTIC to teach applied mathematics and conduct research in the visual field. Since 2013, she has coordinated the image processing and computer vision group, which was recently renamed the Intelligent Multimodal Vision Analysis (IMVA) group.

‘We investigate how we understand what we see, how our brains use the light that reaches the retina to understand what is in front of us. And we try to model that mathematically’, explained Ballester, specifying, ‘I’m interested in developing mathematical models that translate the complexity of what we see into algorithms capable of reproducing it computationally. For example, one goal would be to bring visual processes to people who are blind or visually impaired, intelligent systems that might include a camera and an earpiece to describe in real time the scene in front of them.’

Cultural technologies sensitive to educational needs

Unlike other engineering departments at other universities, the DTIC has focused its activity in the field of cultural and creative industries (CCIs)

The commitment to develop information technologies that are sensitive to educational and cultural needs, that facilitate universal access to cultural resources, support enculturation and promote creativity, has also been one of the department’s defining traits from the start.

For instance, the group led by Narcís Parés explores full-body interaction in areas of learning and play, as well as special needs. ‘My research is based on the possibility of incorporating the user’s body into the entire communicative cycle of interaction with the system. We have been working on the idea that the human experience must always be mediated by the body for twenty years’, said the researcher, who is currently participating in the Horizon Europe project European Media Immersive Lab (EMIL), which has received 8 million euros in funding and will aim to develop a pan-European network of extended reality (XR) laboratories.

Parés – who was already part of the IUA in the 1990s, where he did research and experimental production in virtual reality with his brother Roc, who is currently a lecturer in the Department of Communication – is a lecturer in the DTIC and leads the Full-Body Interaction Lab. He has developed installations such as the world’s first interactive fountains, made for the Universal Forum of Cultures (Barcelona, 2004), or, in 2008, a project in which an enlarged image of cave paintings of a hunting scene from Ulldecona (Tarragona, Spain) is projected on a screen, where it comes to life, allowing children to control the hunters. When they act alone, they find they are unable to hunt any animals; only when they develop team-based strategies do they succeed. Children with autism also have to work as a team in Lands of Fog (a project funded by RecerCaixa), an interactive mixed-reality space with full-body interaction in which children who are on the spectrum work with others who are not.

Thirty years of leadership

The DTIC’s success in the different European calls for research proposals in the field of digital media in the last decades bears witness to its leading role in audiovisual research. For instance, it has been awarded five grants under the Horizon Europe framework programme, the European Union’s new research and innovation programme for the 2021-2027 period. A good example is the MAX-R project, co-led by Blat and Boris Bellalta, focused on the creation of extended and augmented reality multimedia content. ‘MAX-R will develop the tools to enable the large-scale extended reality required for the metaverse and on-set virtual production, which is in high demand in the media industry’, said Blat.

Another project selected to receive European funding is FINDHR, in which Carlos Castillo participates. It is a consortium of twelve institutions that aims to fight discrimination in the field of artificial intelligence (AI). There is also ‘ReSilence: restoring the soundscape of future cities through the collaboration of art and science’, led by Leo Wanner, which seeks to improve the quality of urban soundscapes.

The DTIC’s success in the different European calls for research proposals in the field of digital media in the last decades bears witness to its leading role in audiovisual research; it has been awarded five grants under the Horizon Europe programme for the 2021-2027 period

The DTIC was also recently accredited for the second time as a María de Maeztu unit of excellence, after earning its first accreditation in the 2018 call. The distinction is awarded by the State Research Agency, a body attached to the Spanish Ministry of Science and Innovation. Together with the Severo Ochoa Centre of Excellence distinction, it is the highest form of institutional recognition awarded to scientific research in Spain. Both distinctions aim to fund and accredit public research centres and units, in any scientific field, that demonstrate impact and international scientific leadership and actively collaborate with their social and business communities.

‘Being recognized once again as a María de Maeztu unit is a remarkable achievement for the collective effort and commitment of our scientists and researchers’, said Estivill.

The distinction will entail 2 million euros in funding spread over four years for the DTIC, which was the first engineering unit to receive it and which has now firmly established itself on the national map of excellence. ‘This huge achievement undeniably recognizes the DTIC as the leading Spanish research department in our transdisciplinary fields in engineering, computing, biomedicine, neuroscience, interactive systems, music, computer vision and artificial intelligence (AI)’, said Estivill.