Interview with Emilia Gómez: "PHENICX is a new way of experiencing a classical music concert"

PHENICX (February 2013-February 2016) is a project funded by the European Commission within the 7th. Framework Programme. The consortium consists of technological partners (Pompeu Fabra University, Delft University of Technology, the Austrian Research Institute for Artificial Intelligence, Johannes Kepler University of Linz), musical partners (the Royal Concertgebauw Orchestra, Amsterdam and the Catalonia College of Music) and the Dutch company Video Dock.

Emilia Gómez, a researcher in the Music Technology Research Group (MTG) and the Department of Information and Communication Technologies (DTIC) at Pompeu Fabra University, is the lead researcher of PHENICX. She is a Telecommunications Engineer (University of Seville), a musician (the Music Conservatory of Seville) and holds a doctorate in Computer Science and Digital Communication (UPF). The research focuses on the combination of sound processing, music theory and artificial intelligence.

The members of the MTG-DTIC involved in PHENICX are Alba Rosado (project management) and the researchers Agustín Martorell, Jordi Janer, Julio J. Carabias, Enric Guaus, Álvaro Sarasúa, Òscar Major, Joan J. Bosch, Marius Miron and Carles Fernández.

- Is PHENICX a different way to enjoy the experience of music?

Yes, the main aim of PHENICX is to create new digital experiences that allow the public to access, understand and thereby enjoy concerts more. We focus on classical music in the project, and the symphonic repertoire in particular, bringing it to new audiences in an innovative thanks to the application of technology.

With PHENICX, the concert becomes a digital object that is complemented with tools aimed at improving the the audience's experience during the preparation stage prior to the concert, and during and after the event. These include various types of information (audio, video, score, text, gestures, etc.) and superimposed layers (such as analysis of the work, performers, musical reviews, social networks and other resources that are available, such as on the Internet). These data can easily be explored, customized and shared using a mobile, tablet or computer.

- What tools developed by the Music Technology Group are part of the project?

In the project, we face the challenges that symphonic music poses for the modern technologies of music and sound processing, video analysis, information retrieval, social networks, human-computer interaction and personalization of content. At the MTG, we primarily work in four areas of technological research and development. First, we are continuing with the development of the platform Repovizz which enables simultaneous remote access, interaction and sharing of the multimodal data that make up digital concerts (sound recordings, scores, videos and textual information). Here's a video of how it works:

So this way we can "connect" to the information about the concert, for example, to create a customized hand programme. Second, we research and develop technologies for the automatic description and visualization of music. The listener can use them while listening to the piece to view various characteristics that help to understand and enjoy it better: what the predominant melody is, which instruments are playing at each point in time and the structure of the piece, which can help to place you within it, or even know when you have to applaud. You can see an example in this video:

Third, we are developing source separation technologies that analyze the sound signal and enable the different instruments of the orchestra to be isolated or can virtually move the user to different positions in the concert hall. You can hear an example of this technology here:

Finally, we are researching and developing technologies that capture the conductor's gestures to study the different styles of conducting, and we develop tools for learning how to conduct an orchestra virtually. We can see an example in the video.

- What is the main innovation in the PHENICX project?

The most innovative parts of the project are that it can "integrate" all the available information about a classical music concert; it can enrich this information using automatic techniques, such as a description and following the score, sound, video and gesture; and finally, it can customize access to this information based on each listener's interests, needs and prior knowledge. Furthermore, the use of technology in classical music is already something innovative in itself, which we hope will bring this music to new audiences and encourage the preservation of this cultural heritage for future enjoyment.

- Have social networks helped to connect with the audience?

Of course. Because we can use social networks to find out each person's particular interests and personalize the concert experience and connect with audiences with the same interests. We can also find out users' opinions about what the most exciting parts of the concert are, for example.

- What impressions do you have of the users so far?

So far we have had a very positive response to the prototype we have developed, from classical music lovers, musicians, and from people who don't often go to classical music concerts and who use the technology we have developed to make this repertoire more accessible.

Orchestras have also shown an interest in the tools we are developing. This was made clear in the recent presentation of the project at the AEOS Conference organized by the BBVA Foundation.

- What challenges are you setting yourselves for the near future?

In the project, we have been discovering more applications that could help improve the experience "during" the concert which we want to explore. We would also like to be able to adapt the tools so that they are useful for the musicians performing the works as well as the audience, and so that they are optim ized for other repertoires.