A study published in the Journal of Electrical and Computer Engineering by members of the Music Information Research Lab of the Music Technology research group (MTG) of the Department of Information and Communication Technologies (DTIC), presents the development of a technology that separates the sound corresponding to the different instruments involved in an orchestra sound recording (mix). This work is part of the European project PHENICX, coordinated by UPF’s MTG group, whose aim is to transform how classical music is listened to. Indeed, by separating the audio corresponding to the different instruments the authors have been able to implement interesting applications, for example, by focusing our listening on certain instruments or recreating experiences of the concert in virtual reality.

In this video you can hear first a part of Ludwig van Beethoven’s Eroica, followed by the same piece but separating the different instruments.

In this demo of the PHENICX project the user can select the different instruments and listen to them separately.

In their study, the authors propose a method to separate sources based on a much used convex optimization technique called Non-negative Matrix Factorization (NMF).

As Marius Miron, the first author of the study, comments: “this approach improves if we have recordings of the piece with multiple microphones, if we know what instruments play in the piece and if we know what notes are played by each instrument. In fact, the more information we have about the piece, the more we can restrict our model and the better we can perform the separation”.

“For orchestral music we know what instruments are playing, so we can train timbre models for each instrument. In addition, we have the scores of all of the orchestra works, which can help us to improve the separation”, he added.

In addition, the authors wanted to simulate a real recording to control factors such as reverberation, the position of the microphones in the hall, the number of instruments in an orchestra section, etc. “Like that we can design robust evaluation strategies by analysing the way all these parameters influence the quality of the separation. As far as we are aware, this is the first time that a database of this type has been proposed for this scenario: orchestral music”, explained Marius Miron.

PHENICX anechoic, a tool that evaluates the separation of sources objectively

To evaluate the separation algorithms of sources objectively requires databases which, in addition to the final mix, have the separate recordings of the instruments used to generate the mix. If you do not have this algorithm, the only solution to perform the evaluation would be to use human listeners, which would be very costly and difficult to replicate.

Therefore, in this work the authors propose the use of PHENICX anechoic, a database that has not only allowed them to evaluate their own system, but can be useful for future research in this area.

Currently, the MTG’s Music Information Research Lab is working so that the method developed does not dependent so much on the score. Hence, they are working on other musical styles, and are integrating advanced algorithms of deep learning that can be used in a low latency scenario.

Reference work:

M. Miron, J. Carabias-Orti, J. J. Bosch, E. Gómez and J. Janer, “Score-informed source separation for multi-channel orchestral recordings” (2016), Journal of Electrical and Computer Engineering, vol. 2016, ID 8363507, 19 pp. http://dx.doi.org/10.1155/2016/8363507