At first glance music and computer science might have little to do with one another, but Prof. Dr. Meinard Müller thinks differently: he aims to use artificial intelligence to analyze complex characteristics and hidden relations in music.
Professor Müller is an expert in semantic audio signal processing at Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) and his research deals with analyzing well-known music corpora and developing apps for recognizing music.
To expand the freedom and scope of his work, the German Research Foundation (DFG) is providing funding of 1.25 million euros as part of the Reinhart Koselleck Program.
Analyzing music data as an independent field in computer science
Meinard Müller likes striking the keys. For example, when he plays Schubert, Liszt und Chopin on the piano, or while he is writing a new algorithm to help analyze classical music. Müller disagrees that music and computer science are two separate worlds “I want to use one to understand the other better,” he says.
Meinard Müller has been Professor of Semantic Audio Signal Processing at the International Audio Laboratories Erlangen, AudioLabs for short, since 2012. AudioLabs is a joint project of FAU and the Fraunhofer Institute for Integrated Circuits IIS – the institution where the legendary MP3 format was developed.
“The way in which music is presented, used, disseminated and stored has evolved rapidly in recent years,” explains Müller. “Today there is great interest in technologies and tools for managing music-related data.” With his work, the computer scientist has made a decisive contribution to developing Music Information Retrieval (MIR) into an independent field of research.
His early enthusiasm for music and computer science was certainly very helpful: At the age of six, Meinard Müller already played classical pieces on the piano and has remained faithful to both the instrument and the musical genre, “even if it was not enough for a career as a musician”.
Later, before graduating from high school, he acquired basic programming skills. However, Müller chose mathematics as a major at the University of Bonn, “the perfect training for my current work,” he explains.
Computer science, which he originally studied as a minor subject, became the focus of his doctorate and from then on shaped his academic career. After a stay as a postdoctoral researcher at Keio University in Tokyo, Müller devoted his work to signal and music processing and especially digital music libraries. Through a research position at the Max Planck Institute for Computer Science in Saarbrücken, his path eventually led to Erlangen.
Deep learning makes hidden connections visible
Meinard Müller has named his current research project LEARN. The name considers different perspectives: Firstly, it alludes to deep learning technologies that are designed to extract complex features and hidden relationships directly from music signals.
As with the analysis of image data, this involves pattern recognition, in this case of pitches, chords, rhythm and lyrics. This results, for example, in algorithms that are able to find the right song based on a hummed melody or to make suggestions for songs with a similar rhythm.
Meinard Müller is involved in the development of such apps. “Deep learning has led to tremendous improvements in analyzing music data,” he says. “Today we are able to decipher the sources of multi-track recordings, such as vocals, accompaniment, drums or bass, in detail.”
Secondly, Müller also wants to make a contribution to the Digital Humanities and analyze scientifically or culturally relevant music corpora. “Look at Wagner’s ‘Ring of the Nibelung’ – that’s sixteen hours of music. If you want to work out harmonic-structural references, you won’t get far by looking at sheet music.”
In such projects, the computer scientist worked closely with musicologists, a cooperation that benefits everyone, as Müller emphasizes: “The mutual feedback leads to progress in both disciplines. The algorithms reveal new connections, not only within the pieces, but also in comparison with works by other composers and even different epochs. The musicologists, on the other hand, can tell us whether the discovered references are only artifacts of artificial intelligence or actually relevant.”
Music facilitates access to mathematics, physics and computer science
The third perspective of LEARN includes feedback, which serves the further development of machine learning itself: Müller uses deep learning algorithms to analyze music, but at the same time processing music data can give new impulses for AI in general.
“Music is wonderful, but not trivial,” says Müller. “We are dealing with a wide range of data types, formats, tags and metadata.” In order to use algorithms with this wealth of data, the computer scientists at AudioLabs want to develop new models that are less susceptible to imbalances and disruptive factors. Müller: “I can well imagine that our findings will advance AI as a whole.”
Fourthly, Meinard Müller is concerned with learning in the sense of imparting knowledge: “Behind every sound there are hidden physical parameters – frequencies, amplitudes, timbre. Music can therefore be a motivating medium to introduce young people to the fundamental principles of physics, mathematics, computer science and engineering.”
As part of his teaching activities at FAU, Müller strives to initiate interactive and challenging research projects for students and doctoral candidates. He often finds inspiration for his own work: “Music formats, tools, programming languages – all this is constantly changing, and sometimes it is my students who bring me up to date.”
What unites both sides is the interest in both computer science and music – most students play one or more instruments. Müller: “You can say without exaggeration that we have a lot of fun in our work because we love our data.”
About the DFG Reinhart Koselleck funding
Reinhart Koselleck projects give researchers more freedom and scope to work on particularly innovative and promising but higher-risk projects. With this funding, the DFG hopes to give outstanding researchers the possibility to carry out projects such as these. The projects are provided funding of up to 1.25 million euros over a period of five years.
Reinhart Koselleck funded projects at FAU