PHENICX, a project of the European Commission's 7th Framework Programme coordinated by Emilia Gómez, a researcher with the Musical Technology Research Group (MTG) of the Department of Information and Communication Technologies (DTIC) at UPF, has attempted to create new digital experiences to enrich the experience of a classical music concert (before, during and after the concert itself) from different areas in order to bring classical music to new audiences in an innovative way and via technology.
"One of the applications we investigated in the post-concert context were a type of facility in which the user takes on the role of conductor so that their movements "conduct" a virtual orchestra. This type of facility has been successful in places such as the House of Music in Vienna", says Álvaro Sarasúa, first author of a study published on 25 February in the journal Frontiers in Digital Humanities, directed by Emilia Gómez, both members of UPF's MTG, together with Julián Urbano, a researcher at the Delft University of Technology (Netherlands).
The authors used the conductor metaphor
In human-computer interaction, interface metaphors provide users with a way to interact with the machine that resembles a known activity and are widely used in digital musical instruments. In this study, the authors used the conductor metaphor. The interest in using these metaphors is so that the user, in a new virtual context (in this case, a facility in which to control the tempo and dynamics at which an orchestra plays), can apply their knowledge or their prior intuition about how a certain activity functions in the real world (in this case, conducting an orchestra).
"From the research point of view, in this study we were interested in determining how intuitive notions available to users of this kind of facility may be explicitly used by the system to respond to their gestures of conducting an orchestra more intuitively and, therefore, may provide a more expressive performance", says Sarasúa.
In the initial stages of the project, the researchers conducted observational studies where they asked people with different degrees of musical training to listen to several different excerpts of classical music and then, while the music was playing again, to make the movements they would have made conducting an orchestra to communicate various aspects of it such as tempo and intensity. They recorded the participants' movements using a Kinect camera, which recorded the position of various parts of the body.
Different interpretative tendencies could be seen in people
Analysis of these data revealed that different tendencies could be seen in people. On the one hand, to communicate the tempo of the music, there were people who consistently communicated beat somewhat ahead of the tempo of the music, while others did so in total synchrony or with some delay. Regarding the dynamics with which the instruments played, some people tended to move more energetically during the forte parts, while others simply raised their arms more.
Based on these observations, "our starting hypothesis for this study was that a system that allows controlling the tempo and volume of a piece of music by movements mimicking those of a conductor may be more intuitive if it can be adapted to these personal tendencies" explains Sarasúa
"In the example of our experiment, there is an "implicit" advantage by using the metaphor, in that by telling the user that they must "conduct" the orchestra, they know that they must move their arms and have some intuition as to their ability to control tempo and dynamics. Using the proposed system, this improvement is more "explicit" in that the system observes and analyses what the user senses they must do and adapts to perform according to these intuitions"
The authors refer to this strategy as mapping by observation, since mapping, or the way in which the user's movements are connected to the resulting music, specifically adapts for each user by observing what they do when asked to perform movements imitating a conductor without giving any specific instructions.
Two systems were compared on the basis of objective and subjective measures
In evaluating the system, the researchers had 24 participants who performed, first with a baseline system that did not adapt to each user, and then the system based on the mapping strategy, and they compared the results. This comparison was carried out in a context in which users learned to "conduct" the virtual orchestra through practice. The two systems were compared on the basis of objective and subjective measures on a series of tasks in which the participants had to make the orchestra play at different volumes and do so in synchrony with the metronome at different tempos.
The usability of the tailored system is better
The results of the experiment showed that the usability of the tailored system is better, both in terms of providing more intuitive control of the dynamics with which the user conducts the orchestra, and in terms of having more precise control over the tempo. Furthermore, the results also show a strong correlation between measures that can be drawn from the data used to make the adaptation and improvement introduced by the system, which indicates that it is possible to estimate in advance the extent to which the spontaneous movements of a particular user will be really useful to adapt the system to their style.
The findings of this study are particularly relevant in contexts where interaction is designed from using interface metaphors, as is the case of the conductor of an orchestra. "In the example of our experiment, there is an "implicit" advantage in using the metaphor, in that by telling the user that they must "conduct" the orchestra, they know that they must move their arms and have some intuition as to controlling tempo and dynamics. Using the proposed system, this improvement is more "explicit" in that the system observes and analyses what the user senses they must do and adapts to perform according to these intuitions", the authors conclude.
###
Journal
Frontiers in Digital Humanities