See. Hear. Marvel.New audiovisual feature at the Musical Instrument Museum

It’s not every day you come across something like the Sound & Vision Experience Lab at the State Institute for Music Research (SIM). The two researchers behind the project explain how the lab works and why it can offer us insights into current issues in music research in an unusual way.

“I only buy vinyl records. The sound is simply better.” Who hasn’t heard that before? Lately, even cassettes are back in fashion. Is it just nostalgia among music geeks, or is there some truth to it? How significant are the differences between the various music storage formats really? At the Sound & Vision Experience Lab (SV_XL for short) in the basement of the Musical Instrument Museum, you can compare them in real time. The lab is not yet open to the public. SPKmagazin gets an exclusive sneak preview to listen and watch, and speaks to the two scientists behind the mysterious music lab: Michael Horn and Hans-Joachim Maempel.

What can the Sound & Vision Experience Lab do?

Hans-Joachim Maempel: The lab is designed to bring the methods and findings of musicology to life. It is a dedicated space equipped with a system for high-quality audio and video playback, which is also 3D-capable.

Michael Horn: The system is interactive: whilst a performance is underway, you can alter influencing factors in real time and immediately perceive their auditory and visual effects. The whole concept is open-ended in terms of content, meaning that demonstrations on a wide variety of topics are technically possible. The lab can thus also be used to conduct new experiments, so it serves a dual purpose for both education and research.

What was the motivation behind the project?

HJM: In systematic musicology, which also deals with acoustics and musical perception, experiments are frequently carried out. The methods and results of such studies are difficult to convey to interested parties who are not specialists in the field through text and graphics – partly because there is a great deal of statistics involved.

If, on the other hand, one can experience the test stimuli for oneself, it immediately becomes clear, without the need for many words, what kind of stimuli are used in the experiment, how they are varied and how they might affect the listener. One experiences, so to speak, both the research method and the result.

MH: The specific impetus came from a research project funded by the German Research Foundation (DFG) on audiovisual spatial perception, for which we had created the Virtual Concert Hall at considerable effort. In it, one can hear and see music and speech in 3D simulations of real concert halls.

Because the simulation system had sparked many questions and a great deal of interest among both test subjects and visitors, we decided to set up a similar system for the interested public. When setting up the SV_XL, we were therefore able to draw on the existing conceptual and technical experience of the Virtual Concert Hall.

At first glance, the set-up resembles the control centre of a spaceship: at its heart is a so-called Hardware User Interface (HUI) – featuring numerous illuminated buttons, knobs and displays – through which the demonstrations can be operated. The entire experience is delivered via a complex speaker array and a screen with 3D playback capability.

What makes operating the SV_XL so special, and why did you decide against a touch screen as the user interface?

HJM: Of course, you could control everything with a touch screen, and cheaply at that. However, we believe that many people have grown rather bored of touch screens by now. Our aim of creating a control console that is unique in design and feel, and which radiates an invitation to experimental interaction, is hardly achievable with a touch screen.

From an ergonomic perspective, a touch screen also has a major disadvantage for an audiovisual laboratory: you have to actively search for and locate the controls on it. We don’t want visitors to be distracted from the large display showing the visual demo content. You might also want to be able to close your eyes for certain listening comparisons.

Conventional tactile controls, whose positions are fixed and which can be felt, are therefore advantageous. Categorical parameters (e.g. music track) can be adjusted using buttons, whilst continuous parameters (e.g. distance or volume) can be adjusted using rotary encoders.

MH: With a view to mechanical stability and a pleasant feel, we also placed great emphasis on special controls such as metal rotary knobs and illuminated sensor buttons, high-quality materials such as acrylic glass and aluminium, height and tilt adjustability, and meticulous craftsmanship.

Due to these specific requirements, we not only had to develop the HUI from scratch ourselves, but also build it ourselves in a DIY manner – from the design and mechanics to the electronics and programming. At least we were able to draw on many open-source libraries.

As a result, it is therefore less of a fully-fledged device and more of a proof of concept with a certain retro feel in the style of the 1990s.

Can you give an example of an audiovisual demonstration that can be selected here?

HJM: One thematic focus is on questions concerning the sonic realisation of music. The demo topics cover both fundamental popular and specific experimental scientific questions. Many people are interested, for example, in the sonic differences between various sound carriers or the interpretative differences brought about by musicians or sound creators. We have identified these influencing factors and thus made their various manifestations comparable in the laboratory.

MH: Experiences that one cannot actually have in real everyday life include acoustic performance spaces that do not correspond to the visual performance spaces, or the interior of an organ chamber that one is physically not allowed to enter.

What were the unexpected hurdles and difficulties you encountered during the preparation and implementation of the project?

MH: We underestimated the technical complexity of a system with open-ended content that needed to be developed from scratch, for which there is no blueprint. Each demo follows its own unique workflow logic, requires different parameter sets, and in some cases different input and output devices, and is based on entirely different open-source applications and libraries. We also underestimated the duration of the work due to the high degree of diversity and the depth of production: preparing the space, procurement, drilling, screwing, soldering, designing and 3D printing of special parts, programming, literature research, graphic design, testing, and clarifying rights.

What are the most surprising insights you’ve gained so far from your own experiments and feedback from early users?

MH: Actually, what applies to all demos is that identifying the influencing factors that can be altered in real time appears to be very instructive. This is evident in the popular discussion comparing analogue and digital audio media. If you keep all non-media-specific factors constant – for example, by ensuring synchronous playback and equal loudness across vinyl, tape, cassette, CD and MP3 – many people are surprised to find that the actual differences in sound between the media are not nearly as great as one might expect given the passion of the discussions in relevant forums.

HJM: Conversely, many are surprised by the diversity of the music, stemming from the musicians’ primary interpretation as well as the producers’ secondary interpretation during audio processing. What is generally striking is the visitors’ eagerness to discuss the matter following each demo.

Can SV_XL advance musicology?

HJM: There are, of course, more than enough interesting research questions in the fields of acoustics and music psychology. Not all of them can be resolved empirically. The lab allows for listening tests using virtual acoustics – but also experiments with acoustically and visually varied test stimuli. This makes it ideal for research into questions of audiovisual music perception, e.g. the role of hearing and seeing in the impact of musical interpretation.

MH: As the laboratory offers the possibility of recording reactions to specific stimuli independently of any particular research question, users can, with their consent, become test subjects themselves.

What happens next with the project? Will the SV_XL also be open to visitors to the Musical Instrument Museum?

HJM: The project was time-limited and has now concluded. The future technical maintenance of the system and the expansion of the demo portfolio’s content remain unresolved. We will try to mitigate this somewhat with our existing staff. So far, seven demos have been implemented on restoration methods, loudspeaker reproduction techniques, sound carriers, musical interpretations, mixing, listening positions and performance spaces. These will be demonstrated to visitors to the Musical Instrument Museum regularly at specific times in small groups. We will decide, based on our experience with the guided tours, whether and under what conditions the laboratory can eventually be opened without a guide and thus for a longer period each week.