Appearances matter most in musical performance

This is a long version of the news story I’ve just published with Nature – there is just so much to talk about here.

_________________________________________________________________

Our judgements of quality depend more on how a musician moves than what they sound like.

He’d whip his long hair around as he played, beads of sweat flying into the audience, and women would swoon or throw their clothes onto the stage. No, this isn’t the young Mick Jagger or Jimmy Page but Franz Liszt, sometimes dubbed the first rock star, of whose famously theatrical piano recitals Robert Schumann once said that, if he played behind a screen, “a great deal of poetry would be lost”.

But who cares about the histrionics – it’s the music that matters, right? Not according to a new study which shows that people’s judgements about the quality of a musical performance are influenced more by what they see than by what they hear [1].

These findings by social psychologist Chia-Jung Tsay of University College London, who is also an acclaimed classical pianist, may be embarrassing and even shocking to music lovers. The vast majority of participants in Tsay’s experiments – around 83 percent of both untrained participants and professional musicians – insisted at the outset that sound was their key criterion in assessing video and audio recordings of performances.

Yet it wasn’t. The participants were presented with recordings for the three finalists in each of 10 prestigious international competitions, and were asked to guess the winner. With just sound, or sound plus video, novices and experts both guessed right at about the same level as chance (33 percent of the time), or a little less. But with video alone, the success rate for both rose to about 46-53 percent. The experts did no better than the novices.

In experiments where participants were randomly assigned to receive the silent videos, Tsay says “they expressed much frustration and lack of confidence in their choices, not realizing that they were the ones to best approximate the original decisions.” She would receive comments like “It’s impossible to take seriously without sound” and “This is meaningless since I can't hear [the performer]”

This is a brilliant paper”, says philosopher of music Vincent Bergeron of the University of Ottawa in Canada. “The fact that the judgments of both novice and expert participants are affected in the very same way suggests that the visual channel constitute a powerful and robust factor in the evaluation of musical performances.”

Rethinking performance

The results raise provocative questions about what musical performance really is. Classical audiences in particular might like to claim that they are there to enjoy the exquisite sounds the performers are making – but it seems their assessments are based primarily on what they are seeing.

“As an academic I was delighted to find these counterintuitive results”, says Tsay. “As a classical musician, I was initially somewhat disturbed. It was surprising to find that there is such a wide gap between what we believe matters in the evaluation of music performance and what is actually being used to judge performances.”

But Bergeron isn’t perturbed. He has previously argued that visuals do play a part in how we experience music when we see it performed [2]. “One could plausibly argue that music made for performance, such as classical music, is a visual as well as a sonic art, and that it should also be evaluated on the basis of how it looks”, he says.

Bergeron’s earlier case built partly on the work of Jane Davidson on the University of Western Australia in Crawley, Australia, who also found that judgements of quality depend on sight as well as sound [3]. Music neuropsychologist Daniel Levitin of McGill University in Canada agrees that Tsay’s results might have been anticipated, both because of earlier work on the subject [4,5] and because of what we know about cognition in general.

“In a sense, the visual channel is more primordial than the auditory”, he says. Besides, “there are lots of ways in which our intuitions about our own cognition are wrong”, he says. “The whole field of perception and cognition is full of these, such as visual and auditory illusions.”

But Davidson says that there seem to be nuances in such judgements. For example, in her studies “musicians were still able to differentiate between poorer and better quality musical sound”, she says.

In Davison’s experience, the assessments of non-musicians may rely more on visuals than that of professional musicians. “In my own studies, musicians were able to use sound and vision independently”, she says. “It was only non-musicians who relied mainly on the visual information.”

She adds that some studies, including her own, that lay sound from one performance on top of visuals from another find that, although the visuals dominate perceptions, such tricks “don’t fool experts”.

What kind of messages do we take away from visual information? Tsay was able to rule out the possibility that a performer’s gender, race or attractiveness influence judgements, at least in her experiments, by tests in which she reduced the video data to black-and-white outlines of the performers. Participants still guessed the competition winners correctly with much the same better-than-chance success rate (48 percent) as before.

Tsay thinks that, at least for this kind of music, visual cues carry implications about the degree of passion and motivation that the performer displays. These are qualities that many participants cited as important in their evaluations, and even musical novices can identify them visually. Perhaps they do it even more accurately than the ‘experts’, Tsay says, because they are unencumbered by the sound “that professional musicians unintentionally and non-consciously discard.”

Looking good

One has also to wonder if musicians already unconsciously know that it matters what they look like in performance. Tsay suspects they do. “Many performers do have the intuition that the role of visual information is an important one”, she says. Moreover, having studied at top musical institutions such as the Juillard School in New York, she says “Some teachers at conservatories seem to be quite attuned to the important role that visual information plays in the judgment of music, and they make their students aware of its impact for effective performance.”

Davidson agrees that performers sense the significance of how they look. “Look at the artistry of Judy Garland”, she says. “Every move is integrated into a smooth action plan, as if it were created in the moment, yet it is totally rehearsed and polished as an integrated essential element of her vocal performance.”

“Really good musicians do this too,” she adds. “There are data from Glenn Gould’s career which shows how he moved very differently when only concerned with creating a sound recording rather than when in the recital room.”

The topic is probably an under-researched aspect of musical performance, however. (Musicologist Susan Fast of McMaster University in Hamilton, Canada, has provided a rare analysis of the visual body language cultivated by the rock group Led Zeppelin [6].) “I think the claims of the current paper need some really good social psychological contextualizations and clarifications”, says Davidson.

Whether or not musicians do learn to enhance their “visual merit”, the question is sure to arise: “should they?” Isn’t this a bit like cheating? Is the celebrated Chinese pianist Lang Lang (see above), for example, fooling audiences as he makes great sweeping gestures with his arms, eyes closed and head thrown back in ecstasy?

Perhaps the key question is whether the visual information is reliable – helping us to pick out the most deserving winner, say – or misleading, making us prefer performers who rely on visual flair rather than musical depth. “It is possible that some performances can be ‘trained’ or ‘choreographed’ in a way that may not be authentic or true to the meaning of the musical composition, but may still remain effective as a performance as judged by audiences”, Tsay admits.

“The video is a "bad" signal if it leads to bad outcomes, that is, if we reward musicians in competitions conducted this way and then find that those musicians fail to sustain creative careers”, says Levitin. “I don't know of any study that looks at these outcome measures.”

But one might also argue that a competition is seeking only to identify who is “best” on the day. In which case, what should “best” mean?

“I would say that it depends on one's ontology”, says Bergeron. “Someone who thinks that musical performances are essentially sonic events should recognize that our aesthetic evaluations of musical performances might be systematically mistaken. However, someone who is not prepared to accept that our aesthetic evaluations of musical performances might be systematically mistaken should recognize that musical performances might be visual as well as sonic events.”

In any event, says Bergeron, the fact is that ‘experts’ seem to be swayed by visuals whether they like it or not. “This might be a practical reason to embrace the idea that music made for performance is a visual as well as a sonic art, since it might be psychologically impossible to distinguish, in our experience of performances, those aesthetic qualities that belong to the sound from those that belong to the visual aspects.”

In the end, says Tsay, this comes down to a matter of priorities. “It may be less a question whether the visual channel gives us ‘good’ or ‘bad’ data, and more a question of what we as musicians and audience members believe truly reflects quality”, she says. “This likely changes with time and with changes in technologies and the consumption of music.” After all, in Liszt’s day live performance was the only way audiences would ever hear music. And with the cult of the “artist as expressive genius” firmly established since Beethoven’s day, it made sense for him to perform with flair. Evidently, it still does.

References
1. Tsay, C.-J. Proc. Natl Acad. Sci. USA doi: 10.1073/1221454110.
2. Bergeron, V. & Lopes, D. M. Philos. Phenomenol. Res. 78, 1 (2009).
3. Davidson, J. W. Psychol. Music 21, 103 (1993).
4. Vines, B. et al., Ann. N. Y. Acad. Sci. 1060, 462 (2005)
5. Vines, B. et al., Cognition 101, 80 (2006).
6. Fast, S. In the Houses of the Holy (Oxford University Press, Oxford, 2001).