The psychology of music in multimedia

For most of the history of film-making, music has played an integral role serving many functions - such as conveying emotion, heightening tension, and influencing interpretation and inferences about events and characters. More recently, with the enormous growth of the gaming industry and the Internet, a new role for music has emerged. However, all of these applications of music depend on complex mental processes which are being identified through research on human participants in multimedia contexts. The Psychology of Music in Multimedia is the first book dedicated to this fascinating topic.

The Psychology of Music in Multimedia presents a wide range of scientific research on the psychological processes involved in the integration of sound and image when engaging with film, television, video, interactive games, and computer interfaces. Collectively, the rich chapters in this edited volume represent a comprehensive treatment of the existing research on the multimedia experience, with the aim of disseminating the current knowledge base and inspiring future scholarship. The focus on empirical research and the strong psychological framework makes a unique and distinct contribution to the field. The international collection of contributors represents eight countries and a broad range of disciplines including psychology, musicology, neuroscience, media studies, film, and communications. Each chapter includes a comprehensive review of the topic and, where appropriate, identifies models that can be empirically tested.

Siu-Lan Tan, Department of Psychology Kalamazoo College, USA,Annabel Cohen, Department of Psychology University of Prince, USA,Scott D. Lipscomb, School of Music, University of Minnesota, USA,Roger A. Kendall, University of California, Los Angeles, USA

Siu-Lan Tan is Associate Professor of Psychology at Kalamazoo College in Michigan USA. Born in Indonesia and raised in Hong Kong, she holds diplomas from the Associated Board of the Royal Schools of Music (England) and degrees in Music and Piano Pedagogy, and taught music in Hong Kong and California for many years before completing a PhD in Psychology at Georgetown University USA and a term on scholarship at Oxford University in England. Tan's research on video games and virtual reality games has been published in the Journal of Applied Developmental Psychology, International Journal of Gaming and Computer-Mediated Simulations, Interacting with Video (Praeger 1996), and Interdisciplinary Advancements in Gaming, Simulations and Virtual Environments (IGI 2012). Her research on film music and other topics has appeared in Music Perception, Psychology of Music, Psychomusicology: Music, Mind, & Brain, and College Music Symposium.

Annabel J. Cohen is Professor of Psychology at the University of Prince Edward Island in Canada. Her doctorate is from Queen's University, B.A from McGill, and Associate diploma (ARCT) from the Royal Conservatory of Music, Toronto. She has dedicated her career to the study of music perception and cognition, with extensions to multimedia. She has published on the topics as tonality, music transposition, the acquisition of music grammar, and film music, from a cognitive perspective. She has led multi-institutional research projects focusing on harnessing multimedia for education in the context of culture and cognition. On that foundation, she currently directs a major international collaborative research initiative (AIRS - Advancing Interdisciplinary Research in Singing). Cohen is the Editor of Psychomusicology: Music, Mind, & Brain and serves on the consulting boards of several journals (e.g., Music and the Moving Image, Musicae Scientiae, Music Perception, Psychology of Music, The Soundtrack).

Scott D. Lipscomb is Associate Professor and Division Head of Music Education and Music Therapy and Associate Director for the School of Music at the University of Minnesota - Twin Cities, USA. In addition to his primary research interest in "sound for multimedia," he is currently pursuing investigations related to surround sound presentation of cinema and musical sound, the effect of music in video game contexts, the impact of visual performance information on listener preference for "complex" music, integration of music across the K-12 curriculum, and the development of interactive instructional media to enhance the music learning experience. Lipscomb served two three-year terms as Treasurer for the Society for Music Perception and Cognition, four two-year terms as President of the Association for Technology in Music Instruction (ATMI), and serves as a member of the Executive Board and as Chair of the Research Committee for TI:ME (Technology Institute for Music Educators).

Roger A. Kendall is a Professor in the Department of Ethnomusicology at the University of California, Los Angeles, USA in the specialization of Systematic Musicology. He co-authored an invited chapter on music perception and cognition for the Ecological Psychology volume of the Handbook on Perception, as well as a chapter on comparative music psychology for Psychology of Music (D. Deutsch, Ed. 2nd edition). His current research interests include comparative perceptual and acoustical analyses of natural versus synthetic and sampled orchestral timbres and spectra, tuning models and perception of the slendro mode in the Gamelan, expressive music performance modelled in terms of communication theory, and perception of meaning in film music. Kendall co-edited Perspectives in Systematic Musicology in Selected Reports in Ethnomusicology, Volume 12 and contributed a chapter outlining connections of experimental empirical research in meaning to visual and musical elements.

Foreword, Nicholas Cook
1. Introduction: The psychology of music in multimedia, Annabel J. Cohen, Scott D. Lipscomb, Siu-Lan Tan, & Roger A. Kendall
Part I. Models and Multidisciplinary Perspectives
2. Congruence-Association Model of music and multimedia: Origin and evolution, Annabel J. Cohen
3. Experimental semiotics applied to visual, sound, and musical structures, Roger A. Kendall and Scott D. Lipscomb
4. Integrating media effects research and music psychology, Mark Shevy
5. Musical analysis for multimedia: A view from music theory, David Bashwiner
6. Music and emotion in narrative films: A neuroscientific perspective, Lars Kuchinke, Hermann Kappelhoff, & Stefan Koelsch
Part II. Cross-modal Relations in Multimedia
7. Perceived congruence between auditory and visual elements in multimedia, Shin-ichiro Iwamiya
8. How pitch and loudness shape musical space and motion, Zohar Eitan
9. Cross-modal alignment of accent structures in multimedia, Scott D. Lipscomb
Part III. Interpretation and Meaning
10. Music videos and visual influences on music perception and appreciation: Should you want your MTV?, Marilyn G. Boltz
11. Music and memory in film and other multimedia: The Casablanca effect, Berthold Hoeckner and Howard C. Nusbaum
Part IV. Applications: Music and Sound in Multimedia
12. Children's media: The role of music and audio features, Sandra L. Calvert
13. Playing with sound: The role of music and sound effects in gaming, Mark Grimshaw, Siu-Lan Tan, & Scott D. Lipscomb
14. Music in television advertising and other persuasive media, Mark Shevy & Kineta Hung
15. Auditory icons, earcons, and displays: Information and expression through sound, Agnieszka Roginska
16. Understanding the impact of surround sound in multimedia, Mark Kerins
Part V. Future Research Directions
17. Future research directions for music and sound in multimedia, Siu-Lan Tan, Annabel J. Cohen, Scott D. Lipscomb, & Roger A. Kendall