Designing the user experience of musical sonification in public and semi-public spaces
DOI:
https://doi.org/10.7146/se.v10i1.124202Abstract
Sonification refers to sonic expression of data or information. It is often thought of as an auditory complement, providing additional information about data which can reveal patterns and facilitate interpretation and understanding of the data. Hence, the listening space created by a sonifi cation is always a hybrid where auditory augmentation complements other information modalities and, in some cases, also spatial qualities. In this work, we focus on sonifi cation in public and semi-public spaces, and specifi cally on musical sonifi cation – the use of musical sounds to create a sonic environment, augmenting or complementing a physical shared space. We draw upon established approaches in interaction design to focus our work on the user experience of musical sonifi cation in public and semi-public spaces. Specifi cally, we fi rst identify the experiential qualities of sonic atmosphere and performativity as important aspects of sonifi cation in public and semi-public spaces, then use those experiential qualities generatively in the speculative design of a musical sonifi cation sketch. The design sketch comprises a dynamic musical sonifi cation of air quality data, intending to give citizens an awareness and an enhanced individual and interpersonal understanding of air quality in their city.
References
Argstatter, H. (2016). Perception of basic emotions in music: Culture-specifi c or multicultural? Psychology of Music, 44(4), 674-690. doi.org/10.1177/0305735615589214
Bañagale, R. (2014). Arranging Gershwin: Rhapsody in blue and the creation of an American icon. Oxford University Press. doi.org/10.1093/acprof:oso/9780199978373.001.0001
Bardzell, J., Bolter, J., & Löwgren, J. (2010). Interaction criticism: Three readings of an interaction design, and what they get us. Interactions, 2, vol. 17. doi.org/10.1145/1699775.1699783
Blandino, M.V. (2018). Toxsampler: Locative sound art exploration of the toxic release inventory. In: proceedings 24th International Conference on Auditory Display (ICAD2018). Georgia Institute of Technology. doi.org/10.21785/icad2018.018
Blumstein, D.T., Bryant, G.A., & Kaye, P. (2012). The sound of arousal in music is context-dependent. Biology letters, 8(5), 744-747. doi.org/10.1098/rsbl.2012.0374
Bronkhorst, A.W. (2000). The cocktail party phenomenon: A re-view of research on speech intelligibility in multiple-talker conditions. Acta Acustica united with Acustica, 86(1), 117-128.. doi.org/10.3758/s13414-015-0882-9
Delíege, I., & Sloboda, J. (1997). Perception and Cognition of Music. Hove, East Sussex: Psychology Press Ltd. doi.org/10.1093/mtp/16.2.91
Durrant, A., Trujillo-Pisanty, D., Moncur, W., & Orzech, K. (2015). Charting the digital lifespan: Picture Book. University of Newcastle.
Getzmann, S., & Näätänen, R. (2015). The mismatch negativity as a measure of auditory stream segregation in a simulated “cocktail-party” scenario: effect of age. Neurobiology of aging, 36(11), 3029-3037. doi.org/10.1016/j.neurobiolaging.2015.07.017
Gorbman, C. (1987). Unheard melodies: narrative fi lm music. London: Bloomington: BFI; Indiana University Press.
Hermann, T., Hunt, A., & Neuhoff, J.G. (2011). The Sonifi cation Handbook. Berlin, Germany: Logos Publishing House.
Hunter, P.G., Schellenberg, E.G., & Schimmack, U. (2010). Feelings and perceptions of happiness and sadness induced by music: Similarities, differences, and mixed emotions. Psychology of Aesthetics, Creativity, and the Arts, 4(1), 47. doi.org/10.1037/a0016873
Iakovides, S.A., Iliadou, V.T.H., Bizeli, V.T.H., Kaprinis, S.G., Fountoulakis, K.N., & Kaprinis, G.S. (2004). Psychophysiology and psychoacoustics of music: Perception of complex sound in normal subjects and psychiatric patients. Annals of General Hospital Psychiatry, 3, 1–4. doi.org/10.1186/1475- 2832-3-6
Juslin, P.N., & Laukka, P. (2004). Expression, perception, and induction of musical emotions: A review and a questionnaire study of everyday listening. Journal of New Music Research, 33, 217–238. doi.org/10.1080/0929821042000317813
Koelsch, S. (2009). Neural substrates of processing syntax and semantics in music. Music that works (pp. 143-153). Springer. doi.org/10.1007/978-3-211-75121-3_9
Levitin, D.J. (2006). This is your brain on music: The science of a human obsession. New York, US: Dutton; Penguin Books.
Liu, Y., Liu, G., Wei, D., Li, Q., Yuan, G., Wu, S., Wang, G., & Zhao, X. (2018). Effects of musical tempo on musicians’ and non-musicians’ emotional experience when listening to music. Frontiers in Psychology, 9, 2118. doi.org/10.3389/fpsyg.2018.02118
Löwgren, J. (2009). Toward an articulation of interaction esthetics. New Review of Hypermedia and Multimedia, 15(2), 129-146. doi.org/10.1080/13614560903117822
Meyer, L.B. (2008). Emotion and meaning in music. University of Chicago Press.
Morrison, S.J., & Demorest, S. M. (2009). Cultural constraints on music perception and cognition. Progress in brain research, 178, 67-77. doi.org/10.1016/S0079-6123(09)17805-6
Pallesen, K.J., Brattico, E., Bailey, C., Korvenoja, A., Koivisto, J., Gjedde, A., & Carlson, S. (2005). Emotion processing of major, minor, and dissonant chords: A functional magnetic resonance imaging study. Annals New York Academy of Sciences, 1060, 450-453. doi.org/10.1196/annals.1360.047
St Pierre, M., & Droumeva, M. (2016). Sonifying for public engagement: A context-based model for sonifying air pollution data. In: Proceedings 22nd International Conference on Auditory Display (ICAD2016). International Community on Auditory Display. doi.org/10.21785/icad2016.033
Pinch, T., & Bijsterveld, K. (2012). The Oxford Handbook of Sound Studies. Oxford, UK: Oxford University Press. doi.org/10.1093/oxfordhb/9780195388947.001.0001
Polli, A. (2014). Toxic airs: Body, place, planet in historical perspective. A. Johnson, & J.R. Fleming, editors. Pittsburgh. doi.org/10.2307/j.ctt5vkgsj
Prendergast, R.M. (1992). Film Music: A Neglected Art. W.W. Norton.
Russell, J.A. (1980). A circumplex model of affect. Journal of personality and social psychology, 39(6), 1161. doi.org/10.1037/h0077714
Rönnberg, N., & Löwgren, J. (2016). The sound challenge to visualization design research. In: Proceedings EmoVis2016, ACMIUI2016 Workshop on Emotion and Visualization. Linköping Electronic Conference Proceedings, vol. 103, pp. 31–34. doi.org/10.3384/ecp10305
Schafer, R.M. (1993). The soundscape: Our sonic environment and the tuning of the world. Simon and Schuster.
Schubert, E. (2004). Modeling perceived emotion with continuous musical features. Music perception, 21(4), 561-585. doi.org/10.1525/mp.2004.21.4.561
Seashore, C.E. (1967). Psychology of Music. New York, US: Dover.
Supper, A. (2014). Sublime frequencies: The construction of sublime listening experiences in the sonification of scientific data. Social Studies of Science, 44(1), 34-58. doi.org/10.1177/0306312713496875
Ståhl, A., Höök, K., & Löwgren, J. (2014). Evocative balance: Designing for interactional empowerment. International Journal of Design, 1(8).
Thibaud, J.P. (2002). From situated perception to urban ambiences.
Tittel, C. (2009). Sound art as sonifi cation, and the artistic treatment of features in our surroundings. Organised Sound, 14(1), 57-64. doi.org/10.1017/S1355771809000089
Tsuchiya, T., Freeman, J., & Lerner, L.W. (2015). Data-to-music API: Real-time data-agnostic sonification with musical structure models. In: Proceedings 21st International Conference on Auditory Display (ICAD2015). Graz, Styria, Austria: Georgia Institute of Technology, pp. 244–251.
Vickers, P. (2016). Sonifi cation and music, music and sonifi cation. In: The Routledge Companion to Sounding Art (pp. 135-144). Taylor & Francis. doi.org/10.4324/9781315770567
Vuust, P., & Kringelbach, M.L. (2010). The pleasure of making sense of music. Interdisciplinary science reviews, 35(2), 166-182. doi.org/10.1179/030801810X12723585301192
Webster, G.D., & Weir, C.G. (2005). Emotional responses to music: Interactive effects of mode, texture, and tempo. Motivation and Emotion, 29(1), 19-39. doi.org/10.1007/s11031-005-4414-0
Wong, P.C., Ciocca, V., Chan, A.H., Ha, L.Y., Tan, L.H., & Peretz, I. (2012). Effects of culture on musical pitch perception. PLoS one, 7(4), e33424. doi.org/10.1371/journal.pone.0033424
Wood, N., & Cowan, N. (1995). The cocktail party phenomenon re-visited: how frequent are attention shifts to one’s name in an irrelevant auditory channel? Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(1), 255. doi.org/10.1037//0278-7393.21.1.255
Zentner, M. R., & Kagan, J. (1998). Infants’ perception of consonance and dissonance in music. Infant Behavior and Development, 21(3), 483-492. doi.org/10.1016/S0163-6383(98)90021-2
Bañagale, R. (2014). Arranging Gershwin: Rhapsody in blue and the creation of an American icon. Oxford University Press. doi.org/10.1093/acprof:oso/9780199978373.001.0001
Bardzell, J., Bolter, J., & Löwgren, J. (2010). Interaction criticism: Three readings of an interaction design, and what they get us. Interactions, 2, vol. 17. doi.org/10.1145/1699775.1699783
Blandino, M.V. (2018). Toxsampler: Locative sound art exploration of the toxic release inventory. In: proceedings 24th International Conference on Auditory Display (ICAD2018). Georgia Institute of Technology. doi.org/10.21785/icad2018.018
Blumstein, D.T., Bryant, G.A., & Kaye, P. (2012). The sound of arousal in music is context-dependent. Biology letters, 8(5), 744-747. doi.org/10.1098/rsbl.2012.0374
Bronkhorst, A.W. (2000). The cocktail party phenomenon: A re-view of research on speech intelligibility in multiple-talker conditions. Acta Acustica united with Acustica, 86(1), 117-128.. doi.org/10.3758/s13414-015-0882-9
Delíege, I., & Sloboda, J. (1997). Perception and Cognition of Music. Hove, East Sussex: Psychology Press Ltd. doi.org/10.1093/mtp/16.2.91
Durrant, A., Trujillo-Pisanty, D., Moncur, W., & Orzech, K. (2015). Charting the digital lifespan: Picture Book. University of Newcastle.
Getzmann, S., & Näätänen, R. (2015). The mismatch negativity as a measure of auditory stream segregation in a simulated “cocktail-party” scenario: effect of age. Neurobiology of aging, 36(11), 3029-3037. doi.org/10.1016/j.neurobiolaging.2015.07.017
Gorbman, C. (1987). Unheard melodies: narrative fi lm music. London: Bloomington: BFI; Indiana University Press.
Hermann, T., Hunt, A., & Neuhoff, J.G. (2011). The Sonifi cation Handbook. Berlin, Germany: Logos Publishing House.
Hunter, P.G., Schellenberg, E.G., & Schimmack, U. (2010). Feelings and perceptions of happiness and sadness induced by music: Similarities, differences, and mixed emotions. Psychology of Aesthetics, Creativity, and the Arts, 4(1), 47. doi.org/10.1037/a0016873
Iakovides, S.A., Iliadou, V.T.H., Bizeli, V.T.H., Kaprinis, S.G., Fountoulakis, K.N., & Kaprinis, G.S. (2004). Psychophysiology and psychoacoustics of music: Perception of complex sound in normal subjects and psychiatric patients. Annals of General Hospital Psychiatry, 3, 1–4. doi.org/10.1186/1475- 2832-3-6
Juslin, P.N., & Laukka, P. (2004). Expression, perception, and induction of musical emotions: A review and a questionnaire study of everyday listening. Journal of New Music Research, 33, 217–238. doi.org/10.1080/0929821042000317813
Koelsch, S. (2009). Neural substrates of processing syntax and semantics in music. Music that works (pp. 143-153). Springer. doi.org/10.1007/978-3-211-75121-3_9
Levitin, D.J. (2006). This is your brain on music: The science of a human obsession. New York, US: Dutton; Penguin Books.
Liu, Y., Liu, G., Wei, D., Li, Q., Yuan, G., Wu, S., Wang, G., & Zhao, X. (2018). Effects of musical tempo on musicians’ and non-musicians’ emotional experience when listening to music. Frontiers in Psychology, 9, 2118. doi.org/10.3389/fpsyg.2018.02118
Löwgren, J. (2009). Toward an articulation of interaction esthetics. New Review of Hypermedia and Multimedia, 15(2), 129-146. doi.org/10.1080/13614560903117822
Meyer, L.B. (2008). Emotion and meaning in music. University of Chicago Press.
Morrison, S.J., & Demorest, S. M. (2009). Cultural constraints on music perception and cognition. Progress in brain research, 178, 67-77. doi.org/10.1016/S0079-6123(09)17805-6
Pallesen, K.J., Brattico, E., Bailey, C., Korvenoja, A., Koivisto, J., Gjedde, A., & Carlson, S. (2005). Emotion processing of major, minor, and dissonant chords: A functional magnetic resonance imaging study. Annals New York Academy of Sciences, 1060, 450-453. doi.org/10.1196/annals.1360.047
St Pierre, M., & Droumeva, M. (2016). Sonifying for public engagement: A context-based model for sonifying air pollution data. In: Proceedings 22nd International Conference on Auditory Display (ICAD2016). International Community on Auditory Display. doi.org/10.21785/icad2016.033
Pinch, T., & Bijsterveld, K. (2012). The Oxford Handbook of Sound Studies. Oxford, UK: Oxford University Press. doi.org/10.1093/oxfordhb/9780195388947.001.0001
Polli, A. (2014). Toxic airs: Body, place, planet in historical perspective. A. Johnson, & J.R. Fleming, editors. Pittsburgh. doi.org/10.2307/j.ctt5vkgsj
Prendergast, R.M. (1992). Film Music: A Neglected Art. W.W. Norton.
Russell, J.A. (1980). A circumplex model of affect. Journal of personality and social psychology, 39(6), 1161. doi.org/10.1037/h0077714
Rönnberg, N., & Löwgren, J. (2016). The sound challenge to visualization design research. In: Proceedings EmoVis2016, ACMIUI2016 Workshop on Emotion and Visualization. Linköping Electronic Conference Proceedings, vol. 103, pp. 31–34. doi.org/10.3384/ecp10305
Schafer, R.M. (1993). The soundscape: Our sonic environment and the tuning of the world. Simon and Schuster.
Schubert, E. (2004). Modeling perceived emotion with continuous musical features. Music perception, 21(4), 561-585. doi.org/10.1525/mp.2004.21.4.561
Seashore, C.E. (1967). Psychology of Music. New York, US: Dover.
Supper, A. (2014). Sublime frequencies: The construction of sublime listening experiences in the sonification of scientific data. Social Studies of Science, 44(1), 34-58. doi.org/10.1177/0306312713496875
Ståhl, A., Höök, K., & Löwgren, J. (2014). Evocative balance: Designing for interactional empowerment. International Journal of Design, 1(8).
Thibaud, J.P. (2002). From situated perception to urban ambiences.
Tittel, C. (2009). Sound art as sonifi cation, and the artistic treatment of features in our surroundings. Organised Sound, 14(1), 57-64. doi.org/10.1017/S1355771809000089
Tsuchiya, T., Freeman, J., & Lerner, L.W. (2015). Data-to-music API: Real-time data-agnostic sonification with musical structure models. In: Proceedings 21st International Conference on Auditory Display (ICAD2015). Graz, Styria, Austria: Georgia Institute of Technology, pp. 244–251.
Vickers, P. (2016). Sonifi cation and music, music and sonifi cation. In: The Routledge Companion to Sounding Art (pp. 135-144). Taylor & Francis. doi.org/10.4324/9781315770567
Vuust, P., & Kringelbach, M.L. (2010). The pleasure of making sense of music. Interdisciplinary science reviews, 35(2), 166-182. doi.org/10.1179/030801810X12723585301192
Webster, G.D., & Weir, C.G. (2005). Emotional responses to music: Interactive effects of mode, texture, and tempo. Motivation and Emotion, 29(1), 19-39. doi.org/10.1007/s11031-005-4414-0
Wong, P.C., Ciocca, V., Chan, A.H., Ha, L.Y., Tan, L.H., & Peretz, I. (2012). Effects of culture on musical pitch perception. PLoS one, 7(4), e33424. doi.org/10.1371/journal.pone.0033424
Wood, N., & Cowan, N. (1995). The cocktail party phenomenon re-visited: how frequent are attention shifts to one’s name in an irrelevant auditory channel? Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(1), 255. doi.org/10.1037//0278-7393.21.1.255
Zentner, M. R., & Kagan, J. (1998). Infants’ perception of consonance and dissonance in music. Infant Behavior and Development, 21(3), 483-492. doi.org/10.1016/S0163-6383(98)90021-2
Downloads
Published
2021-01-15
How to Cite
Rönnberg, N., & Löwgren, J. (2021). Designing the user experience of musical sonification in public and semi-public spaces. SoundEffects - An Interdisciplinary Journal of Sound and Sound Experience, 10(1), 125–141. https://doi.org/10.7146/se.v10i1.124202
Issue
Section
Articles
License
The journal allow the author(s) to hold the copyright without restrictions. The journal allows the author(s) to retain publishing rights without restrictions.
