research
interests
Artificial life, Neural networks
| Sound ecology
|
current position and project
Postdoctoral Research Fellow - Department of Music, University of SheffieldCommunication of emotion in Music and Speech: computational and psycho-physiological investigations
selected publications
Music Perception, Emotion and Cognition
Coutinho, E. & Cangelosi, A. (2010, in press). Musical emotions: predicting second-by-second subjective feelings of emotion from psycho-physiological measurements. Manuscript submitted for publication
Abstract
In this article we present a new methodology for replicating and predicting subjective feelings of emotion in response to music. We argue that music evokes emotion by creating dynamic temporal patterns to which our evolved socio-emotional brain is particularly sensitive, and we will show that the ways composers organize the acoustic building blocks of music, induce similar psycho-physiological responses in listeners. These claims will be supported by novel methodological investigations based on a combination of computational models and empirical psycho-physiological studies. We present evidence that the music psychoacoustic structure can account for a large proportion of the emotion reported by human listeners, by showing that a significant part of the listeners' affective response can be predicted from a set of six low level features of music: loudness, pitch level, pitch contour, tempo, texture and sharpness. We will also analyze how peripheral feedback in music can account for the predicted emotional responses, i.e., the role of physiological arousal in determining the intensity and valence of musical emotions. The work presented here provides a new methodology to the field of music and emotion research based on combinations of computational and experimental work, which aid the analysis of emotional responses to music, while offering a platform for the abstract representation of those complex relationships. Future developments may conduct to fundamental advances in different areas of research since they may provide coherent descriptions of the emotional effects of specific music stimuli, which can aid specific areas, such as, psychology and music therapy.Key words: Emotion, Arousal and Valence, Physiology, Psychoacoustics, Neural Networks
Coutinho, E. (2010, in press). Modeling psycho-physiological measurements of emotional responses to multiple music genres. To appear in: Proceedings of 11th International Conference of Music Perception and Cognition (ICMPC11), 23-27 August 2010.
Abstract
In previous work its was shown that spatiotemporal patterns of music structural features resonate with affective features underlying judgments of subjective feelings while listening to western classical music. In this article, that work is extend to multiple music genres, and re-evaluate the role of peripheral feedback to the prediction of self-reported emotions. Akin to our previous modeling studies, results show that low-level psychoacoustic features can be used to predict to a great extent the subjective emotional experience of a group of listeners. It will be shown that a computational model can explain 70% of the variance in arousal and 81% in valence using six psychoacoustic features - loudness, tempo, pitch level, melodic pitch, sharpness and texture – and one physiological variable - skin conductance response. This work contributes with new insights to the study of musical emotions, by showing that similar acoustic templates are shared across genres to convey emotional meaning.
Coutinho, E., & Cangelosi, A. (2010). A Neural Network Model for the Prediction of Musical Emotions. In S. Nefti & J.G. Gray (Ed.), Advances in Cognitive Systems (pp.331-368). London: IET Publisher.
Abstract
This chapter presents a novel methodology to analyse the dynamics of emotional responses to music in terms of computational representations of perceptual processes (psychoacoustic features) and self-perception of physiological activation (peripheral feedback). The approach consists of a computational investigation of musical emotions based on spatio-temporal neural networks sensitive to structural aspects of music. We present two computational studies based on connectionist network models that predict human subjective feelings of emotion. The first study uses six basic psychoacoustic dimensions extracted from the music pieces as predictors of the emotional response. The second computational study evaluates the additional contribution of physiological arousal to the subjective feeling of emotion. Both studies are backed up by experimental data. A detailed analysis of the simulation models’ results demonstrates that a significant part of the listener’s affective response can be predicted from a set of psychoacoustic features of sound - tempo, loudness, multiplicity (texture), power spectrum centroid (mean pitch), sharpness (timbre) and mean STFT flux (pitch variation) and one physiological cue - heart rate. This work provides a new methodology to the field of music and emotion research based on combinations of computational and experimental work, which aid the analysis of emotional responses to music, while offering a platform for the abstract representation of those complex relationships.
Coutinho, E (2010). Cognitive dissonance, knowledge instinct and musical emotions. Physics of Life Reviews, 7(1), 32-33. doi:10.1016/j.plrev.2009.12.005
Excerpt (initial paragraph)
My starting point for this commentary is Perlovsky’s suggestion that cognitive dissonance is a similar concept to "damaged" cognition due to differentiation, and a possible framework to test his hypotheses that the major role of musical emotions is to reconcile contradictions in consciousness, i.e., to restore “synthesis” (Perlovsky, in press). I will first analyze and address the commonalities between both phenomena. Then, I will briefly discuss the specialization of music as a referential for emotion, and propose an experimental framework to test the capacity for music to reduce cognitive dissonance and to promote the acquisition of differentiated knowledge. (...)
Coutinho, E. & Cangelosi, A. (2009). The use of spatio-temporal connectionist models in psychological studies of musical emotions. Music Perception, 27 (1), 1-15.
Abstract
This article presents a novel methodology to analyze the dynamics of emotional responses to music. It consists of a computational investigation based on spatiotemporal neural networks, which “mimic” human affective responses to music and predict the responses to novel music sequences. The results provide evidence suggesting that spatiotemporal patterns of sound resonate with affective features underlying judgments of subjective feelings (arousal and valence). A significant part of the listener’s affective response is predicted from a set of six psychoacoustic features of sound – loudness, tempo, texture, mean pitch, pitch variation, and sharpness. A detailed analysis of the network parameters and dynamics also allows us to identify the role of specific psychoacoustic variables (e.g., tempo and loudness) in music emotional appraisal. This work contributes new evidence and insights to the study of musical emotions, with particular relevance to the music perception and cognition research community.
Received February 25, 2008, accepted January 22, 2009.
Key words: emotion, music, arousal and valence, psychoacoustics, neural networks
Coutinho, E. (2008). Computational and Psycho-Physiological Investigations of Musical Emotions. Unpublished PhD dissertation. University of Plymouth (UK).
Abstract
The ability of music to stir human emotions is a well known fact (Gabrielsson & Lindstrom, 2001). However, the manner in which music contributes to those experiences remains obscured. One of the main reasons is the large number of syndromes that characterise emotional experiences. Another is their subjective nature: musical emotions can be affected by memories, individual preferences and attitudes, among other factors (Scherer & Zentner, 2001). But can the same music induce similar affective experiences in all listeners, somehow independently of acculturation or personal bias? A considerable cor pus of literature has consistently repor ted that listeners agree rather strongly about what type of emotion is expressed in a par ticular piece or even in par ticular moments or sections (Juslin & Sloboda, 2001). Those studies suggest that music features encode impor tant characteristics of affective experiences, by suggesting the influence of various structural factors of music on emotional expression. Unfor tunately, the nature of these relationships is complex, and it is common to find rather vague and contradictory descriptions.
This thesis presents a novel methodology to analyse the dynamics of emotional responses to music. It consists of a computational investigation, based on spatiotemporal neural networks sensitive to structural aspects of music, which “mimic” human affective responses to music and permit to predict new ones. The dynamics of emotional responses to music are investigated as computational representations of perceptual processes (psychoacoustic features) and self-perception of physiological activation (peripheral feedback). Modelling and experimental results provide evidence suggesting that spatiotemporal patterns of sound resonate with affective features underlying judgements of subjective feelings. A significant par t of the listener’s affective response is predicted from the a set of six psychoacoustic features of sound - tempo, loudness, multiplicity (texture), power spectrum centroid (mean pitch), shar pness (timbre) and mean STFT flux (pitch variation) - and one physiological variable - hear t rate. This work contributes to new evidence and insights to the study of musical emotions, with par ticular relevance to the music perception and emotion research communities.
Music, Speech and Emotion
Coutinho, E. & Dibben, N. (2010, in press). Music, Speech and Emotion: psycho-physiological and computational investigations. To appear in: Proceedings of CIM10 Nature versus Culture: International Conference on Interdisciplinary Musicology, 23-24 July 2010.
Abstract
Behavioural and neurological studies suggest that the affective content of music and speech is dependent on some of the same structural-auditory characteristics and mechanisms. For instance, music and speech prosody are involved in affiliative interactions between mothers and infants (Dissanayake, 2000), and the sensitivity to both stimuli may have similar developmental underpinnings (McMullen & Saffran, 2004). Moreover, musical behaviour and vocal communication also share common ancestry (Brown, 2000; Dissanayake, 2000) and are related by overlapping neural resources (Deutsch, Henthorn, & Dolson, 2004). Due to these similarities, some theories have been proposed to explain the origins and mechanisms of emotion expression and perception and, although still controversial, there is now converging evidence that the perception of emotion in auditory stimuli shares characteristics across domains (Juslin & Laukka, 2003).
Both speech prosody and music make use of acoustic signals to encode emotion-related information (Juslin & Laukka, 2003). Based on findings indicating that listeners’ affective responses to music can be accounted for at least in part by “basic” acoustic cues (Coutinho & Cangelosi, 2009), and physiological variables (Coutinho & Cangelosi, in press), we study and model the perception of emotion in music and speech. Our aim is to investigate whether the perception of emotion is related to similar acoustic patterns in both music and speech stimuli.
The method involves asking participants to listen to a set of music pieces and speech excerpts, while reporting continuously their emotional experience using a computer framework. At the same time we measure participants’ physiological reactions during listening, namely heart rate, skin conductance, respiration and blood volume pressure. The data collected is used to analyze the participants’ psychological and physiological reactions to the stimuli and to compare them with the psychoacoustic properties of the music and speech.
This research will determine the acoustic similarity between affective speech prosody and music. This will provide insight into the cross-cultural universality of some aspects of affective prosody and music and shed light on the extent to which they share mechanisms involved in evoking emotional response. The identification of shared characteristics of emotion expression in music and speech prosody may contribute to evolutionary perspectives on music and language. This is the first time that an attempt has been made to study the relationship between expression of emotion in music and speech prosody in longer excerpts of music and speech.
References
Brown, S. (2000). Are Music and Language Homologues? In N. Wallin, B. Merker, & S. Brown, The origins of music (pp. 271-300). Cambridge USA: MIT Press.
Coutinho, E., & Cangelosi, A. (2009). The use of spatio-temporal connectionist models in psychological studies of musical emotions. Music Perception, 27 (1), pp. 1-15.
Coutinho, E., & Cangelosi, A. (in press). Music and Core Affect: predicting second-by-second subjective feelings of emotion from psycho-physiological measurements.
Deutsch, D., Henthorn, T., & Dolson, M. (2004). Absolute Pitch, Speech, and Tone Language: Some Experiments and a Proposed Framework. Music Perception , 21 (3), 339-356.
Dissanayake, E. (2000). Antecedents of the temporal arts in early mother-infant interaction . In N. L. Wallin, B. Merker, & S. Brown, The origins of music (pp. 389–410). Cambridge, USA: MIT Press.
Juslin, P., & Laukka, P. (2003). Communication of emotions in vocal expression and music performance: Different channels, same code? Psychological Bulletin, 129 (5), pp. 770-814.
McMullen, E., & Saffran, J. R. (2004). Music and Language: A Developmental Comparison. Music Perception , 21, 289-311.
Alife Models
Coutinho E. & Cangelosi A. (2007). Emotion and Embodiment in Cognitive Agents: from Instincts to Music. In H. Hexmoor & C. Thompson (Eds.), Proceedings of 2007 International Conference on Integration of Knowledge Intensive Multi-Agent Systems (KIMAS’07). Waltham (MA, USA): IEEE Press, pp. 133-138.
Abstract
Due to a progressive change in theoretical approaches to cognition based on embodiment theories ([1]), new models of cognition, attention and behavior frequently include Emotions as a mechanism of integration. The idea that emotional cues act as states that might influence behavior and adaptation has recently gained special attention in computational models of cognition and behavior (e.g. [2], [3], [4], [5]). Whilst some of these models focus on different properties of an emotional system for task solving issues (e.g. using facial expressions for social engagement), we are interested in using computational models to understand the basic mechanisms of the emotional systems and their interaction with other processes. Our aim is to investigate how these systems evolved and interact, which mechanisms do they rely on, and especially what is the role of the embodiment.
We are developing computational models of simulated autonomous agents that use emotion as a mechanism for organization of behavior. This way we intend to create an integrated model of instincts, perception, motivation and action, based on artificial environments and embodiments. We suggest that the agent should be embodied so as to allow its behavior to be affected by motivational processes, focusing on the internal demands and activity. By artificial embodiment we mean that the agent has a virtual physical body whose states can be sensed by the agent itself. We will discuss a theoretical framework and specific scenarios to test our hypothesis. We present results for one test framework on the emergence of motivational processes in embodied agents. These simulations show the emergence of a stable motivational system with emotional contexts resulting from dynamical categorization of objects in the world, in answer to survival pressures. The preliminary framework seems to be effective and versatile enough to allow the agent to adapt itself to unknown world configurations, maintaining controlled “healthy” states. The results are coherent with Antonio Damasio’s theory of background emotional system [6]. We demonstrate that body/world categorizations and body maps can evolve from the simple self-survival rule.
Future work aims to further develop the model, for example integrating communication processes in the agents and allowing for social interaction with the other agents towards cooperative tasks. This way we would extend the simulations to the role of emotion under communicative pressures. We are also studying the extension of this model to musical emotions [7]. A potential field of application of these models arises from recent theoretical proposal in robotics, more specifically that of Internal Robotics [8].
References
[1] R. Picard, E. Vyzas, and J. Healey, “Toward machine emotional intelligence: Analysis of affective physiological state,” IEEE Transactions Pattern Analysis and Machine Intelligence, vol. 23, p. 11751191, 2001.
[2] J. D. Velasquez, “Modeling emotion-based decision-making,” in Proceeding of 1998 AAAI Fall Symposium Emotional and Intelligent: The Tangled Knot of Cognition (Technical Report FS-98-03). Orlando, FL: AAAI Press, 1998, pp. 164–169.
[3] D. Canamero, “A hormonal model of emotions for behavior control,” in 4th European Conference on Artificial Life ECAL’97, 1997.
[4] C. Breazeal, “Emotions and sociable humanoid robots,” International Journal Human-Computer Studies, vol. 59, pp. 119–155, 2003.
[5] S. C. Gadanho and J. Hallam, “Robot learning driven by emotions”. Adaptive Behavior, vol. 9, pp. 42–64, 2001.
[6] A. Damasio, The Feeling of What Happens: Body, Emotion and the Making of Consciousness. Vintage, 2000.
[7] E. Coutinho, A. Cangelosi. “The dynamics of music perception and emotional experience: a connectionist model.”, Proceedings of ICMPC’06, 2006.
[8] D. Parisi, “Internal robotics,” Connection Science, vol. 16, no. 4, pp. 325–338, December 2004.
Full article
.pdf
Coutinho, E., Miranda, E. & Silva, P. (2005). Evolving emotional behaviour for expressive performance of music.In T. Panayiotopoulos, J. Gratch, R. Aylett, D. Ballin, P. Olivier, & T. Rist (Ed.), , Lecture Notes in Artificial Intelligence: Intelligent Virtual Agents: 5th International Working Conference IVA 2005, 3661, 147. Berlin: Springer-Verlag (LNAI 3661), pp. 147-147.
Abstract
Today computers can be programmed to compose music automatically, using tech-niques ranging from rule-based to evolutionary computation (e.g., genetic algorithms and cellular automata). However, we lack good techniques for programming the com-puter to play or interpret music with expression. Expression in music is largely asso-ciated with emotions. Therefore we are looking into the possibility of programming computer music systems with emotions. We are addressing this problem from an A-Life perspective combined with recent discoveries in the neurosciences with respect to emotion. Antonio Damasio refers to the importance of emotions to assist an individual to maintain survival, as they seem to be an important mechanism for adaptation and decision-making. Specifically, environmental events of value should be susceptible to preferential perceptual processing, regarding their pleasant or unpleasant. This ap-proach assumes the existence of neural pathways that facilitate survival. Stable emo-tional systems should then emerge from self-regulatory homeostatic processes. We implemented a system consisting of an agent that inhabits an environment con-taining with a number of different objects. These objects cause different physiological reactions to the agent. The internal body state of the agent is defined by a set of inter-nal drives and a set of physiological variables that vary as the agent interacts with the objects it encounters in the environment. The agent is controlled by a feed-forward neural network that integrates visual input with information about its internal states. The network learns through a reinforcement-learning algorithm, derivate from differ-ent body states, due to pleasant or unpleasant stimuli. The playback of musical recordings in MIDI format is steered by the physiological variables of the agent in different phases of the adaptation process. The behaviour of the system is coherent with Damasio's theory of background emotional system. It demonstrates that specific phenomena, such as body/world cate-gorization and existence of a body map, can evolve from a simple rule: self-survival in the environment. Currently, we are in the process of defining a system of higher-level emotional states (or foreground system) that will operate in social contexts; i.e., with several agents in the environment reacting to objects and interacting with each other.
Poster presentation
.pdf
Coutinho, E., Miranda, E. & Cangelosi, A. (2005). Towards a Model for Embodied Emotions. In C. Bento, A. Cardoso & G. Dias (Ed.), Proceedings of the Portuguese Conference on Artificial Intelligence EPIA2005 (pp. 54-63). Covilha: UBI/IEEE Press.
Abstract
We are interested in developing A-Life-like models to study the evolution of emotional systems in artificial worlds inhabited by autonomous agents. This paper focuses on the emotional component of an agent at its very basic physical level. We adopt an evolutionary perspective by modelling the agent based on biologically plausible principles, whereby Emotions emerge from homeostatic mechanisms. We suggest that the agent should be embodied so as to allow its behaviour to be affected by low-level physical tasks. By embodiment we mean that the agent has a virtual physical body whose states can be sensed by the agent itself. The simulations show the emergence of a stable emotional system with emotional contexts resulting from dynamical categorization of objects in the world. This proved to be effective and versatile enough to allow the agent to adapt itself to unknown world configurations. The results are coherent with Antonio Damasio's theory of background emotional system (Damasio, 2000). We demonstrate that body/world categorizations and body maps canevolve from a simple rule: self-survival.
References
Damasio, A. (2000). The Feeling of What Happens: Body, Emotion and the Making of Consciousness. London, UK: Vintage.
Full article
.pdf
Coutinho, E. (2003). Ecological Simulator Of Life. Unpublished MsC dissertation,University of Porto (Portugal). (MsC dissertation)