current research interests

Computational models of emotions;
Emotional communication in music and speech;
Artificial Life and Evolutionary Computation;
Multi-agent systems;
Emotion and cognition;

other research interests

Human-computer interaction;
Algorithmic composition and sound design;
Computer music systems;
Sound ecology.


current appointment and project

Honorary Research Fellow - School of Music, University of Liverpool
Communication of emotion in Music and Speech: computational and psycho-physiological investigations


publications


Computational musicology: An artificial life approach
E. Coutinho, M. Gimenes, J.M. Martins & E.R. Miranda
Proceedings of the Portuguese Conference on Artificial Intelligence (EPIA'05)


Abstract
Artificial Life (A-Life) and Evolutionary Algorithms (EA) provide a variety of new techniques for making and studying music. EA have been used in different musical applications, ranging from new systems for composition and performance, to models for studying musical evolution in artificial societies. This paper starts with a brief introduction to three main fields of application of EA in Music, namely sound design, creativity and computational musicology. Then it presents our work in the field of computational musicology. Computational musicology is broadly defined as the study of Music with computational modelling and simulation. We are interested in developing A-Life-based models to study the evolution of musical cognition in an artificial society of agents. In this paper we present the main components of a model that we are developing to study the evolution of musical ontogenies, focusing on the evolution of rhythms and emotional systems. The paper concludes by suggesting that A-Life and EA provide a powerful paradigm for computational musicology.

Citation (APA style)
Coutinho, E., Gimenes, M., Martins, J.M., & Miranda, E.R. (2005). Computational musicology: An artificial life approach. In C. Bento, A. Cardoso, & G. Dias (Eds.), Proceedings of the Portuguese Conference on Artificial Intelligence (EPIA'05) (pp. 85-93). Covilha: IEEE Press. DOI: 0.1109/EPIA.2005.341270.

DOI
0.1109/EPIA.2005.341270

Towards a Model for Embodied Emotions
E. Coutinho, E.R. Miranda, & A. Cangelosi
Proceedings of the Portuguese Conference on Artificial Intelligence (EPIA'05)


Abstract

We are interested in developing A-Life-like models to study the evolution of emotional systems in artificial worlds inhabited by autonomous agents. This paper focuses on the emotional component of an agent at its very basic physical level. We adopt an evolutionary perspective by modelling the agent based on biologically plausible principles, whereby Emotions emerge from homeostatic mechanisms. We suggest that the agent should be embodied so as to allow its behaviour to be affected by low-level physical tasks. By embodiment we mean that the agent has a virtual physical body whose states can be sensed by the agent itself. The simulations show the emergence of a stable emotional system with emotional contexts resulting from dynamical categorization of objects in the world. This proved to be effective and versatile enough to allow the agent to adapt itself to unknown world configurations. The results are coherent with Antonio Damasio's theory of background emotional system [1]. We demonstrate that body/world categorizations and body maps can evolve from a simple rule: self-survival.

[1] A. Damasio, The Feeling of What Happens: Making of Consciousness. Vintage, 2000.

Citation (APA style)
Coutinho, E., Miranda, E.R. & Cangelosi, A. (2005). Towards a Model for Embodied Emotions. In C. Bento, A. Cardoso, & G. Dias (Eds.), Proceedings of the Portuguese Conference on Artificial Intelligence (EPIA'05) (pp. 54-63). Covilha: IEEE Press. DOI: 10.1109/EPIA.2005.341264

DOI
10.1109/EPIA.2005.341264

Evolving emotional behaviour for expressive performance of music
E. Coutinho, E.R. Miranda & P. Silva
Lecture Notes in Computer Science: Intelligent Virtual Agents


Abstract
Today computers can be programmed to compose music automatically, using techniques ranging from rule-based to evolutionary computation (e.g., genetic algorithms and cellular automata). However, we lack good techniques for programming the computer to play or interpret music with expression. Expression in music is largely associated with emotions. Therefore we are looking into the possibility of programming computer music systems with emotions. We are addressing this problem from an A-Life perspective combined with recent discoveries in the neurosciences with respect to emotion.
Antonio Damasio refers to the importance of emotions to assist an individual to maintain survival, as they seem to be an important mechanism for adaptation and decision-making. Specifically, environmental events of value should be susceptible to preferential perceptual processing, regarding their pleasant or unpleasant. This approach assumes the existence of neural pathways that facilitate survival. Stable emotional systems should then emerge from self-regulatory homeostatic processes.
We implemented a system consisting of an agent that inhabits an environment containing with a number of different objects. These objects cause different physiological reactions to the agent. The internal body state of the agent is defined by a set of internal drives and a set of physiological variables that vary as the agent interacts with the objects it encounters in the environment. The agent is controlled by a feed-forward neural network that integrates visual input with information about its internal states. The network learns through a reinforcement-learning algorithm, derivate from different body states, due to pleasant or unpleasant stimuli.
The playback of musical recordings in MIDI format is steered by the physiological variables of the agent in different phases of the adaptation process.
The behaviour of the system is coherent with Damasio's theory of background emotional system. It demonstrates that specific phenomena, such as body/world categorization and existence of a body map, can evolve from a simple rule: self-survival in the environment. Currently, we are in the process of defining a system of higher-level emotional states (or foreground system) that will operate in social contexts; i.e., with several agents in the environment reacting to objects and interacting with each other.

Citation (APA style)
Coutinho, E., Miranda, E.R. & Silva, P. (2005). Evolving emotional behaviour for expressive performance of music. In T. Panayiotopoulos, J. Gratch, R. Aylett, D. Ballin, P. Olivier & T. Rist (Eds.), Lecture Notes in Computer Science: Intelligent Virtual Agents, 3661, 147. Berlin/Heidelberg: Springer. DOI: 10.1007/11550617_48

DOI
10.1007/11550617_48

The dynamics of music perception and emotional experience: a connectionist model
Eduardo Coutinho, Angelo Cangelosi
Proceedings of the 9th International Conference on Music Perception and Cognition (ICMPC9)


Abstract
In this paper we present a methodological framework for the study of musical emotions, incorporating psychophysiological experiments and modelling techniques for data analysis. Our focus is restricted to the body implications as a possible source of information about the emotional experience, and responsible to certain levels of emotional engagement in music. We present and apply the use of spatiotemporal connectionist models, as a modelling technique. Simulation results using a simple recurrent network, demonstrate that our connectionist approach leads to a better fit of the simulated process, compared with previous models. We demonstrate that a spatiotemporal connectionist model trained on music and emotional rating data is capable of generalizing the level of arousal in response to novel music input. The model is also capable of identifying the main variables responsible for such an emotional rating behaviour.

Key words: Music, emotion, brain, body, neural networks

Citation (APA style)
Coutinho, E. & Cangelosi, A. (2006). The dynamics of music perception and emotional experience: a connectionist model. In M. Baroni, A.R. Addessi, & M. Costa (Eds.), Proceedings of the 9th International Conference on Music Perception and Cognition (ICMPC9) (pp. 1096-1104). Bologna: Bologna University Press. ISBN 88-7395-155-4.

ISBN
88-7395-155-4

Emotion and Embodiment in Cognitive Agents: from Instincts to Music
E. Coutinho & A. Cangelosi
Proceedings of the 2007 International Conference on Integration of Knowledge Intensive Multi-Agent Systems (KIMAS'07)


Abstract
This paper suggests the use of modeling techniques to tack into the emotion/cognition paradigm. We presented two possible frameworks focusing on the embodiment basis of emotions. The first one explores the emergence of emotion mechanisms, by establishing the primary conditions of survival and exploring the basic roots of emotional systems. These simulations show the emergence of a stable motivational system with emotional contexts resulting from dynamical categorization of objects in the environment, in answer to survival pressures and homeostatic processes. The second framework uses music as a source of information about the mechanism of emotion and we propose a model based on recurrent connectionist architectures for the prediction of emotional states in response to music experience. Results demonstrate that there are strong relationships between arousal reports and music psychoacoustics, such as tempo and dynamics. Finally we discuss future directions of research on emotions based on cognitive agents and mathematical models.

Citation (APA style)
Coutinho E. & Cangelosi A. (2007). Emotion and Embodiment in Cognitive Agents: from Instincts to Music. In H. Hexmoor & C. Thompson (Ed.), Proceedings of the 2007 International Conference on Integration of Knowledge Intensive Multi-Agent Systems (KIMAS'07) (pp. 133-138). Waltham, MA: IEEE Press. DOI: 10.1109/KIMAS.2007.369798

DOI
10.1109/KIMAS.2007.369798.

Psycho-physiological patterns of musical emotions and their relation with music structure
Eduardo Coutinho, Angelo Cangelosi
Proceedings of the 10th International Conference on Music Perception and Cognition (ICMPC10)


Abstract
This study investigates the dynamics of psychological and physiological reactions during music listening to determine whether differentiated psychological and physiological patterns could be related with differentiated patterns of music variables. We asked 39 participants to give continuous self-reports of the intensity (Arousal) and hedonic value (Valence) of the emotions felt while listening to 9 pieces of western instrumental art (classical) music. Simultaneously we recorded their Heart Rate (HR) and Skin Conductance Response (SCR) for the full length of the pieces. Arousal was found to increase for higher levels of tempo, loudness, mean pitch, sharpness and timbral width, and Valence was correlated with variations in tempo and (tonal) dissonance. Psychological and physiological reports showed that increased Arousal and Valence are related with increased SCR, while increased HR related with higher Arousal. We also found a negative relationship between Valence and Heart Rate. Our study supports the idea that significant patterns of interactions between music structure and differentiated levels of intensity and hedonic value do exist in musical emotions. Our study also shows differentiable physiological patterns across different emotions, supporting the claim that physiological arousal is also a component of musical emotions. We are currently creating a neural network model to analyse in detail such interactions.

Citation (APA style)
Coutinho, E. & Cangelosi, A. (2008). Psycho-physiological patterns of musical emotions and their relation with music structure. In K. Miyazaki, Y. Hiraga, M. Adachi, Y. Nakajima, & M. Tsuzaki (Eds.), Proceedings of the 10th International Conference on Music Perception and Cognition (ICMPC10) (p. 94). Sapporo (Japan): Causal Productions. ISBN: 978-4-9904208-0-2

ISBN
978-4-9904208-0-2

Computational and Psycho-Physiological Investigations of Musical Emotions
E. Coutinho
University of Plymouth


Abstract
The ability of music to stir human emotions is a well known fact (Gabrielsson & Lindstrom, 2001). However, the manner in which music contributes to those experiences remains obscured. One of the main reasons is the large number of syndromes that characterise emotional experiences. Another is their subjective nature: musical emotions can be affected by memories, individual preferences and attitudes, among other factors (Scherer & Zentner, 2001). But can the same music induce similar affective experiences in all listeners, somehow independently of acculturation or personal bias? A considerable corpus of literature has consistently repor ted that listeners agree rather strongly about what type of emotion is expressed in a particular piece or even in particular moments or sections (Juslin & Sloboda, 2001). Those studies suggest that music features encode important characteristics of affective experiences, by suggesting the influence of various structural factors of music on emotional expression. Unfortunately, the nature of these relationships is complex, and it is common to find rather vague and contradictory descriptions.
This thesis presents a novel methodology to analyse the dynamics of emotional responses to music. It consists of a computational investigation, based on spatiotemporal neural networks sensitive to structural aspects of music, which "mimic" human affective responses to music and permit to predict new ones. The dynamics of emotional responses to music are investigated as computational representations of perceptual processes (psychoacoustic features) and self-perception of physiological activation (peripheral feedback). Modelling and experimental results provide evidence suggesting that spatiotemporal patterns of sound resonate with affective features underlying judgements of subjective feelings. A significant part of the listener's affective response is predicted from the a set of six psychoacoustic features of sound - tempo, loudness, multiplicity (texture), power spectrum centroid (mean pitch), sharpness (timbre) and mean STFT flux (pitch variation) - and one physiological variable - heart rate. This work contributes to new evidence and insights to the study of musical emotions, with particular relevance to the music perception and emotion research communities.

Citation (APA style)
Coutinho, E. (2009). Computational and Psycho-Physiological Investigations of Musical Emotions. Unpublished doctoral dissertation, University of Plymouth (United Kingdom).

The use of spatio-temporal connectionist models in psychological studies of musical emotions
Eduardo Coutinho, Angelo Cangelosi
Music Perception


Abstract
This article presents a novel methodology to analyze the dynamics of emotional responses to music. It consists of a computational investigation based on spatiotemporal neural networks, which "mimic" human affective responses to music and predict the responses to novel music sequences. The results provide evidence suggesting that spatiotemporal patterns of sound resonate with affective features underlying judgments of subjective feelings (arousal and valence). A significant part of the listener's affective response is predicted from a set of six psychoacoustic features of sound - loudness, tempo, texture, mean pitch, pitch variation, and sharpness. A detailed analysis of the network parameters and dynamics also allows us to identify the role of specific psychoacoustic variables (e.g., tempo and loudness) in music emotional appraisal. This work contributes new evidence and insights to the study of musical emotions, with particular relevance to the music perception and cognition research community.

Key words: emotion, music, arousal and valence, psychoacoustics, neural networks

Citation (APA style)
Coutinho, E. & Cangelosi, A. (2009). The use of spatio-temporal connectionist models in psychological studies of musical emotions. Music Perception, 27 (1), 1-15. DOI: 10.1525/mp.2009.27.1.1

DOI
10.1525/mp.2009.27.1.1

Cognitive Dissonance, Knowledge Instinct and Musical Emotions
E. Coutinho
Physics of Life Reviews


Abstract
(Introductory paragraph)

My starting point for this commentary is Perlovsky's suggestion that cognitive dissonance is a similar concept to "damaged" cognition due to differentiation, and a possible framework to test his hypotheses that the major role of musical emotions is to reconcile contradictions in consciousness, i.e. to restore "synthesis" [11]. I will first analyze and address the commonalities between both phenomena. Then, I will brie?y discuss the specialization of music as a referential for emotion, and propose an experimental framework to test the capacity for music to reduce cognitive dissonance and to promote the acquisition of differentiated knowledge.

Key words: Music; Emotion; Cognition; Differentiation; Synthesis

Citation (APA style)
Coutinho, E. (2010). Cognitive Dissonance, Knowledge Instinct and Musical Emotions. Physics of Life Reviews, 7(1), 30-32. DOI: 10.1016/j.plrev.2010.01.016

DOI
10.1016/j.plrev.2010.01.016

Modeling psycho-physiological measurements of emotional responses to multiple music genres
E. Coutinho
Proceedings of 11th International Conference of Music Perception and Cognition (ICMPC11)


Abstract
We sustain that the structure of affect elicited by music is largely dependent on low-level psychoacoustic properties to which humans are particularly sensitive. In support of this claim, we have previously provided evidence that spatiotemporal dynamics in "secondary" music structural parameters resonate with two psychological dimensions of affect underlying judgments of subjective feelings: arousal and valence. In this article we extend those investigations in two aspects. Firstly, whilst previously we focused on western classical music, here we use a repertoire of multiple music genres in order to verify the extent to which the relationships linking sound features and emotion reflect general principles across music genres, or, instead, are genre-specific. The second aspect involves the evaluation of physiological cues as predictors of emotional responses to music. Akin to our previous findings, we will show that a significant part of the listeners' reported emotions (more than 70% of the variance in arousal and valence) can be predicted from a set of six psychoacoustic features - loudness, tempo, pitch level, melodic pitch, sharpness and texture. The accuracy of those predictions is improved with the inclusion of one physiological variable - skin conductance response. Furthermore, we will demonstrate that the configurations of features with emotional meaning learned from pop, dance or rock music, are applicable to the prediction of emotional responses to classical and film music. This work contributes with new insights to the study of musical emotions, by showing that similar acoustic templates are shared across genres to convey emotional meaning.

Key words: Music and emotions; Cognitive modeling of music; Acoustics and psychoacoustics

Citation (APA style)
Coutinho, E. (2010). Modeling psycho-physiological measurements of emotional responses to multiple music genres. In S.M. Demorest, S.J. Morrison, & P.S. Campbell (Eds.), Proceedings of the 11th International Conference of Music Perception and Cognition (ICMPC11) (p. 53). Seattle, WA, USA.

Music, Speech and Emotion: psycho-physiological and computational investigations
E. Coutinho & N. Dibben
Proceedings of the International Conference on Interdisciplinary Musicology: Nature versus Culture (CIM'10)


Abstract

1. Background in Music Cognition.
There is growing evidence that perception of emotion expressed in vocal prosody and in music shares certain psychoacoustic attributes (Juslin & Laukka 2003). From the perspective of musicology these findings are significant because they indicate that listeners' affective responses to music can be accounted for at least in part by "basic" acoustic cues.

2. Background in Computer Science.
Previous research has shown that a large part of listeners' affective response to music can be predicted from a small set of psychoacoustic cues (Coutinho & Cangelosi, 2009), and physiological variables (Coutinho & Cangelosi, in press).

3. Aims
Our goal is to better understand the way people perceive emotion in music and speech prosody, i.e. the way people infer the emotional state of others from nonverbal aspects of speech.

4. Main Contribution
The method involves asking participants to listen to a set of music pieces and speech excerpts, while reporting their emotional experience using a computer framework interfaced with a device that controls the movement of the cursor or pointer on a display screen. At the same time we measure participants' physiological reactions during listening, namely heart rate, skin conductance, respiration and blood volume pressure. The data collected from the mouse position and the physiological readings is used to analyze the participants' psychological and physiological reactions to the stimuli and to compare them with the psychoacoustic properties of the music and speech. Each participant also completes a set of short questionnaires that gather relevant personal information, such as age and gender (and other demographic variables), exposure to music, years of musical training, personality traits and mood state. This information is used to test various sub-hypotheses about the factors influencing perception of emotion in music and speech prosody. This is the first time that an attempt has been made to study the relationship between expression of emotion in music and speech prosody in longer excerpts of music and speech.

5. Implications
Evidence that the perception of emotion conveyed by music and speech relies on shared psychoacoustic characteristics lends credence to the idea that, to some extent at least, emotion perception relies on 'basic' attributes that are adaptive responses to environmental cues. This emphasises the role of the natural in emotional expression in music, contrary to the predominance of cultural (semiotic) models of musical expression. Furthermore, the identification of shared characteristics of emotion expression in music and speech prosody may contribute to evolutionary perspectives on music and language.

6. References
Coutinho, E., & Cangelosi, A. (2009). The use of spatio-temporal connectionist models in psychological studies of musical emotions. Music Perception, 27 (1), 1-15.
Coutinho, E., & Cangelosi, A. (in press). Musical emotions: predicting second-by-second subjective feelings of emotion from psychophysiological measurements. Manuscript submitted for publication.
Juslin, P., & Laukka, P. (2003). Communication of emotions in vocal expression and music performance: Different channels, same code? Psychological Bulletin, 129 (5), 770-814.

Citation (APA style)
Coutinho, E. & Dibben, N. (2010). Music, Speech and Emotion: psycho-physiological and computational investigations. In N. Dibben & R. Timmers (Eds.), Proceedings of the International Conference on Interdisciplinary Musicology: Nature versus Culture (CIM'10).Sheffield, UK.

A Neural Network Model for the Prediction of Musical Emotions
E. Coutinho & A. Cangelosi
Advances in Cognitive Systems



Abstract
This chapter presents a novel methodology to analyse the dynamics of emotional responses to music in terms of computational representations of perceptual processes (psychoacoustic features) and self-perception of physiological activation (peripheral feedback). The approach consists of a computational investigation of musical emotions based on spatio-temporal neural networks sensitive to structural aspects of music. We present two computational studies based on connectionist network models that predict human subjective feelings of emotion. The first study uses six basic psychoacoustic dimensions extracted from the music pieces as predictors of the emotional response. The second computational study evaluates the additional contribution of physiological arousal to the subjective feeling of emotion. Both studies are backed up by experimental data. A detailed analysis of the simulation models' results demonstrates that a significant part of the listener's affective response can be predicted from a set of psychoacoustic features of sound - tempo, loudness, multiplicity (texture), power spectrum centroid (mean pitch), sharpness (timbre) and mean STFT flux (pitch variation) - and one physiological cue, heart rate. This work provides a new methodology to the field of music and emotion research based on combinations of computational and experimental work, which aid the analysis of emotional responses to music, while offering a platform for the abstract representation of those complex relationships.

Citation (APA style)
Coutinho, E. & Cangelosi, A. (2010). A Neural Network Model for the Prediction of Musical Emotions. In S. Nefti-Meziani & J.G. Grey (Ed.). Advances in Cognitive Systems (pp. 331-368). London: IET Publisher. ISBN: 978-1849190756

ISBN
978-1849190756

Musical Emotions: Predicting Second-by-Second Subjective Feelings of Emotion From Low-Level Psychoacoustic Features and Physiological Measurements
E. Coutinho & A. Cangelosi
Emotion


Abstract
We sustain that the structure of affect elicited by music is largely dependent on dynamic temporal patterns in low-level music structural parameters. In support of this claim, we have previously provided evidence that spatiotemporal dynamics in psychoacoustic features resonate with two psychological dimensions of affect underlying judgments of subjective feelings: arousal and valence. In this article we extend our previous investigations in two aspects. First, we focus on the emotions experienced rather than perceived while listening to music. Second, we evaluate the extent to which peripheral feedback in music can account for the predicted emotional responses, that is, the role of physiological arousal in determining the intensity and valence of musical emotions. Akin to our previous findings, we will show that a significant part of the listeners’ reported emotions can be predicted from a set of six psychoacoustic features—loudness, pitch level, pitch contour, tempo, texture, and sharpness. Furthermore, the accuracy of those predictions is improved with the inclusion of physiological cues—skin conductance and heart rate. The interdisciplinary work presented here provides a new methodology to the field of music and emotion research based on the combination of computational and experimental work, which aid the analysis of the emotional responses to music, while offering a platform for the abstract representation of those complex relationships. Future developments may aid specific areas, such as, psychology and music therapy, by providing coherent descriptions of the emotional effects of specific music stimuli.

Key words: Emotion, Arousal and Valence, Physiology, Psychoacoustics, Neural Networks

Citation (APA style)
Coutinho, E., & Cangelosi, A. (2011). Musical Emotions: Predicting Second-by-Second Subjective Feelings of Emotion From Low-Level Psychoacoustic Features and Physiological Measurements. Emotion, 11(4), 921-937. doi:10.1037/a0024700

DOI
10.1037/a0024700



Share |