Applied Affective Computing

Applied Affective Computing Contents List of Figures List of Tables Preface Acknowledgments 1 Introduction to Applied Affective Computing 1.1 Affective Computing 1.2 Applied Affective Computing 1.2.1 Challenges of Applied Affective Computing 1.2.1.1 Developing a Human-centered Affective Computing System 1.2.1.2 Developing a Reliable Affective Computing System 1.2.1.3 Developing an Adaptive Affective Computing System 1.2.1.4 Developing an Integrated Affective Computing System 1.2.1.5 Developing an Ethical Affective Computing System 1.2.2 A Living Ontology of Affective Computing 1.3 Book Overview and Highlights 1.4 Contributions 2 Emotions as Studied in Psychology and Cognitive Science 2.1 Theories of Emotion 2.1.1 The Evolutionary Approach 2.1.2 The Appraisal Approach 2.1.3 The Constructionism Approach 2.2 Emotion as an Adaptive Function 2.3 Emotion as a Social Communicative Function 2.3.1 Empathy 2.3.2 Social Intelligence and Socio-affective Competence 2.3.3 Social Norms 2.4 Summary 3 Machine Learning Approaches for Applied Affective Computing 3.1 Machine Learning for Affective Computing 3.2 Deep Machine Learning for Affective Computing 3.3 Multimodal Representation of Affect: Knowledge-inspired versus Data-driven Features 3.4 Unimodal Features for Affect Recognition 3.4.1 Visual Modality—Facial Expressions and Body Gestures 3.4.1.1 Facial Expression 3.4.1.2 Body Gestures 3.4.2 Deep Learning-based Visual Feature Extraction 3.4.3 Audio Modality 3.4.4 Deep Learning-based Audio Feature Extraction 3.4.5 Physiological Modality 3.4.6 Deep Learning-based Physiological Feature Extraction 3.4.7 Linguistic Modality 3.4.8 Deep Learning-based Linguistic Feature Extraction 3.5 Multimodal Emotion-aware Systems 3.6 Evaluation of Emotion-aware Systems 3.7 Synthesis of Emotion and Emotional Behaviors 3.8 Discussion 3.8.1 Context Matters 3.8.2 Emotion Representation: Categories versus Dimensions 3.8.3 Perceived versus Felt Affect 3.9 Conclusion 4 Multimodal Data Collection and Processing for Applied Affective Computing 4.1 Multimodal Data Collection 4.1.1 Emotion Labels 4.1.2 Multimodal Data 4.1.2.1 Facial Expression 4.1.2.2 Speech 4.1.2.3 Body Movement 4.1.2.4 Text 4.1.2.5 Physiological Signals 4.1.2.6 Context Information 4.2 Multimodal Data Processing 4.2.1 Preprocessing 4.2.2 Feature Extraction 4.2.3 Data Reduction/Selection 4.2.4 Feature Normalization 4.3 Multimodal Data Fusion 4.3.1 Model-agnostic Approaches 4.3.1.1 Feature Level Fusion (Early Fusion) 4.3.1.2 Intermediate Fusion 4.3.1.3 Late Fusion (Decision Level Fusion) 4.3.2 Model-based Approaches 4.3.2.1 Kernel Learning 4.3.2.2 Graphical Models 4.3.2.3 Neural Networks 4.3.2.4 Multimodal Deep Autoencoders 4.4 Conclusion and Future Work 5 Emotion Recognition in the Wild 5.1 Modalities 5.1.1 Emotion Recognition from Audio and Visual Modality 5.1.2 Emotion Recognition from Physiological Signals 5.2 Deep Learning for Emotion Recognition in the Wild 5.3 Emotion Representation in the Wild 5.4 Evaluation of Affect Recognition in the Wild 5.5 The Role of Context in Affect Recognition in the Wild 5.6 The Role of Environment in Affect Recognition in the Wild 5.7 Discussion 5.7.1 Emotion Annotations: Self-report versus Third-person Annotations 5.7.2 Long-term Research on Affect in the Wild 5.7.3 Ethical and Privacy Issues of Affect Recognition in the Wild 5.8 Conclusion 6 Reinforcement Learning and Affective Computing 6.1 Reinforcement Learning 6.1.1 Bandits and the Exploration–Exploitation Tradeoff 6.1.2 State Valuation 6.2 Affective Computing 6.2.1 Theories of Emotion 6.2.2 Behavioral Psychology 6.3 Extending the Reinforcement Learning Framework 6.3.1 Modeling Agent, Environment, and Critic 6.3.2 A Multilayered Affective Critic 6.3.3 Dynamic Goals and Rewards 6.3.4 Prioritization 6.4 Conclusions 7 Synthesizing Natural and Believable Emotional Expressions 7.1 Introduction 7.1.1 Historical Context 7.2 Emotional Expression Models 7.3 Synthesizing Emotional Expressions 7.4 The Role of Agency 7.5 Additional Considerations and Challenges 8 Emotion-aware Human–Robot Interaction and Social Robots 8.1 Developing Emotion-aware Human–Robot Interaction Systems 8.2 Improving Social Intelligence of a Robot by Addressing Social Errors in Human–Robot Interaction 8.3 Case Study: Prompting Human Assistance in Human–Robot Collaboration Using Robots' Emotional Expressions 8.3.1 Emotional Model of the Robot 8.3.2 Human–Multirobot Collaboration Task: The Tower Construction Game 8.3.3 Influence of Artificial Emotions on Human–Multirobot Collaboration 8.3.4 Validity and Expressiveness of the Robots' Emotional Expressions 8.3.5 Influence of Robots' Emotional Expressions on Human's Perception 8.3.6 Summary 8.4 Case Study: Understanding Users' Expectations of Robots in Public Spaces 8.4.1 Robots in Today's Public Spaces 8.4.2 Adopting Participatory Design to Understand Expectations and Perceptions of Robots in Public Spaces 8.4.3 Summary 8.5 Challenges in Affective HRI 8.6 Guidelines for Developing Affective HRI Systems and Social Robots 9 Affective Computing for Enhancing Well-Being 9.1 Motivation 9.2 Sensing and Analytics 9.3 Examples of Well-Being Studies 9.3.1 MIT Friends and Family 9.3.2 MoodScope 9.3.3 Student Life 9.3.4 SNAPSHOT Study 9.3.5 Detecting Momentary Affective Changes in Daily Life 9.4 Sensing Beyond the Phone 9.4.1 Stationary Compute 9.4.2 Home Monitoring 9.4.3 In Vehicle 9.4.4 Cross-Platform Integration 9.5 Actions and Interventions 9.5.1 Conversing 9.5.2 Reminding and Recommending 9.5.3 Behavior Modification 9.5.4 Positive Computing 9.6 Conclusions 10 Applied Affective Computing in Built Environments 10.1 Setting the Foundation Toward Emotionally Aware Planning and Design 10.1.1 Design and Evaluation of Urban Spaces 10.1.2 Emotion and Perception 10.1.3 Sensors and Sensor Fusion 10.2 Case Study: Passive Responses to Urban Infrastructure 10.3 Experimental Task 10.4 Data 10.4.1 Human Subjects 10.4.2 Built Environment 10.5 Approach 10.5.1 Data Normalization 10.5.2 Machine Learning Models 10.6 Results 10.7 Conclusions 10.8 Guidelines for Affective Computing in Built Environments 11 Addressing Ethical Issues of Affective Computing 11.1 Ethical Concerns of Affective Computing 11.2 A Fair System 11.2.1 Biases in Data and Models 11.2.2 Increasing Fairness 11.3 A Privacy-preserving System 11.3.1 Issues of Invasive Surveillance 11.3.2 Privacy Preservation 11.4 A Transparent System 11.4.1 Overtrust Toward Intelligent Systems 11.4.2 Supporting Informed Decisions 11.5 A Beneficial System 11.5.1 Understanding Human Preferences 11.5.2 Contradicting Interests 11.6 A Responsible System 11.6.1 Accountability 11.6.2 Governance 11.7 Conclusion 12 Future of Affective Computing and Applied Affective Computing 12.1 Applied Affective Computing: Guidelines for Best Practice 12.2 Example Uses of the Guidelines 12.2.1 Scenario-based Discussion 12.2.1.1 Example A: Novice Researcher 12.2.1.2 Example B: Experienced Researcher 12.2.1.3 Example C: Practitioner 12.2.2 Examples in Existing Studies 12.3 Open Challenges and Future Directions 12.3.1 Toward Rigorous Affective Computing Research Methodologies 12.3.1.1 Hypotheses Grounded in Emotion and Communication Theories 12.3.1.2 Ecological Validity 12.3.1.3 Evaluation with the Right Metrics 12.3.1.4 Reproducibility and Generalizability 12.3.2 Toward Personalized Affective Computing 12.3.3 Toward Adaptive Affective Computing 12.3.4 Toward Embodied Affective Computing 12.3.5 Application Domains of Interests 12.3.5.1 Healthcare 12.3.5.2 Education 12.3.5.3 Business 12.3.5.4 Design 12.3.5.5 Security 12.3.5.6 Entertainment 12.4 Summary Bibliography Authors' Biographies Index