Multimodal Affective Computing: Affective Information Representation, Modelling, and Analysis

Computer Assistive Technologies for Physically and Cognitively Challenged Users focuses on the technologies and devices that assist individuals with physical and cognitive disabilities. These technologies facilitate independent activity and participation, serving to improve daily functional capabilities. The book features nine chapters that cover a wide range of computer assistive technologies that give readers an indepth understanding of the available resources to help the elderly or individuals with disabilities. The topics covered in the book include 1) The category and ontology of assistive devices, 2) Web accessibility and ICT accessibility for persons with disability (PWD), 3) Assistive technologies for blind and visually impaired people, 4) Assistive technologies for home comfort and care, 5) Assistive technologies for hearing impaired people using Indian sign language synthetic animations, 6) Augmentative and alternative communication/hearing impairments, 7) Accessibility innovations to help physically disabled users, 8) Adhesive tactile walking surface indicators for elderly and visually impaired people mobility, 9) future of assistive technologies This book serves as a textbook resource for students undertaking modular courses that require learning material on computer assistive technology. It also serves as a reference for graduate level courses in disability studies, humancomputer interaction, gerontology and rehabilitation engineering. Researchers working in the allied fields intersecting computer science, medicine and psychology will also benefit from the information provided in the book. Cover 1 Title 2 Copyright 3 End User License Agreement 4 Contents 6 Foreword 10 Preface 11 CONSENT FOR PUBLICATION 12 CONFLICT OF INTEREST 12 Acknowledgements 13 Affective Computing 14 1.1. INTRODUCTION 14 1.2. WHAT IS EMOTION? 15 1.2.1. Affective Human-Computer Interaction 15 1.3. BACKGROUND 16 1.4. THE ROLE OF EMOTIONS IN DECISION MAKING 17 1.5. CHALLENGES IN AFFECTIVE COMPUTING 18 1.5.1. How Can Many Emotions Be Analyzed in a Single Framework? 18 1.5.2. How Can Complex Emotions Be Represented in a Single Framework Or Model? 19 1.5.3. Is The Chosen Theoretical Viewpoint Relevant to other Areas Of Affective Computing? 19 1.5.4. How Can Physiological Signals Be Used to Anticipate Complicated Emotions? 19 1.6. AFFECTIVE COMPUTING IN PRACTICE 19 1.6.1. Avatars or Virtual Agents 20 1.6.2. Robotics 20 1.6.3. Gaming 21 1.6.4. Education 22 1.6.5. Medical 22 1.6.6. Smart Homes and Workplace Environments 23 CONCLUSION 23 REFERENCES 23 Affective Information Representation 26 2.1. INTRODUCTION 26 2.2. AFFECTIVE COMPUTING AND EMOTION 26 2.2.1. Affective Human-Computer Interaction 27 2.2.2. Human Emotion Expression and Perception 28 2.2.2.1. Facial Expressions 28 2.2.2.2. AudioHG 28 2.2.2.3. Physiological Signals 29 2.2.2.4. Hand and Gesture Movement 30 2.3. RECOGNITION OF FACIAL EMOTION 30 2.3.1. Facial Expression Fundamentals 31 2.3.2. Emotion Modeling 32 2.3.3. Representation of Facial Expression 33 2.3.4. Facial Emotion's Limitations 34 2.3.5. Techniques for Classifying Facial Expressions 34 CONCLUSION 38 REFERENCES 39 Models and Theory of Emotion 43 3.1. INTRODUCTION 43 3.2. EMOTION THEORY 43 3.2.1. Categorical Approach 44 3.2.2. Evolutionary Theory of Emotion by Darwin 45 3.2.3. Cognitive Appraisal and Physiological Theory of Emotions 46 3.2.4. Dimensional Approaches to Emotions 47 CONCLUSION 50 REFERENCES 50 Affective Information Extraction, Processing and Evaluation 53 4.1. INTRODUCTION 53 4.2. AFFECTIVE INFORMATION EXTRACTION AND PROCESSING 53 4.2.1. Information Extraction from Audio 53 4.2.2. Information Extraction from Video 54 4.2.3. Information Extraction from Physiological Signals 54 4.3. STUDIES ON AFFECT INFORMATION PROCESSING 55 4.4. EVALUATION 56 4.4.1. Types of Errors 56 4.4.1.1. False Acceptance Ratio 56 4.4.1.2. False Reject Ratio 57 4.4.2. Threshold Criteria 57 4.4.3. Performance Criteria 57 4.4.4. Evaluation Metrics 58 4.4.4.1. Mean Absolute Error (MAE) 58 4.4.4.2. Mean Square Error (MSE) 58 4.4.5. ROC Curves 58 4.4.6. F1 Measure 59 CONCLUSION 60 REFERENCES 60 Multimodal Affective Information Fusion 62 5.1. INTRODUCTION 62 5.2. MULTIMODAL INFORMATION FUSION 62 5.2.1. Early Fusion 63 5.2.2. Intermediate Fusion 64 5.2.3. Late Fusion 64 5.3. LEVELS OF INFORMATION FUSION 66 5.3.1. Sensor or Data-level Fusion 67 5.3.2. Feature Level Fusion 68 5.3.3. Decision-Level Fusion 68 5.4. MAJOR CHALLENGES IN INFORMATION FUSION 68 CONCLUSION 69 REFERENCES 69 Multimodal Fusion Framework and Multiresolution Analysis 72 6.1. INTRODUCTION 72 6.2. THE BENEFITS OF MULTIMODAL FEATURES 72 6.2.1. Noise In Sensed Data 73 6.2.2. Non-Universality 73 6.2.3. Complementary Information 74 6.3. FEATURE LEVEL FUSION 74 6.4. MULTIMODAL FEATURE-LEVEL FUSION 75 6.4.1. Feature Normalization 75 6.4.2. Feature Selection 76 6.4.3. Criteria For Feature Selection 76 6.5. MULTIMODAL FUSION FRAMEWORK 78 6.5.1. Feature Extraction and Selection 78 6.5.1.1. Extraction of Audio Features 78 6.5.1.2. Extraction of Video Features 78 6.5.1.3. Extraction of Peripheral Features from EEG 79 6.5.2. Dimension Reduction and Feature-level Fusion 79 6.5.3. Emotion Mapping to a 3D VAD Space 80 6.6. MULTIRESOLUTION ANALYSIS 83 6.6.1. Motivations for the use of Multiresolution Analysis 84 6.6.2. The Wavelet Transform 84 6.6.3. The Curvelet Transform 85 6.6.4. The Ridgelet Transform 86 CONCLUSION 86 REFERENCES 86 Emotion Recognition From Facial Expression In A Noisy Environment 88 7.1. INTRODUCTION 88 7.2. THE CHALLENGES IN FACIAL EMOTION RECOGNITION 89 7.3. NOISE AND DYNAMIC RANGE IN DIGITAL IMAGES 91 7.3.1. Characteristic Sources Of Digital Image Noise 92 7.3.1.1. Sensor Read Noise 92 7.3.1.2. Pattern Noise 92 7.3.1.3. Thermal Noise 92 7.3.1.4. Pixel Response Non-uniformity (PRNU) 92 7.3.1.5. Quantization Rrror 92 7.4. THE DATABASE 93 7.4.1. Cohn-Kanade Database 93 7.4.2. JAFFE Database 93 7.4.3. In-House Database 93 7.5. EXPERIMENTS WITH THE PROPOSED FRAMEWORK 93 7.5.1. Image Pre-Processing 95 7.5.2. Feature Extraction 95 7.5.3. Feature Matching 95 7.6. RESULTS AND DISCUSSIONS 97 7.7. RESULTS UNDER ILLUMINATION CHANGES 100 7.8. RESULTS UNDER GAUSSIAN NOISE 100 7.8.1. Comparison with other Strategies 100 CONCLUSION 107 REFERENCES 107 Spontaneous Emotion Recognition From Audio-Visual Signals 110 8.1. INTRODUCTION 110 8.2. RECOGNITION OF SPONTANEOUS EFFECTS 111 8.3. THE DATABASE 111 8.3.1. eNTERFACE Database 112 8.3.2. RML Database 113 8.4. AUDIO-BASED EMOTION RECOGNITION SYSTEM 113 8.4.1. Experiments 114 8.4.2. System Development 114 8.4.2.1. Audio Features 114 8.5. VISUAL CUE-BASED EMOTION RECOGNITION SYSTEM 117 8.5.1. Experiments 117 8.5.2. System Development 117 8.5.2.1. Visual Feature 117 8.6. EXPERIMENTS BASED ON THE PROPOSED AUDIO-VISUAL CUES FUSION FRAMEWORK 120 8.6.1. Results 122 8.6.2. Comparison To Other Research 1 CONCLUSION 124 REFERENCES 124 Multimodal Fusion Framework: Emotion Recognition From Physiological Signals 128 9.1. INTRODUCTION 128 9.1.1. Electrical Brain Activity 129 9.1.2. Muscle Activity 130 9.1.3. Skin Conductivity 130 9.1.4. Skin Temperature 130 9.2. MULTIMODAL EMOTION DATABASE 130 9.2.1. DEAP Database 131 9.3. FEATURE EXTRACTION 131 9.3.1. Feature Extraction from EEG 132 9.3.2. Feature Extraction from Peripheral Signals 132 9.4. CLASSIFICATION AND RECOGNITION OF EMOTION 133 9.4.1. Support Vector Machine (SVM) 133 9.4.2. Multi-Layer Perceptron (MLP) 134 9.4.3. K-Nearest Neighbor (K-NN) 135 9.5. RESULTS AND DISCUSSION 136 9.5.1. Emotion Categorization Results Based On The Proposed Multimodal Fusion Architecture 136 CONCLUSION 139 REFERENCES 139 Emotions Modelling in 3D Space 141 10.1. INTRODUCTION 141 10.2. AFFECT REPRESENTATION IN 2D SPACE 142 10.3. EMOTION REPRESENTATION IN 3D SPACE 144 10.4. 3D EMOTION MODELING VAD SPACE 146 10.5. EMOTION PREDICTION IN THE PROPOSED FRAMEWORK 149 10.5.1. Multimodal Data Processing 150 10.5.1.1. Prediction of Emotion from a Visual Cue 151 10.5.1.2. Prediction of Emotion from Physiological Cue 152 10.5.2. Ground Truth Data 152 10.5.3. Emotion Prediction 153 10.6. FEATURE SELECTION AND CLASSIFICATION 153 10.7. RESULTS AND DISCUSSIONS 154 CONCLUSION 158 REFERENCES 159 Subject Index 161 Back Cover 167 Affective computing is an emerging field situated at the intersection of artificial intelligence and behavioral science. Affective computing refers to studying and developing systems that recognize, interpret, process, and simulate human emotions. It has recently seen significant advances from exploratory studies to real-world applications. Multimodal Affective Computing offers readers a concise overview of the state-of-the-art and emerging themes in affective computing, including a comprehensive review of the existing approaches in applied affective computing systems and social signal processing. It covers affective facial expression and recognition, affective body expression and recognition, affective speech processing, affective text, and dialogue processing, recognizing affect using physiological measures, computational models of emotion and theoretical foundations, and affective sound and music processing. This book identifies future directions for the field and summarizes a set of guidelines for developing next-generation affective computing systems that are effective, safe, and human-centered.The book is an informative resource for academicians, professionals, researchers, and students at engineering and medical institutions working in the areas of applied affective computing, sentiment analysis, and emotion recognition.