HCI International 2022 - Late Breaking Papers. Multimodality in Advanced Interaction Environments : 24th International Conference on Human-Computer Interaction, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings

Volume LNCS 13519 is part of the refereed proceedings of the 24th International Conference on Human-Computer Interaction, HCII 2022, which was held virtually during June 26 to July 1, 2022. A total of 5583 individuals from academia, research institutes, industry, and governmental agencies from 88 countries submitted contributions, and 1276 papers and 275 posters were included in the proceedings that were published just before the start of the conference. Additionally, 296 papers and 181 posters are included in the volumes of the proceedings published after the conference, as “Late Breaking Work” (papers and posters). The contributions thoroughly cover the entire field of human-computer interaction, addressing major advances in knowledge and effective use of computers in a variety of application areas. Foreword HCI International 2022 Thematic Areas and Affiliated Conferences Conference Proceedings – Full List of Volumes 24th International Conference on Human-Computer Interaction (HCII 2022) HCI International 2023 Contents Multimodal Interaction and Psychophysiological Computing 3D Hand Pose Recognition Over a Wide Area Using Two Omnidirectional Cameras with Field-of-view Division 1 Introduction 2 3D Hand Pose Recognition Using FOV Division and Two-Stage Skeleton Detection 2.1 Geometric Transformation of Omnidirectional Camera Images 2.2 FOV Division 2.3 Two-Stage Skeleton Detection 2.4 3D Hand Pose Estimation Using Two Cameras 3 Performance Evaluation 3.1 Procedure 3.2 Results 4 Conclusion References Towards a Dynamic Model for the Prediction of Emotion Intensity from Peripheral Physiological Signals 1 Introduction 2 Related Work 3 Models 3.1 NARX Dynamic Model 3.2 Self-assessment 3.3 Proposed Approach 4 Experiment and Data Acquisition 4.1 Participants 4.2 Experimental Design 4.3 Experimental Setup and Procedure 5 Data Processing 5.1 Pre-processing 5.2 Feature Extraction 5.3 Intensity Prediction 6 Evaluation 6.1 Methods 6.2 Results 7 Conclusion and Future Directions References Introduction to the 2nd Edition of ``Semantic, Artificial and Computational Interaction Studies'' 1 Motivation 2 Selected Historical Background 3 Overview of Contributions 4 Conclusion References Towards Efficient Odor Diffusion with an Olfactory Display Using an Electronic Nose 1 Introduction 2 Statement of the Objective 3 Our Olfactory Display System + Electronic Nose Prototype 4 The 3D Virtual Environment 5 Preliminary Results 6 Conclusions References Rendering Personalized Real-Time Expressions While Speaking Under a Mask 1 Introduction 2 Related Work 2.1 GAN-Based Rendering and the Limitations 2.2 Customization or Personalization by Machine Learning 2.3 Multicultural and Diverse Face Masks During COVID-19 3 Generating Facial Expressions from Masked Faces 3.1 Masked Face Segmentation 3.2 Facial Expression Generation 3.3 Landmarks Detection Within Face 3.4 Generating Faces with Expressions for Masked Faces 4 Customizing or Personalizing Facial Expressions Under Masked Faces 4.1 Contrastive Learning 5 Experimental Results, Discussions, and Conclusions References Applying Generative Adversarial Networks and Vision Transformers in Speech Emotion Recognition 1 Introduction 2 Methodology 2.1 Data 2.2 Feature Extraction 2.3 Description of CycleGAN 2.4 Vision Transformer 3 Results 4 Conclusions References Development of a Web-Based Interview Support System Using Characters Nodding with Various Movements 1 Introduction 2 Web-Based Interview Support System Using Characters Nodding with Various Movements 2.1 Questionnaire Survey on Web-Based Interviews 2.2 System Prototype with Interaction Model for Auto-generated Entrained Motion 2.3 Characters Nodding with Various Movements 2.4 Web-Based Interview Support System 3 Evaluation Experiment 3.1 Experimental Setup 3.2 Experimental Results 4 Conclusion References Emotion Recognition from Physiological Signals Using Continuous Wavelet Transform and Deep Learning 1 Introduction 2 Related Work 3 Methods 3.1 Emotional State Classification 3.2 Dataset 3.3 Sliding Window Mechanism 3.4 Signal Transformation Using CWT 3.5 Convolutional Neural Network Model 4 Results and Discussions 4.1 Performance Evaluation 4.2 Comparison with Existing Models 5 Conclusions References Surrogate Sensory Feedback of Grip Force in Older and Younger Participants only Influences Fine Motor Control, but Not the Object Weight Perception 1 Introduction 2 Method 2.1 Participants 2.2 Experimental Tools and Task 3 Results 3.1 Psychophysical Measures 3.2 Precision Grip Measures 3.3 Aging Effects on Hand Skin Sensitivity-Hydration Measures 3.4 Correlation Between Hand Physiological Parameters and Weight Discrimination Task Measures 4 Discussion References Laughter Meaning Construction and Use in Development: Children and Spoken Dialogue Systems 1 Introduction 2 Laughter as Language: Triangulating Methods and Insights 2.1 A Multi-layered Framework for Laughter Analysis 2.2 Positioning Laughter 2.3 Characterising Different Laughter Functions: Gaze and Accompanying Dialogue Acts 3 Learning to Laugh: Developmental Data and Communicative Feedback 3.1 Contingent Responses 3.2 Reformulation and Clarification Requests 4 Laughter to Correct 5 Implication for Spoken Dialogue Systems 5.1 Communicative Feedback Learning for Laughter Meaning Adjustments 5.2 Failure Detection and Failure Management 6 Conclusion, Limitations and Further Directions References Skeleton-Based Sign Language Recognition with Graph Convolutional Networks on Small Data 1 Introduction 2 Related Works 3 System Design 3.1 Collection of Training and Validation Data from CG Data 3.2 Collection of Test Data 3.3 Skeleton Estimation with MediaPipe 3.4 Sign Language Classification Using ST-GCN 4 Experiments 4.1 Data Acquisition 4.2 Skeleton Estimation 5 Result 6 Conclusion References Gesture Elicitation for Augmented Reality Environments 1 Introduction 2 Gestures and AR 3 Theoretical Foundation 3.1 History of Gestures 3.2 Gesture Elicitation 3.3 Concepts and Terminology 4 Related Work 5 Our Methodology for Gesture Elicitation in RA Environments 5.1 Test Scenario 5.2 Questionnaire for the Gesture Elicitation Process 6 Data Collection 7 Data Analysis 8 The Gestures Elicited 9 Discussion and Future Work References A Systematic Procedure for Comparing Template-Based Gesture Recognizers 1 Introduction 2 Related Work 3 Evaluation 3.1 Experiment 3.2 Procedure and Measures 4 Results and Discussion 4.1 SHREC2019 Dataset 4.2 Jackknife-LM Dataset 5 Conclusion A The New Recognizers Considered in the Experiment A.1 $P3+X Recognizer A.2 PennyPincher3D Recognizer B The Datasets Considered in the Experiment B.1 SHREC2019 ch13Caputo:2019 B.2 Jackknife-LM ch13Taranta:2017 C Recognition Rates C.1 Recognition Rate for Best Condition for the SHREC2019 C.2 Recognition Rates Tables for the SHREC2019 C.3 Recognition Rates Tables for the Jackknife-LM C.4 The Ranking of the Recognizers Based on the Best Individual Rates by Articulation for SHREC2019 References An Elderly User-Defined Gesture Set for Audio Natural Interaction in Square Dance 1 Introduction 2 Related Work 2.1 Age-Appropriate Gesture Interaction Study 2.2 Gesture Elicitation Study 2.3 Summary 3 Experiment 1: Gesture Elicitation 3.1 Requirement Analysis and Task Definition 3.2 User-Defined Gesture Naming Principle 3.3 Gesture Interaction Video Preparation 3.4 Preliminary Experiments 3.5 Formal Experiment 4 Experiment 2: Gesture Vocabulary Evaluation 4.1 Purpose 4.2 Preparation 4.3 Subjects 4.4 Procedures and Methods 4.5 Experimental Instructions 4.6 Experimental Results 5 Discussion 6 Conclusion References Using Fiducial Marker for Analyzing Wearable Eye-Tracker Gaze Data Measured While Cooking 1 Introduction 2 Related Works 3 AOI Estimation Method Using Multiple Fiducial Markers 3.1 Inputting Target Object Data 3.2 Collecting Eye Gaze Data 3.3 Detecting Markers 3.4 Recognizing Objects 3.5 Identifying AOI 4 Evaluation 4.1 Eye Gaze Data Collection 4.2 Comparison with Markerless AOI Estimation Method 4.3 Applicability to Different Kitchens and Recipes 5 Discussion 6 Conclusion References Using Wearable Devices for Emotion Recognition in Mobile Human- Computer Interaction: A Review 1 Introduction 2 Background 2.1 Emotion Models 2.2 Related Work 3 Review Methodology 3.1 Data Sources and Search Strategy 3.2 Study Selection and Quality Assessment 3.3 Data Extraction and Data Synthesis 4 Results 4.1 Wearables and Physiological Signals 4.2 Specific Applications 4.3 Experimental Environment 4.4 Pre-experimental Sessions 4.5 Emotionally Stimulating Material 4.6 Emotion Classification 4.7 Baseline for Assessment 5 Discussion 6 Conclusions References Human-Robot Interaction Human-Robot-Collaboration in the Healthcare Environment: An Exploratory Study 1 Introduction 2 Method 2.1 Participants 2.2 Study Design and Materials 2.3 Statistical Analysis 3 Results 3.1 General Attitudes Towards Robots 3.2 Acceptance Regarding Different Robots 4 Discussion References Towards an Active Predictive Relation by Reconceptualizing a Vacuum Robot: Research on the Transparency and Acceptance of the Predictive Behaviors 1 Introduction 1.1 Predictive Relations & Knowledge 1.2 Current Issues and Related Work of AI & the Connected Objects 1.3 Research Direction 2 Methods 2.1 Sensitizing the Expression of Participants: Path of Expression 2.2 Sensitizing the Findings: Domestication Theory 3 From the Current Vacuum Robot to the Predicting Vacuum Robot 3.1 The Main Tasks of the Current Vacuum Robot 3.2 Categories of Predictive Behaviors 4 From Vacuum Robots to General Predicting Things 4.1 Practical Dimension 4.2 Symbolic Dimension 4.3 Cognitive Dimension 4.4 Social Dimension 5 Discussion 5.1 Design Qualities for the Acceptance of the Predicting Things 5.2 Model of Generating Predictive Behaviors that Enhances the Transparency 5.3 The Shifting Role of the Designer When Things Become Predictive: Designers as Facilitators of Human-Robot Collaboration 5.4 The Creative Session as Speculative Trigger to Open up the Discussion About the Future Where the Things Become Predictive 6 Conclusion References On Improving the Acceptance of Intelligent Companion Robots Among Chinese Empty-Nesters with the Application of Emotional Design 1 Introduction 2 Literature Review 2.1 Characteristics of Empty Nesters in China 2.2 User-Centered Design Thinking and Methods 2.3 Contextual Inquiry in User Study 2.4 Demand Analysis 3 Method 3.1 User Study 3.2 Insight 3.3 Want & Need Analysis 4 Results 4.1 User Research Interpreting Basic Ideas and Conclusions 4.2 W&Ns Functional Attribute and Importance Rankings 5 Conclusions and Suggestions References Human Interpretation of Inter-robot Communication 1 Introduction 2 Method 2.1 Robot 2.2 Development 3 Experiment 3.1 Motion Patterns 3.2 Experiment Procedure 4 Results 5 Conclusion References Conversational Agents and Robot Interaction 1 Introduction 2 Conversational Human-Robot Interaction 2.1 Spoken Communication with Robot Agents 2.2 Multimodal Communication 3 Constructing Situational Awareness 3.1 Grounding 3.2 Construction of Dialogues 3.3 Situational Awareness 4 Dialogue Management 4.1 Rasa Dialogue Modelling 4.2 Example Dialogue 5 Conclusions References Surgical Human-Robot Interaction: A Bibliometric Review 1 Introduction 2 Methods 2.1 Data Collection 2.2 Data Analysis 3 Results 3.1 Statistical Analyses 3.2 Trend Analyses 3.3 Co-citation Analysis 3.4 Content Analysis 4 Discussion 5 Conclusion References Human Intention Recognition for Safe Robot Action Planning Using Head Pose 1 Introduction 2 System Design 2.1 Head Pose Estimation 2.2 InHARD Dataset 2.3 Human Intention Recognition 2.4 Simulation and Visualization Environment 3 Results 3.1 Intention Recognition 3.2 Human Action Recognition 3.3 Overall System Results 4 Conclusions and Discussion References Mobile Manipulator for Hospital Care Using Firebase 1 Introduction 2 Methods and Materials 2.1 Planning 2.2 Design 2.3 Development 3 Results 3.1 Evaluation 3.2 Compliance with System Parameters 3.3 Usability Test 3.4 Discussion 4 Conclusions References Iterative Design Process for HRI: Serving Robot in Restaurant 1 Introduction 2 Iterative Design Process for HRI 3 User Test 3.1 Participants 3.2 Weekly Procedure 4 Results 4.1 Awkward Moments in Video Ethnography 4.2 Key Interaction Requirements 5 Limitation 6 Conclusion References Brain-Computer Interfaces Methodology Design of the Correlation Between EEG Signals and Brain Regions Mapping in Panic Attacks 1 Introduction 2 Literary Review 2.1 Lean UX Methodology 2.2 Brain Computer Interfaces (BCI) 2.3 Mood Disorders: Anxiety Cluster 3 Related Work 4 Methodological Proposal 4.1 Segmentation 4.2 Design 4.3 Treatment 4.4 Launching 4.5 Evaluation 5 MVP Design Guidelines 5.1 Segmentation 5.2 Design 5.3 Treatment 5.4 Launching 5.5 Evaluation 6 Conclusions and Future Work References Establishing Clinical Protocols for BCI-Based Motor Rehabilitation in Individuals Post Stroke - The Impact of Feedback Type and Selected Outcome Measures: A Systematic Review 1 Literature Review 1.1 Stroke 1.2 Brain Computer Interface 1.3 Feedback Provided During BCI Interventions 1.4 Measurable Outcomes as a Result of BCI Interventions 2 Study Aims 3 Methods 3.1 Search Strategy 3.2 Selection Criteria 3.3 Methodological Quality Assessment 4 Results 4.1 Analysis of Outcome Measures Analyzed 4.2 Analysis of Feedback Provided 4.3 Analysis of Subject Demographics 4.4 Analysis of Randomized Controlled Trial Recommendations 5 Discussion 6 Recommendations References Training CNN to Detect Motor Imagery in ECoG Data Recorded During Dreaming 1 Introduction 2 Materials and Methods 2.1 Motor Data 2.2 REM Sleep Data 2.3 Data Pre-processing 2.4 Overlap Metric 2.5 Pre-training Feature Extraction 2.6 Spatial Transformation 2.7 CNN Model Architecture 2.8 Model Evaluation 2.9 Data and Software Availability 2.10 Ethics Statement 3 Results and Discussion 3.1 Limitations 4 Conclusion References Time Majority Voting, a PC-Based EEG Classifier for Non-expert Users 1 Introduction 2 Algorithms 2.1 Existing Algorithms 2.2 Our New Algorithms 3 Experiments 3.1 Data Preprocess 4 Results 4.1 Existing Algorithms 4.2 Identify Noisy Sessions 4.3 Time Majority Voting 5 Discussion 5.1 Accuracy and Data Remain 5.2 Runtime and Training Data 5.3 Interpretability 6 Conclusion References It's Easy as ABC Framework for User Feedback 1 Introduction 1.1 Problem Statement 1.2 Literature Review 1.3 Purpose of Study 1.4 Research Questions 2 Experiment 2.1 Experiment Design 3 Result 4 Discussion 4.1 Limitations 4.2 Future Work 5 Conclusion References Single-Subject vs. Cross-Subject Motor Imagery Models 1 Introduction 1.1 Problem Statement 1.2 Research Question 1.3 Literature Review 2 Method 2.1 Data 2.2 Experiment 3 Results 4 Discussion 4.1 Algorithms 4.2 Single Subject versus Cross Subject Models 4.3 Further Research 5 Conclusion A Appendix References Hybrid Convolutional, Recurrent and Attention-Based Architectures of Deep Neural Networks for Classification of Human-Computer Interaction by Electroencephalography 1 Introduction 2 Background and Related Work 3 Methodology 3.1 Dataset 3.2 Models 3.3 Metrics 3.4 Workflow 4 Experiment 4.1 Exploratory Data Analysis (EDA) 4.2 DNN Training/Validation/Testing Stages 5 Discussion 6 Conclusions References Predicting the Future: A ML MI Replication Study 1 Introduction 1.1 Literature Review 1.2 Background 2 Dataset 3 Experiment Design 3.1 Algorithm Comparison 3.2 Experiment Design 4 Baselines 5 Results 5.1 Typical Algorithm Performance 5.2 CNN/DBN Performance 6 Discussion 7 Conclusion References High-Powered Ocular Artifact Detection with C-LSTM-E 1 Introduction 2 Literature Review 2.1 CNN 2.2 LSTM 2.3 Power Reduction 3 Dataset 4 Methods 4.1 Rationale 4.2 Model Architecture 4.3 Data Processing 4.4 Model Training 5 Results 5.1 Variable Prevalence of OAs 5.2 Fixed Prevalence of OAs 5.3 Real-Time Processing 6 Discussion 6.1 Analysis 6.2 Next Steps 7 Conclusion A Appendix References ML vs DL: Accuracy and Testing Runtime Trade-offs in BCI 1 Introduction 1.1 Problem Statement 1.2 Literature Review 1.3 Purpose 1.4 Research Questions 2 Dataset Description 3 Methods 4 Results 5 Discussion 5.1 Trade-offs of Baseline ML Algorithms 5.2 Trade-offs of a Tuned SVM 5.3 Trade-offs of DL Algorithms 5.4 Testing Runtime and Tree-Based Models 5.5 Time Complexity of ML Algorithms 5.6 Limitations and Future Directions 6 Conclusion References CNN with Self-attention in EEG Classification 1 Introduction 1.1 Problem Statement 1.2 Purpose of Study 1.3 Research Questions 2 Dataset 3 Method 3.1 Rationale 3.2 Implementation 3.3 Innovation 3.4 Connection 4 Results 5 Discussion and Future Work 5.1 Comparison with Benchmarks 5.2 Robustness Check 5.3 Future Work 6 Conclusion References Design Thinking the Human-AI Experience of Neurotechnology for Knowledge Workers 1 Introduction 2 Theoretical Background 2.1 Cognitive Enhancement 2.2 Healthy Flow Performance 2.3 Neurofeedback 3 Design Thinking the Human-AI Experience 3.1 Stage 1 - Empathize 3.2 Stage 2 - Define 3.3 Stage 3 - Ideate 3.4 Stage 4 - Prototype 3.5 Stage 5 - Test 4 Evaluation 4.1 Methodology 5 Results 6 Discussion 7 Conclusion References Optimizing ML Algorithms Under CSP and Riemannian Covariance in MI-BCIs 1 Introduction 1.1 Literature Review 1.2 Problem Statement 1.3 Research Questions 2 Methods 2.1 Dataset Source 2.2 Dataset Description 2.3 Experiment Design 3 Results 4 Conclusion 5 Discussion References Author Index