MultiMedia Modeling: 28th International Conference, MMM 2022, Phu Quoc, Vietnam, June 6–10, 2022, Proceedings, Part I (Lecture Notes in Computer Science, 13141)

The two-volume set LNCS 13141 and LNCS 13142 constitutes the proceedings of the 28th International Conference on MultiMedia Modeling, MMM 2022, which took place in Phu Quoc, Vietnam, during June 6–10, 2022. The 107 papers presented in these proceedings were carefully reviewed and selected from a total of 212 submissions. They focus on topics related to multimedia content analysis; multimedia signal processing and communications; and multimedia applications and services. Preface Organization Contents – Part I Contents – Part II Best Paper Session Real-time Detection of Tiny Objects Based on a Weighted Bi-directional FPN 1 Introduction 2 Related Work 2.1 Tiny Object Detection 2.2 Multiscale Feature Fusion 3 The Proposed Model 3.1 YOLO-P4 3.2 YOLO-BiP4 4 Experiments 4.1 Evaluation Methods and Datasets 4.2 Experimental Environment and Parameter Description 4.3 Comparison of Improved Detection Results 5 Conclusions and Future Works References Multi-modal Fusion Network for Rumor Detection with Texts and Images 1 Introduction 2 Related Work 3 Approach 3.1 Overview 3.2 Granularity-Fusion Module 3.3 Knowledge-Aware Module 3.4 Classification Module 4 Experimentation 4.1 Experimental Settings 4.2 Experimental Results 4.3 Ablation 4.4 Case Study 5 Conclusion References PF-VTON: Toward High-Quality Parser-Free Virtual Try-On Network 1 Introduction 2 Related Works 2.1 Fashion Analysis and Image Synthesis 2.2 Transformer 2.3 Virtual Try-On 3 PF-VTON 3.1 Teacher Network 3.2 Student Network 4 Experiments 4.1 Dataset 4.2 Implementation Details 4.3 Qualitative Results 4.4 Quantitative Results 5 Conclusion References MF-GAN: Multi-conditional Fusion Generative Adversarial Network for Text-to-Image Synthesis 1 Introduction 2 Related Work 3 Method 3.1 Overall MF-GAN Architecture 3.2 Conditional Residual Block 3.3 Dual Attention Block 3.4 Triplet Loss 4 Experiment 4.1 Datasets and Training Details 4.2 Evaluation Metrics 4.3 Quantitative Results 4.4 Qualitative Results 4.5 Ablation Study 5 Conclusion References Applications 1 Learning to Classify Weather Conditions from Single Images Without Labels 1 Introduction 2 Related Works 2.1 Weather Classification 2.2 Unsupervised Learning 3 Method 3.1 Overview 3.2 Weather Feature Learning 3.3 Weather Clustering 4 Experiments 4.1 Datasets and Evaluation Metrics 4.2 Implementation Details 4.3 Experimental Results 4.4 Ablation Study 4.5 Parameter Sensitivity 5 Conclusion References Learning Image Representation via Attribute-Aware Attention Networks for Fashion Classification 1 Introduction 2 Related Work 3 Methodology 3.1 Problem Formulation 3.2 Network Structure 3.3 Label Mask Training 3.4 Triplet Network Training 4 Experiments 4.1 Experiments Settings 4.2 Experiment Results 4.3 Attention Visualization 4.4 Impact of Joint Learning and the Pooling Methods 4.5 A Case: Attribute-Specific Image Retrieval 5 Conclusions References Toward Detail-Oriented Image-Based Virtual Try-On with Arbitrary Poses 1 Introduction 2 Related Works 2.1 Generative Adversarial Networks 2.2 Person Image Generation 2.3 Virtual Try-On 3 DO-VTON 3.1 Multi-scale Dilated Convolution U-Net 3.2 Semantic Prediction Module 3.3 Spatial Alignment Module 3.4 Try-On Synthesis Module 4 Experiments 4.1 Datasets and Experimental Setup 4.2 Qualitative Results 4.3 Quantitative Results 5 Conclusion References Parallel DBSCAN-Martingale Estimation of the Number of Concepts for Automatic Satellite Image Clustering 1 Introduction 2 Related Work 3 Background and Methodology 3.1 Background and Notation 3.2 The Proposed Parallel-DBSCAN-Martingale 3.3 HPC Infrastructure 4 Evaluation 4.1 Dataset Description 4.2 Results 4.3 Cost-Benefit Analysis 5 Conclusion and Future Work References Multimedia Applications - Perspectives, Tools and Applications (Special Session) and Brave New Ideas AI for the Media Industry: Application Potential and Automation Levels 1 Introduction 1.1 Definitions 1.2 Related Work 2 Assessing Application Areas 3 Automation Levels in the Media Domain 4 Conclusion References Rating-Aware Self-Organizing Maps 1 Introduction 2 Related Work 3 Rating-Aware Self-Organizing Maps 3.1 Probabilistic Candidates Selection Procedure 3.2 Rating-Aware Extension for BMN Selection 4 Evaluation 4.1 Evaluation Protocol 4.2 Results 5 Discussion and Conclusions References Color the Word: Leveraging Web Images for Machine Translation of Untranslatable Words 1 Introduction 2 Motivation and Background 2.1 Multimedia Analysis 2.2 Other Research Fields 3 Experimental Investigation 3.1 Dataset 3.2 Survey and Expert Ground truth 3.3 Results 4 Discussion and Outlook 4.1 Images on the Web 4.2 Towards the Future of Machine Translation References Activities and Events MGMP: Multimodal Graph Message Propagation Network for Event Detection 1 Introduction 2 Related Work 3 Methodology 3.1 Multimodal Graph Construction 3.2 Graph Message Propagation 3.3 Event Classification 4 Experiments 4.1 Datasets 4.2 Implementation Details 4.3 Baselines 4.4 Comparison Results 4.5 Ablation Study 4.6 Case Study 5 Conclusion References Pose-Enhanced Relation Feature for Action Recognition in Still Images 1 Introduction 2 Related Work 2.1 Action Recognition in Still Images 2.2 Pose Estimation 3 Approach 3.1 Framework Overview 3.2 Pose Regions Generation 3.3 Pose-Enhanced Relation Module 3.4 Score Fusion and Loss Function 4 Experiments 4.1 Experiment Settings 4.2 Comparison with Other Methods 5 Conclusion References Prostate Segmentation of Ultrasound Images Based on Interpretable-Guided Mathematical Model 1 Introduction 2 Related Work 3 Model 3.1 Problem Statement 3.2 Model Overview 3.3 Principal Curve Based CCPS 3.4 Differential Evolution Based IAMCDE 3.5 Prostate Contour’s Interpretable Mathematical Model Denoted by the Parameters of ABPNN 4 Experiments 4.1 Experimental Setups 4.2 Comparison with Hybrid Models 4.3 Comparison with Other State-Of-The-Art Models 5 Discussion and Conclusion References Spatiotemporal Perturbation Based Dynamic Consistency for Semi-supervised Temporal Action Detection 1 Introduction 2 Related Work 3 Method 3.1 Overview 3.2 Network Structure 3.3 Spatiotemporal Feature Perturbation 3.4 Dynamic Consistency 3.5 Overall loss 4 Experiments 4.1 Datasets and Setup 4.2 Comparisons with Start-of-the-Art Methods 4.3 Ablation Study 5 Conclusion References Multimedia Datasets for Repeatable Experimentation (Special Session) A Task Category Space for User-Centric Comparative Multimedia Search Evaluations 1 Introduction 2 Related Work and Background 3 Task Category Space for User-Centric Video Search 4 Challenges of Task Categories Considered so far 4.1 Visual KIS Task 4.2 Textual KIS Task 4.3 Ad-hoc Video Search Tasks 5 Conclusions References GPR1200: A Benchmark for General-Purpose Content-Based Image Retrieval 1 Introduction 2 Related Work 3 Analysis of Available Datasets 4 General-Purpose Retrieval Benchmark 4.1 GPR1200 Evaluation Protocol 5 Experiments 5.1 Evaluation of Pretrained Models 5.2 Training for General-Purpose CBIR 6 Conclusion References LLQA - Lifelog Question Answering Dataset 1 Introduction 2 Related Work 2.1 Lifelogs and Personal Data Analytics 2.2 Video Question Answering (Video QA) 3 LLQA - A Lifelog Question Answering Dataset 3.1 Data Collection 3.2 Generation of Question and Answers 3.3 Review 4 Pilot Experiment 4.1 Human Gold-Standard Baseline 4.2 Question-Only 4.3 Question and Vision 4.4 Results 5 Conclusion References Learning Category-Sensitive Incremental Learning for Image-Based 3D Shape Reconstruction 1 Introduction 2 Related Work 2.1 Category-Wise 3D Reconstruction 2.2 Category-Agnostic 3D Reconstruction 2.3 Incremental Learning 3 Incremental 3D Reconstruction 3.1 Network Overview 3.2 Incremental Embedding Optimizing via Contrastive Learning 3.3 Energy-Based Classification 3.4 Reducing Forgetting 3.5 Implementation Details 4 Experiments and Results 4.1 Experiment Settings 4.2 Incremental 3D Reconstruction Results 4.3 Ablation Study 5 Conclusion References AdaConfigure: Reinforcement Learning-Based Adaptive Configuration for Video Analytics Services 1 Introduction 2 Related Works 2.1 Static Configuration Optimization 2.2 Dynamic Configuration Optimization 3 Reinforcement Learning-Based Adaptive Configuration 3.1 Problem Formulation 3.2 RL Agent Design and Video Characteristic Extraction 3.3 Reinforcement Learning-Based Framework 3.4 Reward Feedback 4 Evaluation 4.1 Experiment Setup 4.2 Datasets and Metric 4.3 Experiment Parameters 4.4 Experiment Results 5 Conclusion References Mining Minority-Class Examples with Uncertainty Estimates 1 Introduction 2 Related Works 3 Our Method 4 Evaluation 4.1 Datasets and Setup 4.2 Experiments 5 Conclusion References Conditional Context-Aware Feature Alignment for Domain Adaptive Detection Transformer 1 Introduction 2 Related Works 2.1 Object Detection 2.2 Unsupervised Domain Adaptation 3 Methodology 3.1 Preliminaries on Detection Transformer 3.2 Context-Aware Feature Alignment 3.3 Conditional Feature Alignment 3.4 Total Loss 4 Experiments 4.1 Datasets 4.2 Implementation Details 4.3 Main Results 4.4 Ablation Study 4.5 Further Analysis 5 Conclusion References Multimedia for Medical Applications (Special Session) Human Activity Recognition with IMU and Vital Signs Feature Fusion 1 Introduction 2 Related Work 3 Methods 4 Results 4.1 Dataset 4.2 Fusion Results 4.3 Feature Selection Results 5 Conclusion References On Assisting Diagnoses of Pareidolia by Emulating Patient Behavior 1 Introduction 2 Related Work 3 Proposed Method 3.1 Emulation of Patient Behavior 3.2 Identification of Patient Type 4 Evaluation 4.1 Experimental Setup 4.2 Comparison Methods and Ablation Studies 4.3 Results 4.4 Discussion 5 Conclusion References Using Explainable AI to Identify Differences Between Clinical and Experimental Pain Detection Models Based on Facial Expressions 1 Introduction 2 Related Work 2.1 Cross-database Evaluation of Pain Datasets 2.2 Explainable Artificial Intelligence 3 Approach 3.1 Datasets 3.2 Pain Training 3.3 Cross-database Evaluation 3.4 Visual Analysis 3.5 Concept Embedding Analysis 4 Results 5 Discussion 6 Conclusion References Applications 2 Double Granularity Relation Network with Self-criticism for Occluded Person Re-identification 1 Introduction 2 Related Work 2.1 Part-Based Scheme 2.2 External Cues 2.3 Relation Learning 3 Proposed Approach 3.1 Architecture of the Proposed Model 3.2 Double Granularity Relation 3.3 Relation Critic Mechanism 3.4 Optimization 4 Experiments 4.1 Implementation and Training Details 4.2 Experimental Results 5 Analyses 5.1 Varying Trade-Off Coefficients 5.2 The Contribution of Each Component 5.3 Visualization 6 Conclusion References A Complementary Fusion Strategy for RGB-D Face Recognition 1 Introduction 2 Related Work 2.1 CNN-Based Multi-modal Feature Learning in RGB-D FR 2.2 Transformers in Vision 2.3 Transformers for Multi-modal Tasks 3 Facial Depth Map Enhancement 4 Complementary Multi-modal Fusion Transformer 4.1 Overall Architecture 4.2 LE-Token: Locally Enhanced Tokenization 4.3 MISA: Mutually Interacted Self-attention Module 4.4 Classification Layer 5 SmartToFFace Dataset 6 Experimental Analysis 6.1 Public Dataset Description 6.2 Implementation Details 6.3 Performance on Public Datasets 6.4 Comparison to Other Cross Modal Self-attention Mechanisms 6.5 Ablation Study to the Depth Encoding Algorithm 7 Conclusion References Multi-scale Cross-Modal Transformer Network for RGB-D Object Detection 1 Introduction 2 Related Work 2.1 Depth Feature Extraction 2.2 Multi-modal Feature Fusion 3 MCTNet 3.1 Network Overview 3.2 Multi-modal Feature Pyramid (MFP) 3.3 Cross-Modal Transformer (CMTrans) 3.4 Loss Function 4 Experimental Results 4.1 Implementation Details 4.2 Results on SUN RGB-D and NYU Depth V2 4.3 Ablation Studies 5 Conclusion References Joint Re-Detection and Re-Identification for Multi-Object Tracking 1 Introduction 2 Related Work 2.1 Tracking-by-Detection 2.2 Appearance Models and ReID 2.3 Non-Maximum Suppression 3 Proposed Method 3.1 Overview 3.2 Trajectory Re-Detection Network 3.3 ReID-NMS 4 Experiments 4.1 Implementation Details 4.2 Datasets and Metrics 4.3 Evaluation Results on Test Set 4.4 Ablation Studies 5 Conclusion References Multimedia Analytics for Contextual Human Understanding (Special Session) An Investigation into Keystroke Dynamics and Heart Rate Variability as Indicators of Stress 1 Introduction 2 Literature Review 3 Heart Rate Variability (HRV) 4 Data Gathered 5 Data Preparation 6 Data Analysis and Results 7 Conclusions References Fall Detection Using Multimodal Data 1 Introduction 2 Related Work 3 Methodology 3.1 UP-Fall Detection Dataset 3.2 Data Processing 3.3 Feature Extraction and Modeling 3.4 Performance Metrics 4 Experiments 4.1 Data Collection 4.2 Previous Methods 4.3 Experimental Results 5 Conclusion and Future Works References Prediction of Blood Glucose Using Contextual LifeLog Data 1 Introduction 2 Background and Related Research 3 Dataset 4 Data Processing 4.1 Data Understanding and Cleaning 4.2 Carbohydrate Estimation 4.3 Additional Feature Extraction 5 Prediction Methods 5.1 State-of-the-Art Prediction Methods 5.2 Evaluation 6 Results Analysis and Discussion 7 Conclusion and Future Work References Multimodal Embedding for Lifelog Retrieval 1 Introduction 2 Related Work 2.1 Feature Learning 2.2 Deep Learning in Text-Image Retrieval 3 Text Embedding for Semantic Retrieval Based on Captions 3.1 Object Detection 3.2 Captioning Model 3.3 Semantic Similarity 4 Joint Embedding for Cross-Modal Retrieval in Lifelog Retrieval 4.1 Visual Representation 4.2 Unsupervised Cross-modal Common Vector Space Learning 5 Experimental Analysis 5.1 Baseline 5.2 Setting 1 - Text Embedding 5.3 Setting 2 - Visual-Semantic Embedding 5.4 Discussion and Contribution 6 Conclusion References Applications 3 A Multiple Positives Enhanced NCE Loss for Image-Text Retrieval 1 Introduction 2 Related Works 3 Method 3.1 Problem Definition and Notations 3.2 Noise Contrastive Estimation: A Revisit 3.3 Multiple Positive Enhanced NCE: Our Approach 4 Experiment 4.1 Experimental Settings 4.2 Comparison to Prior State-of-the-Arts 4.3 Ablation Studies 4.4 Qualitative Analysis 5 Conclusion References SAM: Self Attention Mechanism for Scene Text Recognition Based on Swin Transformer 1 Introduction 2 Related Work 3 Methodology 3.1 Encoder 3.2 Decoder 4 Experiments 4.1 Benchmarks 4.2 Implementation Details 4.3 Performance on Benchmarks 5 Conclusion References JVCSR: Video Compressive Sensing Reconstruction with Joint In-Loop Reference Enhancement and Out-Loop Super-Resolution 1 Introduction 2 Related Technologies 3 Proposed JVCSR 3.1 Overall Framework 3.2 Network Architectures 4 Experiments 4.1 Experimental Settings 4.2 Experimental Results 5 Conclusions References Point Cloud Upsampling via a Coarse-to-Fine Network 1 Introduction 2 Related Work 2.1 Optimization-Based Upsampling. 2.2 Deep Learning-Based Upsampling 3 Proposed Approach 3.1 Overview 3.2 Global Dense Reconstruction 3.3 Local Refinement 4 Experiments 4.1 Experimental Settings 4.2 Results on PU1K Dataset 4.3 Ablation Study 5 Conclusions References Image Analytics Arbitrary Style Transfer with Adaptive Channel Network 1 Introduction 2 Related Work 3 Our Method 3.1 Network Architecture 3.2 Loss Function 4 Experiments 4.1 Experiments Setting 4.2 Comparison with Other Methods 4.3 Ablation Experiments 5 Conclusion References Fast Single Image Dehazing Using Morphological Reconstruction and Saturation Compensation 1 Introduction 2 Preliminaries 2.1 Hazy Image Formation Model 2.2 Dark Channel Prior 2.3 Morphological Reconstruction 3 The Proposed Method 3.1 Air-Light Estimation 3.2 Transmission Estimation 3.3 Image Dehazing 4 Experiments 4.1 Experiments with Synthetic Data 4.2 Experiments with Natural Images 5 Conclusion References One-Stage Image Inpainting with Hybrid Attention 1 Introduction 2 Related Works 2.1 Image Inpainting 2.2 Channel Attention Mechanism 3 Approach 3.1 Overall Architecture 3.2 Hybrid Attention Module 3.3 Loss Functions 4 Experiments 5 Results 5.1 Qualitative Comparisons 5.2 Quantitative Comparisons 5.3 User Study 5.4 Ablation Studies 6 Conclusion References Real-Time FPGA Design for OMP Targeting 8K Image Reconstruction 1 Introduction 2 Background 2.1 Compressed Sensing (CS) 2.2 Orthogonal Matching Pursuit (OMP) 3 Proposals 3.1 Chosen Matrices and Features of Sensing Matrix 3.2 Optimized Dot Product 3.3 Optimized Least Square Problem 3.4 Overall Framework 4 Experiments 4.1 Design Details 4.2 Design Performance 5 Conclusion References Speech and Music Time-Frequency Attention for Speech Emotion Recognition with Squeeze-and-Excitation Blocks 1 Introduction 2 Related Work 3 Proposed Method 3.1 Data Preprocessing 3.2 Time-Frequency Attention 3.3 Squeeze-and-Excitation Block 4 Results and Discussion 4.1 Dataset 4.2 Experiment Setup 4.3 Comparison with Sate-of-the-art Methods 4.4 Ablation Study 4.5 Average Class Accuracy 5 Conclusion References Speech Intelligibility Enhancement By Non-Parallel Speech Style Conversion Using CWT and iMetricGAN Based CycleGAN 1 Introduction 2 Baseline: Non-Parallel SSC Framework 2.1 Basic Framework 2.2 CycleGAN 3 Proposed Method: Non-Parallel SSC Using CWT and iMetricGAN Based CycleGAN 3.1 Framework 3.2 Block: Continuous Wavelet Transform (CWT) 3.3 Block: iMetricGAN 4 Experiments 4.1 Experimental Setup 4.2 Property Performance 4.3 Overall Experiment 5 Conclusion References A-Muze-Net: Music Generation by Composing the Harmony Based on the Generated Melody 1 Introduction 2 Related Work 3 Method 3.1 Midi Representation 3.2 Models 3.3 A-Muze-Net Model Training 4 Experiments 4.1 Results 4.2 User Study 5 Conclusions References Melody Generation from Lyrics Using Three Branch Conditional LSTM-GAN 1 Introduction 2 Related Works 2.1 Music Generation 2.2 Discrete Data Generation Using GANs 3 Methodology 3.1 GANs 3.2 Problem Formulation 3.3 Gumbel-Softmax 3.4 Generator 3.5 Discriminator 4 Experiments and Evaluation 4.1 Experimental Setup 4.2 Quality and Diversity Evaluation 4.3 Music Quantitative Evaluation 4.4 Subjective Evaluation 5 Conclusion References Multimodal Analytics Bi-attention Modal Separation Network for Multimodal Video Fusion 1 Introduction 2 Related Work 2.1 Multimodal Video Fusion 2.2 Bi-attention Transformer 3 Our Approach 3.1 Network Architecture 3.2 Feature Extraction Module 3.3 Modal Fusion Module 3.4 Modal Separation Module 3.5 Learning 4 Experiments 4.1 Datasets 4.2 Features 4.3 Baselines 4.4 Implementation Details 5 Results and Discussion 5.1 Summary of Results 5.2 Ablation Study 6 Future Works References Combining Knowledge and Multi-modal Fusion for Meme Classification 1 Introduction 2 Related Work 2.1 Multi-modal Fusion 2.2 Meme Classification 3 Proposed Methodology 3.1 Concept and Text Encoder 3.2 Concept-Image Attention 3.3 Fusion Module 4 Experiments and Analysis 4.1 Dataset Description 4.2 Implementation Details 4.3 Competitors 4.4 Experimental Results and Analysis 5 Conclusions References Non-Uniform Attention Network for Multi-modal Sentiment Analysis 1 Introduction 2 Related Work 3 Methods 3.1 Long Short-Term Memory 3.2 Attention Mechanism in Multi-modal Sentiment Analysis 3.3 Non-uniform Attention Module 3.4 Multi Attention Mechanism 4 Experiments 4.1 Databases and Competitors 4.2 Experimental Setup 4.3 Experimental Results 4.4 Ablation Studies 5 Conclusions References Multimodal Unsupervised Image-to-Image Translation Without Independent Style Encoder 1 Introduction 2 Related Work 2.1 Image-to-image Translation 2.2 Normalization 3 Method 3.1 General Formulation 3.2 Network Architecture 3.3 Adaptive Layer-Instance-Group Normalization 3.4 Loss Function 4 Experiments 4.1 Experimental Settings 4.2 Qualitative Comparison 4.3 Quantitative Comparison 4.4 Ablation Study 5 Conclusion References Correction to: Real-Time FPGA Design for OMP Targeting 8K Image Reconstruction Correction to: Chapter 41 in: B. Þór Jónsson et al. (Eds.): MultiMedia Modeling, LNCS 13141, https://doi.org/10.1007/978-3-030-98358-1_41 Author Index