MultiMedia Modeling: 30th International Conference, MMM 2024, Amsterdam, The Netherlands, January 29 February 2, 2024, Proceedings, Part I (Lecture Notes in Computer Science, 14554)

This book constitutes the refereed proceedings of the 30th International Conference on MultiMedia Modeling, MMM 2024, held in Amsterdam, The Netherlands, during January 29 – February 2, 2024.The 112 full papers included in this volume were carefully reviewed and selected from 297 submissions. The MMM conference were organized in topics related to multimedia modelling, particularly: audio, image, video processing, coding and compression; multimodal analysis for retrieval applications, and multimedia fusion methods. Preface Organization Contents – Part I Where Are Biases? Adversarial Debiasing with Spurious Feature Visualization 1 Introduction 2 Methods 2.1 Architecture 2.2 GroupDRO 2.3 Training Procedure 3 Datasets and Experimental Settings 3.1 Datasets 3.2 Metrics 3.3 Architecture Implementation 4 Results 4.1 BiasCheXpert-Gender 4.2 BiasCheXpert-Race 5 Limitations 6 Conclusion References Cross-Modal Hash Retrieval with Category Semantics 1 Introduction 2 Related Work 3 Methods 3.1 Problem Definition 3.2 Network Architecture 3.3 Hash Objectives 4 Experiment 4.1 Experimental Setting 4.2 Experimental Results 5 Conclusion References Spatiotemporal Representation Enhanced ViT for Video Recognition 1 Introduction 2 Related Work 3 Method 3.1 Overview of STRE-ViT 3.2 Spatial Feature Extraction Stream 3.3 Spatiotemporal Interaction Stream 3.4 Feature Fusion Strategy 4 Experiments 4.1 Main Results 4.2 Ablation Studies 5 Conclusion References SCFormer: A Vision Transformer with Split Channel in Sitting Posture Recognition 1 Introduction 2 Related Work 2.1 Sitting Posture Recognition 2.2 Vision Transformer 3 Methodology 3.1 Overall Structure Design 3.2 Regular Split Channel 3.3 SCFormer Block 4 Experiments 4.1 Experimental Dataset 4.2 Experimental Environment 4.3 Comparative Experiment on Different Network Models 4.4 Ablation Experiments 5 Conclusion References Dive into Coarse-to-Fine Strategy in Single Image Deblurring 1 Introduction 2 Related Work 2.1 Image Deblurring by Coarse-to-fine Approach 2.2 One-Encoder-Multiple-Decoder Structure 3 Proposed Method 3.1 Architecture of the Proposed ADMMDeblur 3.2 Four Separated Decoders 3.3 Spatial Kernel Rotation for Parameter Sharing 3.4 Loss Function 4 Experiment 4.1 Datasets and Implementation Details 4.2 Performance Comparison 4.3 Ablation Study 5 Conclusion References TICondition: Expanding Control Capabilities for Text-to-Image Generation with Multi-Modal Conditions 1 Introduction 2 Related Work 2.1 Text-to-Image Generation 2.2 Subject-Driven Text-to-Image Generation 2.3 Controllable Diffusion Models 3 Method 3.1 Paired Data Preparation 3.2 Model Designs 4 Experiment 4.1 Implementation and Experiment Details 4.2 Experimental Results 5 Conclusions and Future Work References Enhancing Generative Generalized Zero Shot Learning via Multi-Space Constraints and Adaptive Integration 1 Introduction 2 Related Work 2.1 Embedding Learning Based GZSL Models 2.2 Generative GZSL Models 2.3 Other Methods 3 Method 3.1 Problem Formulation 3.2 Base Architecture: F-CLSWGAN 3.3 Cross-Modal Alignment in Semantic Alignment Space 3.4 Contrastive Learning in Image Embedding Space 3.5 Generalized Zero Shot Classification in Various Spaces 3.6 Adaptively Integrating Classification Results from Multiple Spaces 4 Experiment 4.1 Experimental Setup 4.2 Comparison on Benchmark Datasets 4.3 Ablation Study 4.4 Model Component Analysis 5 Conclusion References Joint Image Data Hiding and Rate-Distortion Optimization in Neural Compressed Latent Representations 1 Introduction 2 Related Works 2.1 Learned Image Compression 2.2 DNN-Based Steganography and Watermarking 3 Proposed Method 3.1 Problem Formulation 3.2 Message Encoder/Decoder Network 3.3 Joint Training and Loss Functions 3.4 Noise Attacks 4 Experimental Results 4.1 Steganography Secrecy 4.2 Watermark Robustness 4.3 Steganalysis 4.4 Embedding Performance and Overhead 5 Conclusion References GSUNet: A Brain Tumor Segmentation Method Based on 3D Ghost Shuffle U-Net 1 Introduction 2 Method 2.1 Architecture of GSUNet 2.2 3D Ghost Module 2.3 Dense Ghost Module 2.4 Ghost Shuffle Module 3 Experiments 3.1 Comparison Studies 3.2 Ablation Studies 4 Conclusion References ACT: Action-assoCiated and Target-Related Representations for Object Navigation 1 Introduction 2 Relate Work 2.1 Visual Navigation 2.2 Visual Representation in Visual Navigation 2.3 Attention in Visual Navigation 3 Task Definition 4 Proposed Method 4.1 Object Semantic Network 4.2 Action Associated Features with Visual Transformer 4.3 Historical Memory Network 4.4 Navigation Network 5 Experimental Results 5.1 Dataset and Metrics 5.2 Training Details 5.3 Comparison Methods 5.4 Quantitative Results 5.5 Ablation Study 5.6 Qualitative Study 6 Conclusion References Foreground Feature Enhancement and Peak & Background Suppression for Fine-Grained Visual Classification 1 Introduction 2 Related Work 2.1 Object-Part-Based Approaches 2.2 Attention-Based Approaches 3 Method 3.1 Foreground Feature Enhancement (FFE) 3.2 Peak and Background Suppression (PBS) 3.3 Network Design 4 Experiments 4.1 Datasets and Implementation Details 4.2 Quantitative Results 4.3 Ablation Studies 4.4 Visualization 5 Conclusion References YOLOv5-SRR: Enhancing YOLOv5 for Effective Underwater Target Detection 1 Introduction 2 Model 2.1 General Architecture 2.2 SPD-Block 2.3 RepBottleneck-ResNets 2.4 Soft-NMS 3 Experiments and Analysis 3.1 Experimental Settings 3.2 Model Hyperparameter Setting 3.3 URPC Dataset 3.4 Experimental Results and Analysis of the URPC Dataset 3.5 Ablation Experiment 4 Conclusions References Image Clustering and Generation with HDGMVAE-I 1 Introduction 2 Methodology 2.1 Disentangled Representation Learning Using HDGMVAE 2.2 Slack Variables 2.3 Fisher Discriminant as Regularization 2.4 Importance Sampling 3 Experimental Settings 3.1 Ablation Experiments on Slack Variables 3.2 Ablation Experiments with Fisher Discriminant as Regularization 4 Results and Discussions 5 Conclusion References ``Car or Bus?" CLearSeg: CLIP-Enhanced Discrimination Among Resembling Classes for Few-Shot Semantic Segmentation 1 Introduction 2 Related Work 2.1 Few-Shot Semantic Segmentation 2.2 CLIP-Based Multi-modal Networks 3 Preliminaries 4 Approach 4.1 Text-Driven Activation Module 4.2 Multi-level Correlation Module 4.3 Objective Function 4.4 Extension to K-Shot Settings 5 Experiments 5.1 Dataset and Evaluation Metrics 5.2 Implementation Details 5.3 Comparison with State-of-the-Arts 6 Ablation Study 7 Conclusions References PANDA: Prompt-Based Context- and Indoor-Aware Pretraining for Vision and Language Navigation 1 Introduction 2 Related Works 3 Method 3.1 VLN Problem Setup 3.2 Prompt Engineering 3.3 Indoor-Aware Stage with Deep Visual Prompts 3.4 Context-Aware Stage with Hard Context Prompts 4 Experiment 4.1 Experimental Setup 4.2 Comparison to State-of-the-Art Methods 4.3 Ablation Study 5 Conclusion References Cross-Modal Semantic Alignment Learning for Text-Based Person Search 1 Introduction 2 Related Work 2.1 Image-Based Person Search 2.2 Text-Based Person Search 3 Methodology 3.1 Overall Architecture 3.2 Token Clustering Learning Module 3.3 Feature Alignment Learning Module 3.4 Model Optimization 4 Experiment 4.1 Experimental Setup 4.2 Comparisons with State-of-the-Art Methods 4.3 Ablation Studies 4.4 Visualization Results 5 Conclusion References Point Cloud Classification via Learnable Memory Bank 1 Introduction 2 Related Work 3 Proposed Method 3.1 Learnable Memory Bank 3.2 Similarity-Based Matching 3.3 Loss Function 4 Experiments 4.1 Implementation and Evaluation Metrics 4.2 Experiment Results 4.3 Ablation Study 5 Conclusion References Adversarially Regularized Low-Light Image Enhancement 1 Introduction 2 Related Work 2.1 Traditional Methods 2.2 Deep Learning Methods 3 Proposed Method 3.1 Adversarial Attack 3.2 Multi-Path Convolution Block 3.3 Loss Function: 4 Experiment 4.1 Datasets and Implementation Details 4.2 Comparison with Previous Methods 4.3 Ablation Study 5 Conclusion References Advancing Incremental Few-Shot Semantic Segmentation via Semantic-Guided Relation Alignment and Adaptation 1 Introduction 2 Related Work 3 Methodology 3.1 Preliminaries 3.2 Semantic Relation Alignment 3.3 Semantic-Guided Adaptation 4 Experiments 4.1 Implementation Details 4.2 Main Experimental Results 4.3 Ablation Study 5 Conclusion References PMGCN:Preserving Measuring Mapping Prototype Graph Calibration Network for Few-Shot Learning 1 Introduction 2 Related Works 2.1 Meta-Learning 2.2 Attentional Mechanism 2.3 Graph Convolutional Network 2.4 Ergodic Theory 3 Methods 3.1 Problem Definition 3.2 Convolutional Block Attention Module 3.3 Preservation Measuring Mapping Graph Calibration Module 3.4 Loss Definition 4 Experiments 4.1 Datasets 4.2 Implementation Details 4.3 Experimental Results 4.4 Importance of Similarity Judgment 4.5 Ablation Study 5 Conclusion References ARE-CAM: An Interpretable Approach to Quantitatively Evaluating the Adversarial Robustness of Deep Models Based on CAM 1 Introduction 2 ARE-CAM: Adversarial Robustness Evaluation Based on CAM 2.1 Preliminaries 2.2 Metrics of ARE-CAM 2.3 Workflow of ARE-CAM 3 Experiments 3.1 Settings 3.2 Results and Analysis 3.3 Comparison with Baseline Metrics 4 Conclusion References SSK-Yolo: Global Feature-Driven Small Object Detection Network for Images 1 Introduction 2 Related Work 2.1 Machine Learning 2.2 Deep Learning 3 SSK-Yolo 3.1 Data Pre-processing 3.2 Backbone 3.3 Data Post-processing 4 Experiments 4.1 Datasets 4.2 Experimental Configuration 4.3 Comparation Study 4.4 Ablation Study 5 Conclusion References MetaVSR: A Novel Approach to Video Super-Resolution for Arbitrary Magnification 1 Introduction 2 Related Works 2.1 Single Image Super Resolution 2.2 Video Super Resolution 3 Proposed Method 3.1 Overview 3.2 Inter-frames Alignment 3.3 Feature Extraction Module 3.4 Upscale Module 3.5 Loss Function 4 Experiments 4.1 Implementation Details 4.2 Comparisons with State-of-the-Art Methods 5 Conclusions References From Skulls to Faces: A Deep Generative Framework for Realistic 3D Craniofacial Reconstruction 1 Introduction 2 Related Work 2.1 Traditional Craniofacial Reconstruction 2.2 Deep Image Generation 3 Proposed Method 3.1 Overview 3.2 Details of Model Architecture 4 Experiments and Analysis 4.1 Datasets 4.2 Implementation Details 4.3 Evaluation Metrics 4.4 Experimental Comparison and Analysis 5 Conclusion and Discussion References Structure-Aware Adaptive Hybrid Interaction Modeling for Image-Text Matching 1 Introduction 2 Related Work 2.1 Image-Text Matching 2.2 Multimodal Interaction Modeling 3 Methodology 3.1 Feature Representation 3.2 Adaptive Hybrid Modeling 3.3 Multimodal Graph Inference 3.4 Entity Attention Enhancement 3.5 Objective Functions 4 Experiments 4.1 Datasets and Implementation Details 4.2 Experimental Results 4.3 Ablation Studies 4.4 Case Studies 5 Conclusion References Using Saliency and Cropping to Improve Video Memorability 1 Introduction 2 Background 2.1 The MediaEval Benchmarking Task on Predicting Video Memorability 2.2 Image Saliency 2.3 The Memento10k Dataset and Memorability Scores 3 Experimental Methodology 3.1 Predicting Video Memorability 3.2 Saliency-Based Cropping 4 Experimental Results 4.1 Cropping at Video Centrepoints 4.2 Cropping with Saliency Tracking 5 Conclusions References Contextual Augmentation with Bias Adaptive for Few-Shot Video Object Segmentation 1 Introduction 2 Related Work 2.1 Video Object Segmentation 2.2 Few-Shot Semantic Segmentation 2.3 Few-Shot Video Object Segmentation 3 Problem Definition 4 Proposed Method 4.1 Context Augmented Learner 4.2 Bias Adaptive Learner 4.3 Segmentation Fusion 5 Experiments 5.1 Dataset and Metrics 5.2 Implementation Details 5.3 Comparisons to Existing Methods 5.4 Ablation Study 6 Conclusion References A Lightweight Local Attention Network for Image Super-Resolution 1 Introduction 2 Related Works 2.1 Lightweight SISR Methods Based on CNNs 2.2 Feature Fusion and Reconstruction 3 Proposed Method 3.1 Network Architecture 3.2 Feature Extraction 3.3 Feature Fusion and Reconstruction 4 Experiments 4.1 Experimental Settings 4.2 Ablation Analysis 4.3 Experiments on Feature Fusion 4.4 Comparison with State-of-the-Art Methods 4.5 Subjective Visual Comparison 5 Conclusions References Domain Adaptation for Speaker Verification Based on Self-supervised Learning with Adversarial Training 1 Introduction 2 Related Works 3 The Proposed Model 3.1 Self-supervised Learning Without Negative Samples 3.2 Adversarial Training and Speaker Classification 3.3 Joint Optimization Strategy 4 Experimental Setup 4.1 Dataset 4.2 Data Preprocessing 4.3 System Configuration 5 Results 5.1 Ablation Study 5.2 Performance Evaluation of Our Method with Other Domain Adaptation Methods 6 Conclusions References Quality Scalable Video Coding Based on Neural Representation 1 Introduction 2 Related Work 3 Methodology 3.1 Overview 3.2 The Base Layer 3.3 The Enhancement Layers 3.4 End-to-End Multi-task Training Strategy 4 Experiments 4.1 Datasets and Settings 4.2 Main Results 4.3 Ablation 5 Conclusion References Hierarchical Bi-directional Temporal Context Mining for Improved Video Compression 1 Introduction 2 Related Work 3 Proposed Method 3.1 Overview 3.2 Multi-scale Bidirectional Contextual Information Prediction 3.3 Bidirectional Contextual Adaptive Encoder-Decoder 3.4 Loss Function 4 Experiments 4.1 Experimental Setup 4.2 Experimental Results 4.3 Ablation Study 5 Conclusion References MAMixer: Multivariate Time Series Forecasting via Multi-axis Mixing 1 Introduction 2 Related Work 2.1 Transformer-Based Models for Time Series Forecasting 2.2 MLP-Based Models for Computer Vision 2.3 MLP-Based Models for Time Series Forecasting 3 Methodology 3.1 Preliminary 3.2 Data Processing 3.3 MAMixer Block 3.4 Intra-axis Module 3.5 Prediction Layer 4 Experiments 4.1 Datasets 4.2 Baselines and Experimental Setup 4.3 Main Results 4.4 Ablation Study 4.5 Model Analysis 5 Conclusion References A Custom GAN-Based Robust Algorithm for Medical Image Watermarking 1 Introduction 2 Related Work 2.1 Spatial Domain Techniques 2.2 Transform Domain Techniques 3 Methods 3.1 Propounded Watermarking Algorithm 3.2 Loss Function 3.3 Watermark Embedding and Extraction 4 Experiment 4.1 Experimental Setup and Results 4.2 Ablation Experiment 4.3 Comparative Experiment 5 Conclusion References A Detail-Guided Multi-source Fusion Network for Remote Sensing Object Detection 1 Introduction 2 Related Work 3 Proposed Method 3.1 Detail-Generating Module 3.2 Detail-Guided Spatial Attention Feature Fusion Module 3.3 Loss Function 4 Datasets and Experiment Settings 4.1 Optical-SAR Fusion Object Detection Datasets 4.2 Experiment Settings 5 Experimental Comparison and Analysis 5.1 Experimental Results 5.2 Ablation Studies 5.3 Visualization 6 Conclusion References A Secure and Fair Federated Learning Protocol Under the Universal Composability Framework 1 Introduction 2 Preliminary Knowledge 2.1 Universal Composability Framework 2.2 Differential Privacy 3 Ideal Function 3.1 Initialization Phase 3.2 Client Selection Phase 3.3 Client Update Phase 3.4 Server-Side Aggregation Phase 3.5 Fairness and Incentive Mechanism Phase 3.6 Termination Condition Phase 4 Fair and Secure Federated Learning Protocol 4.1 Protocol's Input and Output 4.2 Client Model Update 4.3 Server Model Aggregation 4.4 Fairness and Incentive Mechanism 4.5 Detection and Penalty for Dishonest or Malicious Behavior 5 Security Proof 6 Experiment 6.1 Experimental Environment 6.2 Experimental Results 7 Conclusion References Bi-directional Interaction and Dense Aggregation Network for RGB-D Salient Object Detection 1 Introduction 2 Related Work 2.1 RGB Salient Object Detection 2.2 RGB-D Salient Object Detection 3 Method 3.1 Overview 3.2 Bi-directional Interactive Encoder 3.3 Dense Aggregation Network for Prediction 3.4 Loss Function 4 Experiments 4.1 Datasets and Evaluation Metrics 4.2 Implementation Details 4.3 Comparison with State-of-the-Art Methods 4.4 Ablation Study 5 Conclusion References Face Forgery Detection via Texture and Saliency Enhancement 1 Introduction 2 Related Work 2.1 Face Forgery Detection 2.2 Gray-Level Co-occurrence Matrix 2.3 Saliency Map 3 Method 3.1 Overall Framework 3.2 Dynamic Texture Enhancement Module (DTEM) 3.3 Salient Region Attention Module (SRAM) 3.4 Loss Function 4 Experiments 4.1 Experimental Setup 4.2 Experimental Results 4.3 Visualization 5 Conclusion References Author Index This book constitutes the refereed proceedings of the30th International Conference on MultiMedia Modeling, MMM 2024, held in Amsterdam, The Netherlands, during January 29 February 2, 2024. The 112 full papers included in this volume were carefully reviewed and selected from 297 submissions. The MMM conference were organized in topics related to multimedia modelling, audio, image, video processing, coding and compression; multimodal analysis for retrieval applications, and multimedia fusion methods.