Experimental IR Meets Multilinguality, Multimodality, and Interaction: 14th International Conference of the CLEF Association, CLEF 2023, Thessaloniki, ... (Lecture Notes in Computer Science)

This volume LNCS 14163 constitutes the refereed proceedings of 14th International Conference of the CLEF Association, CLEF 2023, in Thessaloniki, Greece, during September 18–21, 2023. The 10 full papers and one short paper included in this book were carefully reviewed and selected from 35 submissions. The conference focuses on authorship attribution, fake news detection and news tracking, noise-detection in automatically transferred relevance judgments, impact of online education on children’s conversational search behavior, analysis of multi-modal social media content, knowledge graphs for sensitivity identification, a fusion of deep learning and logic rules for sentiment analysis, medical concept normalization and domain-specific information extraction. In addition to this, the volume presents 7 “Best of the labs” papers which were reviewed as full paper submissions with the same review criteria. 13 lab overview papers were accepted and represent scientific challenges based on new datasets and real world problems in multimodal and multilingual information access. Preface Organization Contents Conference Papers Inception Models for Fashion Image Captioning: An Extensive Study on Multiple Datasets 1 Introduction 2 Related Work 3 Data 3.1 Reduced-FACAD 3.2 Reduced-InFashAI 3.3 Fashion-Cap 4 Experimental Setting 4.1 Models 4.2 Setup 5 Results 6 Qualitative Analysis 7 Conclusions References The Best is Yet to Come: A Reproducible Analysis of CLEF eHealth TAR Experiments 1 Introduction 2 Background on CLEF 2019 eHealth TAR Lab 3 Original Experiments 4 Reproducing the Experiments 4.1 Fixing the Code 4.2 Understanding the Hyperparameters 4.3 Additional Bugs Fixing 4.4 R Markdown for Better Reproducibility 4.5 Further Operations to Improve Results 5 Conclusions References Predicting Retrieval Performance Changes in Evolving Evaluation Environments 1 Introduction 2 Related Work 3 Approach and Formalizations 3.1 Simulating an Evolving Evaluation Environment 3.2 Knowledge Delta 3.3 Result Delta 4 Experiments, Results and Discussion 4.1 Dataset Preparation 4.2 Classification Process 4.3 Experiments 4.4 Results 5 Summary and Future Work References When Sarcasm Hurts: Irony-Aware Models for Abusive Language Detection 1 Introduction 2 Related Work 3 Methodology 4 Irony in Hateful Contents 5 Conclusions References Cem Mil Podcasts: A Spoken Portuguese Document Corpus for Multi-modal, Multi-lingual and Multi-dialect Information Access Research 1 Introduction 2 Dataset Construction 2.1 Dataset Schema 2.2 Access to the Dataset 3 Descriptive Statistics and Comparison to English Podcast Dataset 4 Podcast Genre Prediction Case Study 4.1 Genre Prediction Experiment Setup 4.2 Genre Prediction Experiment Results 5 Episode Summarization Case Study 5.1 Data Preparation for Summarization 5.2 Models for Summarizatiom 5.3 Results for Summarization 6 Conclusions References Using Authorship Embeddings to Understand Writing Style in Social Media 1 Introduction 2 Related Work 2.1 Authorship with Transformers 2.2 Towards Universal Authorship Embeddings 2.3 Supervised Contrastive Pretraining 2.4 Network Architecture and Hyper-parameters 3 Data 3.1 Data Handling 4 Experimentation 4.1 Authorship Attribution 4.2 Authorship Verification 5 Conclusions References Trend Detection in Crime-Related Time Series with Change Point Detection Methods 1 Introduction 2 Related Work 3 Methodology 3.1 Offline Change Point Detection (CPD) Methods 3.2 Online Change Point Detection (CPD) Methods 4 Datasets 4.1 TCPD Benchmark and Dataset Collection 4.2 Crimes in Boston Dataset 4.3 London Police Records Dataset 5 Experimental Results 5.1 Experimental Results on the TCPD Dataset Collection 5.2 Experimental Results on the Crime-Related Datasets 6 Conclusions and Future Work References DAVI: A Dataset for Automatic Variant Interpretation 1 Introduction 2 Dataset Construction 2.1 Article Retrieval 2.2 Manual Curation 3 Automatic Variant Annotation 3.1 Pre-processing 3.2 Model Construction 3.3 Performance Measures 4 Results 4.1 Manual Curation Results 4.2 Pre-processing Results 4.3 Classification Results 5 Discussion and Future Work References qCLEF: A Proposal to Evaluate Quantum Annealing for Information Retrieval and Recommender Systems 1 Introduction 2 Related Works 3 The qCLEF Proposal 3.1 Task 1 - Quantum Feature Selection 3.2 Task 2 - Quantum Clustering 3.3 Task 3 - Quantum Boosting 3.4 Additional Challenges 3.5 Evaluation of Quantum Annealing 4 Implementation of the Infrastructure 5 Feature Selection in Practice 6 Conclusions and Future Work References Graph-Enriched Biomedical Entity Representation Transformer 1 Introduction 2 Related Work 3 Background and Architecture 3.1 Self-alignment Pretraining 3.2 Graph Neural Networks 3.3 GEBERT 4 Experimental Evaluation 4.1 Results 5 Conclusion References Supervised Machine-Generated Text Detectors: Family and Scale Matters 1 Introduction 2 Related Work 3 Experiments 3.1 Experimental Set-Up 3.2 Generalization in MGT Detection 3.3 Family and Scale Attribution 4 Conclusions References Best of CLEF 2022 Labs Cross-Lingual Candidate Retrieval and Re-ranking for Biomedical Entity Linking 1 Introduction 2 Related Work 3 Materials and Methods 3.1 DisTEMIST Dataset 3.2 Dictionaries and Multilingual Aliases 3.3 Candidate Generation 3.4 Entity Ranking 3.5 Evaluation 4 Results 4.1 Candidate Generation 4.2 Re-ranking 5 Evaluation and Discussion 5.1 Comparison with Baseline 5.2 Error Analysis 5.3 Limitations 6 Conclusion and Outlook References Humour Translation with Transformers 1 Introduction 2 Materials and Methods 2.1 TASK-1: Explain and Classify Instances of Wordplay 2.2 TASK-2: Translate Single Words Containing Wordplay 2.3 TASK-3: Translate Entire Phrases Containing Wordplay 3 Experiments and Results 3.1 IMPLEMENTATION OF SOLUTION FOR TASK-1: Explain and Classify Instances of Wordplay 3.2 IMPLEMENTATION OF SOLUTION FOR TASK-2: Translate Words Containing a Wordplay 3.3 IMPLEMENTATION OF SOLUTION FOR TASK-3: Translate Entire Phrases Containing Wordplay 4 Conclusion References Fight Against Misinformation on Social Media: Detecting Attention-Worthy and Harmful Tweets and Verifiable and Check-Worthy Claims 1 Introduction 2 Tasks 3 Related Work 4 Approaches 4.1 Fine Tuning Various Transformer Models 4.2 Increasing Training Data via Machine Translation 4.3 Language Specific BERT with Manifold Mixup 5 Experiments 5.1 Experimental Setup 5.2 Experimental Results in the Development Phase 5.3 Results of Our Submissions 6 Conclusion References A Re-labeling Approach Based on Approximate Nearest Neighbors for Identifying Gambling Disorders in Social Media 1 Introduction 2 Related Work 3 Early Detection of Signs of Pathological Gambling 3.1 Dataset 3.2 Metrics 4 Proposed Model 4.1 Data Representation 4.2 Approximate Nearest Neighbors 4.3 Relabeling Process 4.4 Tag and Scoring Function 4.5 Crawling New Positive Instances 5 Results and Discussion 6 Conclusions and Future Work References Touché 2022 Best of Labs: Neural Image Retrieval for Argumentation 1 Introduction 2 Related Work 3 The Touché'22 Dataset 4 Development of the NeurArgs Approach 4.1 The Topic Model 4.2 The Argument Model 4.3 The Stance Model 5 Evaluation of the NeurArgs Approach 5.1 Topic and Argument Retrieval 5.2 Comparison of Stance Detection Models 6 Insights into Image Stance Detection 7 Conclusion References SimpleText Best of Labs in CLEF-2022: Simplify Text Generation with Prompt Engineering 1 Introduction 2 Techniques in Our Approach 2.1 Term Frequency Inverse Document Frequency (TF-IDF) 2.2 Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer(T5) 3 Data Set 4 System 4.1 Search Passage Using a Two-Stage TF-IDF Filter 4.2 T5 Model for Summarization 5 Results and Discussion 6 Further Evaluation on the Large Language Model 6.1 Flesch-Kincaid Readability Tests 6.2 The Vocabulary Profile 6.3 Academic Word List 7 Conclusion and Future Works References Answer Retrieval for Math Questions Using Structural and Dense Retrieval 1 Introduction 2 The ARQMath Lab 3 Related Work 4 Structural and Dense Retrieval for MIR 4.1 Structure Search Using Approach0 4.2 Dense Semantic Search Using ColBERT 4.3 Combing Structural and Dense Retrieval 5 Experimental Setup 5.1 Dataset and Evaluation Protocol 5.2 Training Data 5.3 Preprocessing for Math 5.4 Training Configurations 5.5 Task-Specific Settings 6 Evaluation 7 Conclusion and Future Work References CLEF 2023 Lab Overviews Overview of BioASQ 2023: The Eleventh BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering 1 Introduction 2 Overview of the Tasks 2.1 Biomedical Semantic QA - Task 11b 2.2 Task Synergy 11 2.3 Medical Semantic Annotation in Spanish - MedProcNER 3 Overview of Participation 3.1 Task 11b 3.2 Task Synergy 11 3.3 Task MedProcNER 4 Results 4.1 Task 11b 4.2 Task Synergy 11 4.3 Task MedProcNER 5 Conclusions References Overview of the CLEF–2023 CheckThat! Lab on Checkworthiness, Subjectivity, Political Bias, Factuality, and Authority of News Articles and Their Source 1 Introduction 2 Previously on the CheckThat! Lab 3 Description of the 2023 Tasks 3.1 Task 1: Check-Worthiness in Multimodal and Multigenre Content 3.2 Task 2: Subjectivity in News Articles 3.3 Task 3: Political Bias of News Articles and News Media 3.4 Task 4: Factuality of Reporting of News Media 3.5 Task 5: Authority Finding in Twitter 4 Datasets 4.1 Task 1: Check-Worthiness of Multimodal and Multigenre Content 4.2 Task 2: Subjectivity in News Articles 4.3 Task 3: Political Bias of News Articles and News Media 4.4 Task 4: Factuality of Reporting of News Media 4.5 Task 5: Authority Finding in Twitter 5 Results 5.1 Task 1: Check-Worthiness in Multimodal and Unimodal Contents 5.2 Task 2: Subjectivity in News Articles 5.3 Task 3: Political Bias of News Articles and News Media 5.4 Task 4: Factuality of Reporting of News Media 5.5 Task 5: Authority Finding in Twitter 6 Related Work 7 Conclusions and Future Work References Overview of DocILE 2023: Document Information Localization and Extraction 1 Introduction 2 Data 3 Tasks and Evaluation 4 Submissions 5 Results and Discussion 5.1 Text Extraction Evaluation 5.2 Evaluation on Zero/Few/Many-Shot Layouts 5.3 Using Synthetic and Unlabeled Data 5.4 Importance of Score for Average Precision 6 Conclusion References Overview of eRisk 2023: Early Risk Prediction on the Internet 1 Introduction 2 Task 1: Search for Symptoms of Depression 2.1 Dataset 2.2 Assessment Process 2.3 Results 3 Task 2: Early Detection of Pathological Gambling 3.1 Decision-Based Evaluation 3.2 Ranking-Based Evaluation 3.3 Results 4 Task 3: Measuring the Severity of Eating Disorders 4.1 Evaluation Metrics 4.2 Results 5 Conclusions References Overview of EXIST 2023 – Learning with Disagreement for Sexism Identification and Characterization 1 Introduction 2 Tasks 2.1 Task 1: Sexism Identification 2.2 Task 2: Source Intention 2.3 Task 3: Sexism Categorization 3 Dataset: Managing Bias in Data and Providing Pre-aggregated Annotations 3.1 Data Sampling 3.2 Labeling Process 3.3 Pre-aggregated Annotations: The Learning with Disagreement Paradigm 4 Evaluation Methodology and Metrics 5 Overview of Approaches 6 Results 6.1 Task 1 Sexism Identification 6.2 Task 2 Source Intention 6.3 Task 3 Sexism Categorization 7 Conclusions References Intelligent Disease Progression Prediction: Overview of iDPP@CLEF 2023 1 Introduction 2 Related Challenges 2.1 iDPP@CLEF 2022 3 Tasks 3.1 Task 1: Predicting Risk of Disease Worsening (MS) 3.2 Task 2: Predicting Cumulative Probability of Worsening (MS) 3.3 Task 3: Position Papers on the Impact of Exposition to Pollutants (ALS) 4 Dataset 4.1 Task 1 and Task 2: MS Datasets 4.2 Task 3: ALS Dataset 5 Lab Setup and Participation 5.1 Guidelines 5.2 Participants 6 Evaluation Measures 6.1 Task 1: Measures to Evaluate the Prediction of the Risk of Disease Worsening (MS) 6.2 Task 2: Measures to Evaluate the Prediction of the Cumulative Probability of Worsening (MS) 6.3 Task 3: Measures to Evaluate the Impact of Exposition to Pollutants (ALS) 7 Results 7.1 Task 1: Predicting Risk of Disease Worsening (Multiple Sclerosis) 7.2 Task 2: Predicting Cumulative Probability of Worsening (Multiple Sclerosis) 7.3 Task 3: Position Papers on Impact of Exposition to Pollutants (Amyotrophic Lateral Sclerosis) 7.4 Approaches 8 Conclusions and Future Work References Overview of the ImageCLEF 2023: Multimedia Retrieval in Medical, Social Media and Internet Applications 1 Introduction 2 Overview of Tasks and Participation 3 The Caption Task 3.1 Task Setup 3.2 Data Set 3.3 Participating Groups and Submitted Runs 3.4 Results 3.5 Lessons Learned and Next Steps 4 The MEDIQA-Sum Task 4.1 Task Setup 4.2 Data Set 4.3 Participating Groups and Submitted Runs 4.4 Results 4.5 Lessons Learned and Next Steps 5 The GANs Task 5.1 Task Setup 5.2 Data Set 5.3 Participating Groups and Submitted Runs 5.4 Results 5.5 Lessons Learned and Next Steps 6 The MedVQA-GI Task 6.1 Task Setup 6.2 Data Set 6.3 Results 6.4 Lessons Learned and Next Steps 7 The Fusion Task 7.1 Task Setup 7.2 Data Set 7.3 Participating Groups and Submitted Runs 7.4 Results 7.5 Lessons Learned and Next Steps 8 The Aware Task 9 Conclusion References Overview of JOKER – CLEF-2023 Track on Automatic Wordplay Analysis 1 Introduction 2 Task 1: Pun Detection in English, French, and Spanish 2.1 Data 2.2 Participants' Approaches 2.3 Results 3 Task 2: Pun Location and Interpretation in English, French, and Spanish 3.1 Data 3.2 Participants' Approaches 3.3 Results 4 Task 3: Translation of Puns from English to French and Spanish 4.1 Data 4.2 Evaluation 4.3 Participants' Approaches 4.4 Results 5 Open Task 5.1 Pun Generation for Text Transformation and Conversational Systems 5.2 Sentiment Analysis for Wordplay 5.3 Comparison of Machine and Human Performances 6 Conclusion References Overview of LifeCLEF 2023: Evaluation of AI Models for the Identification and Prediction of Birds, Plants, Snakes and Fungi 1 LifeCLEF Lab Overview 2 BirdCLEF Challenge: Bird Call Identification in Soundscapes 2.1 Objective 2.2 Dataset 2.3 Evaluation Protocol 2.4 Participants and Results 3 SnakeCLEF Challenge: Snake Identification in Medically Important Scenarios 3.1 Motivation 3.2 Objective 3.3 Dataset 3.4 Evaluation Protocol 3.5 Participants and Results 4 FungiCLEF Challenge: Fungi Recognition Beyond 0–1 Cost 4.1 Objective 4.2 Dataset 4.3 Evaluation Protocol 4.4 Participants and Results 5 PlantCLEF Challenge: Identify the World's Flora 5.1 Objective 5.2 Dataset 5.3 Evaluation Protocol 5.4 Participants and Results 6 GeoLifeCLEF Challenge: Species Composition Prediction with High Spatial Resolution at Continental Scale Using Remote Sensing 6.1 Objective 6.2 Training Dataset 6.3 Evaluation Protocol 6.4 Participants and Results 7 Conclusions and Perspectives References Overview of the CLEF-2023 LongEval Lab on Longitudinal Evaluation of Model Performance 1 Introduction 2 Task 1 - Retrieval 2.1 Description of the Task 2.2 Dataset 2.3 Submissions 2.4 Absolute Scores 2.5 Changes in the Scores 2.6 Run Rankings 2.7 Discussion and Conclusion 3 Task 2 - Classification 3.1 Description of the Task 3.2 Dataset 3.3 Evaluation 3.4 Results 3.5 Practice Phase 3.6 Evaluation Phase 3.7 Discussion 3.8 Conclusion References Overview of PAN 2023: Authorship Verification, Multi-Author Writing Style Analysis, Profiling Cryptocurrency Influencers, and Trigger Detection 1 Introduction 2 Cross-Discourse Type Authorship Verification 3 Multi-Author Writing Style Analysis 4 Author Profiling 5 Trigger Detection References Overview of the CLEF 2023 SimpleText Lab: Automatic Simplification of Scientific Texts 1 Introduction 2 Task 1: What is in (or Out)? 2.1 Data 2.2 Attended Results 2.3 Evaluation Metrics 2.4 Participants' Approaches 2.5 Results 3 Task 2: What is Unclear? 3.1 Data 3.2 Evaluation Metrics 3.3 Participants' Approaches 3.4 Results 4 Task 3: Rewrite This! 4.1 Data 4.2 Evaluation Metrics 4.3 Participants' Approaches 4.4 Results 4.5 Difficult Terms and Simplification 5 Discussion and Conclusions References Overview of Touché 2023: Argument and Causal Retrieval 1 Introduction 2 Lab Overview and Statistics 3 Task 1: Argument Retrieval for Controversial Questions 3.1 Task Definition 3.2 Data Description 3.3 Evaluation Setup 3.4 Submitted Approaches and Evaluation Results 4 Task 2: Evidence Retrieval for Causal Questions 4.1 Task Definition 4.2 Data Description 4.3 Evaluation Setup 4.4 Submitted Approaches and Evaluation Results 5 Task 3: Image Retrieval for Arguments 5.1 Task Definition 5.2 Data Description 5.3 Evaluation Setup 5.4 Submitted Approaches and Evaluation Results 6 Task 4: Multilingual Multi-target Stance Classification 6.1 Task Definition 6.2 Data Description 6.3 Submitted Approaches and Evaluation Results 7 Conclusion References A Zero-Shot Prompts B Full Evaluation Results of Touché 2023: Argument and Causal Retrieval Author Index