[Lecture Notes in Computer Science] Machine Learning and Knowledge Discovery in Databases Volume 11907 (European Conference, ECML PKDD 2019, WÃ1⁄4rzburg, Germany, September 16–20, 2019, Proceedings, Part II) ||

The three volume proceedings LNAI 11906 – 11908 constitutes the refereed proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2019, held in Würzburg, Germany, in September 2019. The total of 130 regular papers presented in these volumes was carefully reviewed and selected from 733 submissions; there are 10 papers in the demo track. The contributions were organized in topical sections named as follows: Part I: pattern mining; clustering, anomaly and outlier detection, and autoencoders; dimensionality reduction and feature selection; social networks and graphs; decision trees, interpretability, and causality; strings and streams; privacy and security; optimization. Part II: supervised learning; multi-label learning; large-scale learning; deep learning; probabilistic models; natural language processing. Part III: reinforcement learning and bandits; ranking; applied data science: computer vision and explanation; applied data science: healthcare; applied data science: e-commerce, finance, and advertising; applied data science: rich data; applied data science: applications; demo track. Chapter "Incorporating Dependencies in Spectral Kernels for Gaussian Processes" is available open access under a Creative Commons Attribution 4.0 International License via link.springer.com. Preface Organization Contents – Part II Supervised Learning Exploiting the Earth's Spherical Geometry to Geolocate Images 1 Introduction 2 Prior Work 2.1 Image Retrieval 2.2 Classification 3 Geolocation via the MvMF 3.1 The Probabilistic Interpretation 3.2 Interpretation as a Classifier 3.3 Interpretation as an Image Retrieval Method 3.4 Analysis 4 Experiments 4.1 Procedure 4.2 Results 5 Conclusion References Continual Rare-Class Recognition with Emerging Novel Subclasses 1 Introduction 2 Problem Setup and Preliminary Data Analysis 3 Continual Rare-Class Recognition 3.1 Model Formulation 3.2 Convexity and Optimization 3.3 Time and Space-Complexity Analysis 4 Evaluation 4.1 Experiment Setup 4.2 Experiment Results 5 Related Work 6 Conclusion References Unjustified Classification Regions and Counterfactual Explanations in Machine Learning 1 Introduction 2 Background 2.1 Post-hoc Interpretability 2.2 Studies of Post-hoc Interpretability Approaches 2.3 Adversarial Examples 3 Justification Using Ground-Truth Data 3.1 Intuition and Definitions 3.2 Implementation 4 Procedures for Assessing the Risk of Unconnectedness 4.1 LRA Procedure 4.2 VE Procedure 5 Experimental Study: Assessing the Risk of Unjustified Regions 5.1 Experimental Protocol 5.2 Defining the Problem Granularity: Choosing n and 5.3 Detecting Unjustified Regions 5.4 Vulnerability of Post-hoc Counterfactual Approaches 6 Conclusion References Shift Happens: Adjusting Classifiers 1 Introduction 2 Background and Related Work 2.1 Dataset Shift and Prior Probability Adjustment 2.2 Proper Scoring Rules and Bregman Divergences 2.3 Adjusted Predictions and Adjustment Procedures 3 General Adjustment 3.1 Unbounded General Adjustment (UGA) 3.2 Bounded General Adjustment 3.3 Implementation 4 Experiments 4.1 Experimental Setup 4.2 Results 5 Conclusion References Beyond the Selected Completely at Random Assumption for Learning from Positive and Unlabeled Data 1 Introduction 2 Preliminaries 3 Labeling Mechanisms for PU Learning 4 Learning with SAR Labeling Mechanisms 4.1 Case 1: True Propensity Scores Known 4.2 Case 2: Propensity Scores Estimated from Data 5 Learning Under the SAR Assumption 5.1 Reducing SAR to SCAR 5.2 EM for Propensity Estimation 6 Empirical Evaluation 6.1 Data 6.2 Methodology and Approaches 6.3 Results 7 Related Work 8 Conclusions References Cost Sensitive Evaluation of Instance Hardness in Machine Learning 1 Introduction 2 Notation and Basic Definitions 3 Instance Hardness and Cost Curves 3.1 Score-Fixed Instance Hardness 3.2 Score-Driven Instance Hardness 3.3 Rate-Driven Instance Hardness 3.4 Score-Uniform Instance Hardness 3.5 Rate-Uniform Instance Hardness 4 Experiments 5 Conclusion References Non-parametric Bayesian Isotonic Calibration: Fighting Over-Confidence in Binary Classification 1 Introduction 2 Evaluation of Calibration 3 Simple Improvement of Existing Methods 4 Proposed Method 4.1 Non-parametric Bayesian Isotonic Calibration 4.2 Selecting the Prior over Isotonic Maps 4.3 Practically Efficient Sampling from Prior 5 Experiments 5.1 Experiments on Synthetic Data 5.2 Experimental Setup on Real Data 5.3 Experiment Results on Real Data 6 Conclusions References Multi-label Learning PP-PLL: Probability Propagation for Partial Label Learning 1 Introduction 2 Related Work 3 The PP-PLL Method 4 Optimization 4.1 Updating F 4.2 Updating 5 Experiments 5.1 Experimental Setup 5.2 Experimental Results 5.3 Sensitivity Analysis 6 Conclusion References Neural Message Passing for Multi-label Classification 1 Introduction 2 Method: LaMP Networks 2.1 Background: Message Passing Neural Networks 2.2 LaMP: Label Message Passing 2.3 Readout Layer (MLC Predictions from the Label Embeddings) 2.4 Model Details 2.5 Loss Function 2.6 LaMP Variation: Input Encoding with Feature Message Passing (FMP) 2.7 Advantages of LaMP Models 2.8 Connecting to Related Topics 3 Experiments 3.1 LaMP Variations 3.2 Performance Evaluation 3.3 Interpretability Evaluation 4 Conclusion A Appendix: MLC Background A.1 Background of Multi-label Classification A.2 Seq2Seq Models A.3 Drawbacks of Autoregressive Models B Appendix: Dataset Details C Appendix: Extra Metrics D Appendix: More About Experiments D.1 Datasets D.2 Evaluation Metrics D.3 Model Hyperparameter Tuning D.4 Baseline Comparisons References Assessing the Multi-labelness of Multi-label Data 1 Introduction 2 Background: Multi-label Data and Multicollinearity 3 Analytical Models for Measuring Multi-labelness 3.1 Regularisation of Analytical Models 3.2 Split Analytical Model 4 Analysis of Full and Split Analytical Models 4.1 Measuring Multi-labelness 4.2 Generating Multi-label Data 4.3 Investigation: Full Model with l1 and l2 Regularisation 4.4 Investigation: Split Model with l1 and l2 Regularisation 4.5 Comparing Full and Split Regression 5 Full and Split Analytical Models on Real Data 5.1 Label Interdependence 5.2 Effect of Label-Interdependence Reduction on Accuracy 6 Conclusion References Synthetic Oversampling of Multi-label Data Based on Local Label Distribution 1 Introduction 2 Related Work 3 Our Approach 3.1 Selection of Seed Instances 3.2 Synthetic Instance Generation 3.3 Ensemble of Multi-Label Sampling (EMLS) 3.4 Complexity Analysis 4 Empirical Analysis 4.1 Setup 4.2 Results and Analysis 5 Conclusion References Large-Scale Learning Distributed Learning of Non-convex Linear Models with One Round of Communication 1 Introduction 2 Problem Setting 3 The OWA Estimator 3.1 Warmup: The Full OWA 3.2 The OWA Estimator 3.3 Implementing OWA with Existing Optimizers 3.4 Fast Cross Validation for OWA 4 Analysis 4.1 The Sub-Gaussian Tail (SGT) Condition 4.2 The Main Idea: owa Contains Good Solutions 4.3 Bounding the Generalization Error 4.4 Bounding the Estimation Error 5 Other Non-interactive Estimators 6 Experiments 6.1 Synthetic Data 6.2 Real World Advertising Data 7 Conclusion References SLSGD: Secure and Efficient Distributed On-device Machine Learning 1 Introduction 2 Related Work 3 Problem Formulation 3.1 Non-IID Local Datasets 3.2 Data Poisoning 4 Methodology 4.1 Threat Model and Defense Technique 5 Convergence Analysis 5.1 Assumptions 5.2 Convergence Without Data Poisoning 5.3 Convergence with Data Poisoning 6 Experiments 6.1 Datasets and Evaluation Metrics 6.2 SLSGD Without Attack 6.3 SLSGD Under Data Poisoning Attack 6.4 Acceleration by Local Updates 6.5 Discussion 7 Conclusion References Trade-Offs in Large-Scale Distributed Tuplewise Estimation And Learning 1 Introduction 2 Background 2.1 U-Statistics: Definition and Applications 2.2 Large-Scale Tuplewise Inference with Incomplete U-Statistics 2.3 Practices in Distributed Data Processing 3 Distributed Tuplewise Statistical Estimation 3.1 Naive Strategies 3.2 Proposed Approach 3.3 Analysis 3.4 Practical Considerations and Other Repartitioning Schemes 4 Extensions to Stochastic Gradient Descent for ERM 4.1 Gradient-Based Empirical Minimization of U-statistics 4.2 Repartitioning for Stochastic Gradient Descent 5 Numerical Results 6 Future Work References Deep Learning Importance Weighted Generative Networks 1 Introduction 1.1 Related Work 2 Problem Formulation and Technical Approach 2.1 Maximum Mean Discrepancy Between Two Distributions 2.2 Importance Weighted Estimator for Known M 2.3 Robust Importance Weighted Estimator for Known M 2.4 Self-normalized Importance Weights for Unknown M 2.5 Approximate Importance Weighting by Data Duplication 3 Evaluation 3.1 Can GANs with Importance Weighted Estimators Recover Target Distributions, Given M? 3.2 In a High-Dimensional Image Setting, How Does Importance Weighting Compare with Conditional Generation? 3.3 When M Is Unknown, But Can Be Estimated Up to a Normalizing Constant on a Subset of Data, Are We Able to Sample from Our Target Distribution? 4 Conclusions and Future Work References Linearly Constrained Weights: Reducing Activation Shift for Faster Training of Neural Networks 1 Introduction 2 Activation Shift 3 Linearly Constrained Weights 3.1 Learning LCW via Reparameterization 3.2 LCW for Convolutional Layers 4 Variance Analysis 4.1 Variance Analysis of a Fully Connected Layer 4.2 Variance Analysis of a Nonlinear Activation Layer 4.3 Relationship to the Vanishing Gradient Problem 4.4 Example 5 Related Work 6 Experiments 6.1 Deep MLP with Sigmoid Activation Functions 6.2 Deep Convolutional Networks with ReLU Activation Functions 7 Conclusion References LYRICS: A General Interface Layer to Integrate Logic Inference and Deep Learning 1 Introduction 1.1 Previous Work 2 The Declarative Language 3 From Logic to Learning 4 Learning and Reasoning with Lyrics 5 Conclusions References Deep Eyedentification: Biometric Identification Using Micro-movements of the Eye 1 Introduction 2 Related Work 3 Problem Setting 4 Network Architecture 5 Experiments 5.1 Data Collection 5.2 Reference Methods 5.3 Hyperparameter Tuning 5.4 Hardware and Framework 5.5 Multi-class Classification 5.6 Identification and Verification 5.7 Assessing Session Bias 5.8 Additional Exploratory Experiments 6 Discussion 7 Conclusion References Adversarial Invariant Feature Learning with Accuracy Constraint for Domain Generalization 1 Introduction 2 Preliminary and Related Work 2.1 Problem Statement of Domain Generalization 2.2 Related Work 3 Our Approach 3.1 Domain Adversarial Networks 3.2 Trade-Off Caused by Domain-Class Dependency 3.3 Accuracy-Constrained Domain Invariance 3.4 Proposed Method 4 Experiments 4.1 Datasets 4.2 Baselines 4.3 Experimental Settings 4.4 Results 5 Conclusion References Quantile Layers: Statistical Aggregation in Deep Neural Networks for Eye Movement Biometrics 1 Introduction 2 Related Work 3 The Quantile Layer 4 Model Architectures 5 Empirical Study 5.1 Experimental Setup 5.2 Results 6 Conclusions References Multitask Hopfield Networks 1 Introduction 2 Methods 2.1 Problem Definition 2.2 Previous Singletask Model 2.3 Multitask Hopfield Networks 2.4 Model Complexity 3 Preliminary Results and Discussion 3.1 Benchmark Data 3.2 Evaluation Setting 3.3 Model Configuration 3.4 Model Performance 4 Conclusions References Meta-Learning for Black-Box Optimization 1 Introduction 2 Related Work 3 Problem Overview 4 RNN-Opt 4.1 RNN-Opt Without Domain Constraints 4.2 RNN-Opt with Domain Constraints (RNN-Opt-DC) 5 Experimental Evaluation 5.1 Observations 5.2 RNN-Opt with Domain Constraints 6 Conclusion and Future Work A Generating Diverse Non-convex Synthetic Functions References Training Discrete-Valued Neural Networks with Sign Activations Using Weight Distributions 1 Introduction 2 Related Work 3 Neural Networks and Weight Distributions 3.1 Discrete Neural Networks 3.2 Relation to Variational Inference 4 Approximation of the Expected Loss 4.1 Approximation of the Maximum Function 5 Model Details 5.1 Batch Normalization 5.2 Parameterization and Initialization of q 6 Experiments 6.1 Datasets 6.2 Classification Results 6.3 Ablation Study 7 Conclusion References Sobolev Training with Approximated Derivatives for Black-Box Function Regression with Neural Networks 1 Introduction 2 Sobolev Training with Approximated Target Derivatives 2.1 Target Derivative Approximation 2.2 Data Transformation 2.3 Error Functions 2.4 Derivative Approximation Using Finite-Differences 3 Results 3.1 Sobolev Training with Approximated Target Derivatives versus Value Training 3.2 Sobolev Training with Approximated Derivatives Based on Finite-Differences 3.3 Real-World Regression Problems 4 Conclusion References Hyper-Parameter-Free Generative Modelling with Deep Boltzmann Trees 1 Introduction 2 Notation and Background 2.1 Graphical Models 2.2 Deep Boltzmann Machines 3 Deep Boltzmann Trees 3.1 Learning the DBT Weights 4 Experiments 5 Conclusion References L0-ARM: Network Sparsification via Stochastic Binary Optimization 1 Introduction 2 Formulation 3 L0-ARM: Stochastic Binary Optimization 3.1 Choice of g() 3.2 Sparsifying Network Architectures for Inference 3.3 Imposing Shrinkage on Model Parameters theta 3.4 Group Sparsity Under L0 and L2 Norms 4 Related Work 5 Experimental Results 5.1 Implementation Details 5.2 MNIST Experiments 5.3 CIFAR Experiments 6 Conclusion References Learning with Random Learning Rates 1 Introduction 2 Related Work 3 Motivation and Outline 4 All Learning Rates at Once: Description 4.1 Notation 4.2 Alrao Architecture 4.3 Alrao Update for the Internal Layers: A Random Learning Rate for Each Unit 4.4 Alrao Update for the Output Layer: Model Averaging from Output Layers Trained with Different Learning Rates 5 Experimental Setup 5.1 Image Classification on ImageNet and CIFAR10 5.2 Other Tasks: Text Prediction, Reinforcement Learning 6 Performance and Robustness of Alrao 6.1 Alrao Compared to SGD with Optimal Learning Rate 6.2 Robustness of Alrao, and Comparison to Default Adam 6.3 Sensitivity Study to [_min;_max] 7 Discussion, Limitations, and Perspectives 8 Conclusion References FastPoint: Scalable Deep Point Processes 1 Introduction 2 Background 3 FastPoint: Scalable Deep Point Process 3.1 Generative Model 3.2 Sequential Monte Carlo Sampling 4 Related Work 5 Experiments 5.1 Model Performance 5.2 Sampling 6 Conclusion References Single-Path NAS: Designing Hardware-Efficient ConvNets in Less Than 4 Hours 1 Introduction 2 Related Work 3 Proposed Method: Single-Path NAS 3.1 Mobile ConvNets Search Space: A Novel View 3.2 Proposed Methodology: Single-Path NAS Formulation 3.3 Single-Path vs. Existing Multi-Path Assumptions 3.4 Hardware-Aware NAS with Differentiable Runtime Loss 4 Experiments 4.1 Experimental Setup 4.2 State-of-the-Art Runtime-Constrained ImageNet Classification 4.3 Ablation Study: Kernel-Based Accuracy-Efficiency Trade-Off 5 Conclusion References Probabilistic Models Scalable Large Margin Gaussian Process Classification 1 Introduction 2 Related Work 3 Large Margin Gaussian Process 3.1 Probabilistic Hinge Loss 3.2 Generalised Multi-class Hinge Loss 3.3 Scalable Variational Inference for LMGP 3.4 LMGP-DNN 4 Experimental Evaluation 4.1 Classification 4.2 Structured Data Classification 4.3 Image Classification with LMGP-DNN 4.4 Uncertainty Analysis 5 Conclusions References Integrating Learning and Reasoning with Deep Logic Models 1 Introduction 2 Model 2.1 MAP Inference 2.2 Learning 2.3 Mapping Constraints into a Continuous Logic 2.4 Potentials Expressing the Logic Knowledge 3 Related Works 4 Experimental Results 4.1 The PAIRS Artificial Dataset 4.2 Link Prediction in Knowledge Graphs 5 Conclusions and Future Work References Neural Control Variates for Monte Carlo Variance Reduction 1 Introduction 2 Control Variates 3 Neural Control Variates 4 Constrained Neural Control Variates 5 Experiments 5.1 Synthetic Data 5.2 Thermodynamic Integral for Bayesian Model Evidence Evaluation 5.3 Uncertainty Quantification in Bayesian Neural Network 6 Conclusion A Formulas for Goodwin Oscillator B Uncertainty Quantification in Bayesian Neural Network: Out-of-Bag Sample Detection References Data Association with Gaussian Processes 1 Introduction 2 Data Association with Gaussian Processes 3 Variational Approximation 3.1 Variational Lower Bound 3.2 Optimization of the Lower Bound 3.3 Approximate Predictions 3.4 Deep Gaussian Processes 4 Experiments 4.1 Noise Separation 4.2 Multimodal Data 4.3 Mixed Cart-Pole Systems 5 Conclusion References Incorporating Dependencies in Spectral Kernels for Gaussian Processes 1 Introduction 2 Background 3 Related Work 4 Dependencies Between SM Components 5 Generalized Convolution SM Kernels 6 Comparisons Between GCSM and SM 7 Scalable Inference 7.1 Hyper-parameter Initialization 8 Experiments 8.1 Compact Long Term Extrapolation 8.2 Modeling Irregular Long Term Decreasing Trends 8.3 Modeling Irregular Long Term Increasing Trends 8.4 Prediction with Large Scale Multidimensional Data 9 Conclusion References Deep Convolutional Gaussian Processes 1 Introduction 2 Background 2.1 Discrete Convolutions 2.2 Primer on Gaussian Processes 2.3 Variational Inference 3 Deep Convolutional Gaussian Process 3.1 Convolutional GP Layers 3.2 Final Classification Layer 3.3 Doubly Stochastic Variational Inference 3.4 Stochastic Gradient Hamiltonian Monte Carlo 4 Experiments 4.1 MNIST and CIFAR-10 Results 5 Conclusions References Bayesian Generalized Horseshoe Estimation of Generalized Linear Models 1 Introduction 1.1 Bayesian Generalized Linear Models 1.2 Generalized Horseshoe Priors 1.3 Our Contributions 2 Gradient-Based Samplers for Bayesian GLMs 2.1 Algorithm 1: mGrad-1 2.2 Algorithm 2: mGrad-2 2.3 Sampling the Intercept 2.4 Tuning the Step Size 2.5 Implementation Details 3 Two New Samplers for the Generalized Horseshoe 3.1 Inverse Gamma-Inverse Gamma Sampler 3.2 Rejection Sampling 4 Experimental Results 4.1 Comparison of GHS Hyperparameter Samplers 4.2 Comparison of Samplers for Coefficients 5 Summary References Fine-Grained Explanations Using Markov Logic 1 Introduction 2 Background 2.1 Markov Logic Networks 2.2 Related Work 3 Query Explanation 3.1 Sampling 4 Experiments 4.1 User Study Setup 4.2 Application 1: Review Spam Filter 4.3 Application 2: Review Sentiment Prediction 4.4 T-Test 5 Conclusion References Natural Language Processing Unsupervised Sentence Embedding Using Document Structure-Based Context 1 Introduction 2 Related Work 3 Document Structured-Based Context 3.1 Titles 3.2 Lists 3.3 Links 3.4 Window-Based Context (DWn) 4 Neural Network Models 4.1 Inter-sentential Dependency-Based Encoder-Decoder 4.2 Out-Of-Vocabulary (OOV) Mapping 5 Experiments 5.1 Dependency Importance 5.2 Target Sentence Prediction 5.3 Paraphrase Detection 5.4 Coreference Resolution 6 Conclusion and Future Work References Copy Mechanism and Tailored Training for Character-Based Data-to-Text Generation 1 Introduction 2 Model Description 2.1 Summary on Encoder-Decoder Architectures with Attention 2.2 Learning to Copy 2.3 Switching GRUs 3 Experiments 3.1 Datasets 3.2 Implementation Details 3.3 Results and Discussion 4 Conclusion References NSEEN: Neural Semantic Embedding for Entity Normalization 1 Introduction 2 Related Work 3 Approach 3.1 Similarity Learning 3.2 Reference Set Embedding and Storage 3.3 Retrieval 4 Experimental Validation 4.1 Reference Sets 4.2 Query Set 4.3 Baselines 4.4 Results 5 Discussion References Beyond Bag-of-Concepts: Vectors of Locally Aggregated Concepts 1 Introduction 2 Related Work 2.1 Bag-of-Words 2.2 Word Embeddings 2.3 Bag-of-Concepts 2.4 Vector of Locally Aggregated Descriptors (VLAD) 3 Vectors of Locally Aggregated Concepts (VLAC) 4 Experiments 4.1 Experimental Setup 4.2 Experiment 1 4.3 Experiment 2 5 Conclusion References A Semi-discriminative Approach for Sub-sentence Level Topic Classification on a Small Dataset 1 Introduction 2 Related Work 3 Data 3.1 Topic Separability 4 Methods 4.1 Emission Probabilities 4.2 Transition Probabilities 4.3 Decoding 5 Experiments and Results 5.1 MaxEnt as Baseline 5.2 Standard HMM 5.3 MaxEnt Emissions for HMM (ME+HMM) 5.4 Comparison of ME+HMM and CRF 6 Discussion 7 Conclusion and Future Work References Generating Black-Box Adversarial Examples for Text Classifiers Using a Deep Reinforced Model 1 Introduction 2 Related Work 3 Proposed Attack Strategy 3.1 Background and Notations 4 Adversarial Examples Generator (AEG) Architecture 4.1 Encoder 4.2 Decoder 5 Training 5.1 Supervised Pretraining with Teacher Forcing 5.2 Training with Reinforcement Learning 5.3 Training Details 6 Experiments 6.1 Setup 6.2 Quantitative Analysis 6.3 Human Evaluation 6.4 Ablation Studies 6.5 Qualitative Analysis 7 Conclusion References Author Index The three volume proceedings LNAI 11906 - 11908 constitutes the refereed proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2019, held in W©ơrzburg, Germany, in September 2019. The total of 130 regular papers presented in these volumes was carefully reviewed and selected from 733 submissions; there are 10 papers in the demo track. The contributions were organized in topical sections named as follows: Part I: pattern mining; clustering, anomaly and outlier detection, and autoencoders; dimensionality reduction and feature selection; social networks and graphs; decision trees, interpretability, and causality; strings and streams; privacy and security; optimization. Part II: supervised learning; multi-label learning; large-scale learning; deep learning; probabilistic models; natural language processing. Part III: reinforcement learning and bandits; ranking; applied data science: computer vision and explanation; applied data science: healthcare; applied data science: e-commerce, finance, and advertising; applied data science: rich data; applied data science: applications; demo track The three volume proceedings LNAI 11906 - 11908 constitutes the refereed proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2019, held in Wurzburg, Germany, in September 2019.The total of 130 regular papers presented in these volumes was carefully reviewed and selected from 733 submissions;