Fundamentals of Statistical Inference: What is the Meaning of Random Error? (SpringerBriefs in Applied Statistics and Econometrics)

This book provides a coherent description of foundational matters concerning statistical inference and shows how statistics can help us make inductive inferences about a broader context, based only on a limited dataset such as a random sample drawn from a larger population. By relating those basics to the methodological debate about inferential errors associated with p -values and statistical significance testing, readers are provided with a clear grasp of what statistical inference presupposes, and what it can and cannot do. To facilitate intuition, the representations throughout the book are as non-technical as possible. The central inspiration behind the text comes from the scientific debate about good statistical practices and the replication crisis. Calls for statistical reform include an unprecedented methodological warning from the American Statistical Association in 2016, a special issue “Statistical Inference in the 21st Century: A World Beyond p < 0.05” of The American Statistician in 2019, and a widely supported call to “Retire statistical significance” in Nature in 2019. The book elucidates the probabilistic foundations and the potential of sample-based inferences, including random data generation, effect size estimation, and the assessment of estimation uncertainty caused by random error. Based on a thorough understanding of those basics, it then describes the p -value concept and the null-hypothesis-significance-testing ritual, and finally points out the ensuing inferential errors. This provides readers with the competence to avoid ill-guided statistical routines and misinterpretations of statistical quantities in the future. Intended for readers with an interest in understanding the role of statistical inference, the book provides a prudent assessment of the knowledge gain that can be obtained from a particular set of data under consideration of the uncertainty caused by random error. More particularly, it offers an accessible resource for graduate students as well as statistical practitioners who have a basic knowledge of statistics. Last but not least, it is aimed at scientists with a genuine methodological interest in the above-mentioned reform debate. Preface 7 Contents 10 Abbreviations 13 Chapter 1: Introduction 14 Chapter 2: The Meaning of Scientific and Statistical Inference 22 2.1 The Starting Point: Errors and the Assessment of Validity 22 2.2 External Validity 24 2.3 Internal Validity 26 2.4 Chapter Summary: Scientific Inference Is More Than Statistical Inference 28 2.5 Recommended Reading 29 Chapter 3: The Basics of Statistical Inference: Simple Random Sampling 30 3.1 The Starting Point: Descriptive Statistics of a Given Dataset 30 3.2 Random Sampling, Sampling Error, and Sampling Distribution 32 3.3 Estimation and Estimation Uncertainty in Simple Random Sampling 35 3.3.1 Sample-Based Estimation of Effect Sizes and Standard Errors 35 3.3.2 An Illustrative Application: Gender Pay Gap 39 3.3.3 Sample-to-Sample Variability of Point and Standard Error Estimates 40 3.4 Chapter Summary: Statistical Assumptions Are Empirical Commitments 43 3.5 Recommended Reading 44 Chapter 4: Estimation Uncertainty in Complex Sampling Designs 45 4.1 Overview of Different Sampling Designs 45 4.2 Stratified Sampling 46 4.3 Cluster Sampling 48 4.4 Convenience Samples Contaminated by Selection Bias 51 4.4.1 Non-randomness: The Big Challenge in the Social Sciences 51 4.4.2 Approaches to Correct for Selection Bias in Convenience Samples 53 4.5 Full Populations and Finite Population Correction 56 4.6 Chapter Summary: Inference Requires Considering the Sampling Design 59 4.7 Recommended Reading 60 Chapter 5: Knowledge Accumulation Through Meta-analysis and Replications 61 5.1 The Basics of Meta-analysis 61 5.1.1 Dealing with Different Measurements and Model Specifications 61 5.1.2 Synthesizing Effect Sizes and Standard Errors Across Several Studies 65 5.2 Evaluation of the Quality of Research Through Replications 68 5.3 Chapter Summary: Our Best Estimators Estimate Correctly on Average 72 5.4 Recommended Reading 74 Chapter 6: The p-Value and Statistical Significance Testing 75 6.1 The p-Value Concept 75 6.2 Null-Hypothesis-Significance-Testing 80 6.2.1 Dichotomization of the p-Value and Significance Declarations 80 6.2.2 The Statistical Ritual ``NHST ́ ́ and Misinterpretations of Single Studies 82 6.2.3 Perpetuation of the Statistical Ritual ``NHST ́ ́ in Replication Studies 86 6.2.4 Malpractices and Publication Bias Associated with NHST 88 6.2.5 Approaches Aimed at Mitigating Publication Bias 93 6.3 The Historical Origins of the NHST-Framework 94 6.3.1 NHST: An Ill-bred Hybrid of Two Irreconcilable Statistical Approaches 94 6.3.2 Inductive Behavior (Hypothesis Testing) and Type I Error Rates α 96 6.3.3 Inductive Belief (Significance Testing) and p-Value Thresholds 103 6.4 Chapter Summary: Significance Declarations Should Be Avoided 105 6.5 Recommended Reading 107 Chapter 7: Statistical Inference in Experiments 109 7.1 Inferential Cases in Group Mean Comparisons 109 7.2 Causal Inference 111 7.2.1 Overview of Experimental Designs Aimed at Establishing Causality 111 7.2.2 The Uncertainty of Causal Effect Estimates Caused by Randomization 114 7.2.3 Inference in Random Assignment of Randomly Recruited Subjects 116 7.3 Inferences Without Randomization or Random Sampling 117 7.3.1 Fictitious Random Sampling 118 7.3.2 Fictitious Randomization 120 7.4 Chapter Summary: Causal Inference Is Different from Generalization 122 7.5 Recommended Reading 123 Chapter 8: Better Inference in the 21st Century: A World Beyond p < 0.05 124 References 129 Index 136