Objective Bayesian Inference Hb

Bayesian analysis is today understood to be an extremely powerful method of statistical analysis, as well an approach to statistics that is particularly transparent and intuitive. It is thus being extensively and increasingly utilized in virtually every area of science and society that involves analysis of data. A widespread misconception is that Bayesian analysis is a more subjective theory of statistical inference than what is now called classical statistics. This is true neither historically nor in practice. Indeed, objective Bayesian analysis dominated the statistical landscape from roughly 1780 to 1930, long before 'classical' statistics or subjective Bayesian analysis were developed. It has been a subject of intense interest to a multitude of statisticians, mathematicians, philosophers, and scientists. The book, while primarily focusing on the latest and most prominent objective Bayesian methodology, does present much of this fascinating history. The book is written for four different audiences. First, it provides an introduction to objective Bayesian inference for non-statisticians; no previous exposure to Bayesian analysis is needed. Second, the book provides an overview of the development and current state of objective Bayesian analysis and its relationship to other statistical approaches, for those with interest in the philosophy of learning from data. Third, the book presents a careful development of the particular objective Bayesian approach that we recommend, the reference prior approach. Finally, the book presents as much practical objective Bayesian methodology as possible for statisticians and scientists primarily interested in practical applications. Contents Preface 0 Overview and Notation 0.1 Overview 0.2 Probabilistic Notation and Distributions 0.3 Statistical Notation and Models I The Objective Bayesian Paradigm 1 Basics of Bayesian Analysis 1.1 Measuring Conditional Uncertainty 1.2 Parametric Inference 1.2.1 The Learning Process 1.2.2 The Likelihood Principle 1.2.3 Sufficiency 1.2.4 Conjugate Priors 1.2.5 Sequential Updating 1.2.6 Sensitivity Analysis 1.2.7 Nuisance Parameters 1.2.8 Restricted Parameter Spaces 1.3 Inference Summaries 1.3.1 Point Estimation 1.3.2 Credible Regions 1.4 Prediction 1.4.1 Posterior Predictive Distributions 1.4.2 Prediction in Regression 1.5 Hierarchical Modeling and Analysis 1.5.1 Hierarchical Models 1.5.2 Hierarchical Bayesian Analysis 1.6 Decision Making 1.6.1 Definition of a Decision Problem and its Analysis 1.6.2 Formal Point Estimation 1.6.3 Formal Region Estimation 1.6.4 Kullback-Leibler (KL) Divergence 1.6.5 Intrinsic Loss 1.7 Large Sample Behavior 1.7.1 Discrete Parameters 1.7.2 Continuous Parameters 1.8 Generating Samples from the Posteriors 1.8.1 Simulation via the Metropolis-Hastings Algorithm 1.8.2 Importance Sampling 1.8.3 Simulation via Gibbs Sampling 1.8.4 Simulation via Rejection Sampling 1.8.5 Ratio-of-Uniforms Method 1.8.6 Constructive Random Posteriors 1.9 Discussion and Additional References 1.9.1 Formal Definition of Probability 1.9.2 Normative Approach to Foundations 1.9.3 Bayes Theorem 2 Basics of Objective Bayesian Analysis 2.1 Motivation for Objective Bayesian Methods 2.1.1 Ease of Interpretation 2.1.2 Objectivity and Conventionality 2.1.3 Conditioning and Frequentist Performance 2.1.4 General Methodological Advantages 2.1.5 Objective and Subjective Bayesian Analysis 2.2 Informal Objective Bayesian Solutions 2.2.1 Truncation of the Parameter Space 2.2.2 Vague Proper Priors 2.2.3 Weakly Informative Priors 2.2.4 Transformation of the Parameter 2.2.5 Data-Dependent Priors 2.3 Permissible Improper Priors 2.3.1 Introduction 2.3.2 Kullback-Leibler (KL) Convergence 2.3.3 Justifying Posteriors from Improper Priors 2.3.4 Permissible Priors 2.4 Discussion and Additional References 2.4.1 Conditioning 2.4.2 Frequentist Performance 2.4.3 Informal Objective Bayesian Procedures II Select Early Objective Bayesian Developments 3 The Constant Prior 3.1 Bayes 3.2 Laplace 3.3 Inverse Probability 3.4 The Constant Prior after 1950 3.5 Discussion and Additional References 3.5.1 The Historial Controversy on Inverse Probability 3.5.2 Modern Advocates of the Use of a Constant Prior 4 Jeffreys-Rule Priors 4.1 Development 4.2 The Single Parameter Case 4.3 The Multi-parameter Case 4.4 Discussion and Additional References 4.4.1 The Extensive Use of Jeffreys-Rule Priors 4.4.2 Alternative Derivations of the Jeffreys-Rule Prior 4.4.3 Coherence under Reparameterization 5 Frequentist Matching 5.1 Frequentist Matching Priors 5.1.1 Exact Frequentist Matching 5.1.2 Asymptotic Frequentist Matching 5.2 Confidence Distributions 5.3 Fiducial Inference 5.3.1 Traditional Fiducial Inference 5.3.2 Generalized Fiducial Inference 5.4 Discussion and Additional References 6 Invariance Priors 6.1 Introduction 6.2 Groups of Transformations 6.3 Invariant Models 6.4 Invariance Priors 6.4.1 One-Dimensional Parameter 6.4.2 Higher Dimensional Parameters 6.4.3 The Right-Haar Prior is Exact Matching 6.4.4 The Triangular Group 6.5 Discussion and Additional References 7 Evaluating Objective Priors 7.1 Propriety of Posteriors 7.2 Overly Diffuse Priors and Inconsistency 7.3 Overly Concentrated Objective Priors 7.4 Importance of the Quantity of Interest 7.5 Coherence 7.5.1 Invariance to Parameterization 7.5.2 Marginalization Paradoxes 7.5.3 Compatibility with the Likelihood Principle 7.6 Frequentist Validation 7.6.1 Matching 7.6.2 Decision-Theoretic Evaluation and Admissbility 7.7 Computational Issues 7.7.1 Deriving the Objective Prior 7.7.2 Utilizing the Objective Prior in Computations 7.8 Discussion and Additional References 7.8.1 Summary of Comparisons of Methods 7.8.2 Other Objective Bayesian Approaches III Reference Analysis 8 Introduction to Part III 8.1 Introduction 8.2 Importance of the Quantity of Interest 8.3 Maximizing Missing Information 8.4 Determining the Reference Prior Through Iteration 8.5 Discrete Parameter Space Illustrations 8.6 Discussion and Additional References 8.6.1 History of the Reference Prior Approach 9 Models with One Continuous Parameter 9.1 Formal Definition of the Reference Prior 9.2 Properties of Reference Priors 9.3 Existence of the Expected Information 9.4 An Explicit Expression for the Reference Prior 9.5 When There is a Known Asymptotic Distribution 9.6 Irregular Continuous Problems 9.7 Numerical Reference Priors 9.8 Discussion and Additional References 10 Multiple Continuous Parameters 10.1 The Challenge of Many Parameters 10.2 Iterative Algorithm for One Nuisance Parameter 10.2.1 Definition of the Reference Prior and Examples 10.2.2 The Asymptotic Reference Prior 10.2.3 Reference Priors under Asymptotic Bi-Normality 10.3 Multiple Nuisance Parameters 10.3.1 Recommended Reference Prior under Asymptotic Normality 10.3.2 Multinomial Distribution 10.3.3 Bivariate Normal Distribution 10.3.4 Prediction 10.3.5 Grouped Reference Priors 10.4 Discussion and Additional References 10.4.1 Reverse Reference Priors 11 Discrete Parameter Problems 11.1 Introduction 11.2 Possible Embeddings 11.2.1 Approach 1: Just Assume is Continuous 11.2.2 Approach 2: Introducing a Continuous Hierarchical Hyperparameter 11.2.3 Approach 3: Applying Reference Prior Theory with a Consistent Estimator 11.2.4 Approach 4: Using Parameter-Based Asymptotics 11.2.5 Preference among Embeddings 11.3 Estimating a Population Size 11.3.1 Background of the Model 11.3.2 Using Approach 1 to Determine an Objective Prior 11.4 The Hypergeometric Distribution 11.4.1 Using Approach 2 to Determine an Objective Prior 11.4.2 Laplace's Rule of Succession 11.5 The Binomial-Beta Distribution 11.5.1 The Model and Likelihood 11.5.2 Using Approach 3 to Determine an Objective Prior 11.5.3 Comparison with 1/n 11.6 The Binomial Sample Size 11.6.1 Known p 11.6.2 Unknown p 11.7 Prior Model Probabilities 11.7.1 Finite Number of Models with No Structure 11.7.2 Variable Selection 12 Overall Objective Priors 12.1 Introduction 12.1.1 The Problem 12.1.2 Background 12.1.3 Outline of the Chapter 12.2 When the Reference Prior is Common 12.3 Prior Averaging Approach 12.4 Reference Distance Approach 12.4.1 Introduction 12.4.2 Multinomial Example 12.5 Hierarchical Reference Approach 12.5.1 Introduction 12.5.2 The Multinomial Problem 12.6 Hierarchical Normal Models 12.6.1 Introduction 12.6.2 The Hierarchical Model Considered 12.6.3 The Recommended Prior 12.6.4 Covariate Transformations 12.6.5 Computation with the Posterior 12.6.6 Decision-theoretic Justification for the Prior 12.6.7 Performance of the Recommended Prior 12.6.8 Posterior Propriety and Use for General Hierarchical Models 12.7 Discussion and Additional References 13 Reference Priors with Partial Information 13.1 Introduction 13.2 Constrained Parameter Space 13.3 Linear Constraints on the Prior 13.4 A Known Conditional Prior 13.5 A Known Marginal Prior 13.6 Knowing Independence of Parameters 13.7 Discussion and Additional References 13.7.1 Maximum Entropy Priors 14 Models with Special Structures 14.1 A Two Parameter Exponential Family 14.2 Fisher Information of Certain Patterns 14.3 Star-Shaped Models 14.3.1 Definition of Star-Shaped Models 14.3.2 Objective Priors for Star-Shaped Models 14.3.3 Posterior Distributions and Estimates 14.4 Reference Priors for Sequential Experiments 14.4 Reference Priors for Sequential Experiments 14.4.1 Objective Priors with a Given Stopping Rule 14.4.2 Reference Priors under Asymptotic Normality 14.4.3 Computation 14.4.4 Modified Reference Priors 14.5 Discussion and Additional References 15 A Catalog of Objective Priors A Common Distributions A.1 Univariate Discrete Distributions A.2 Univariate Continuous Distributions A.3 Multivariate Discrete Distributions A.4 Multivariate Continuous Distributions Bibliography Author Index Index