Regression for Categorical Data (Cambridge Series in Statistical and Probabilistic Mathematics, Series Number 34)

This book introduces basic and advanced concepts of categorical regression with a focus on the structuring constituents of regression, including regularization techniques to structure predictors. In addition to standard methods such as the logit and probit model and extensions to multivariate settings, the author presents more recent developments in flexible and high-dimensional regression, which allow weakening of assumptions on the structuring of the predictor and yield fits that are closer to the data. A generalized linear model is used as a unifying framework whenever possible in particular parametric models that are treated within this framework. Many topics not normally included in books on categorical data analysis are treated here, such as nonparametric regression; selection of predictors by regularized estimation procedures; ternative models like the hurdle model and zero-inflated regression models for count data; and non-standard tree-based ensemble methods. The book is accompanied by an R package that contains data sets and code for all the examples. Cover......Page 1 Regression for Categorical Data......Page 3 Title......Page 5 Copyright......Page 6 Contents......Page 7 Preface......Page 11 1.1.1 Some Examples......Page 13 Scale Levels: Nominal and Ordinal Variables......Page 16 1.2 Organization of This Book......Page 17 1.3.1 Structured Univariate Regression......Page 18 Structuring the Influential Term......Page 19 Linear Predictor......Page 20 Categorical Explanatory Variables......Page 21 Additive Predictor......Page 22 The Link between Covariates and Response......Page 23 1.3.3 Multivariate Regression......Page 24 Structuring the Influential Term......Page 25 1.3.4 Statistical Modeling......Page 26 Binary Explanatory Variables......Page 27 Multicategorical Explanatory Variables or Factors......Page 28 1.4.2 Linear Regression in Matrix Notation......Page 30 Least-Squares Estimation......Page 31 Properties of Estimates......Page 32 1.4.4 Residuals and Hat Matrix......Page 33 Case Deletion as Diagnostic Tool......Page 34 1.4.5 Decomposition of Variance and Coefficient of Determination......Page 35 1.4.6 Testing in Multiple Linear Regression......Page 37 Submodels and the Testing of Linear Hypotheses......Page 38 1.5 Exercises......Page 39 2.1.1 Single Binary Variables......Page 41 Odds, Logits, and Odds Ratios......Page 43 Comparing Two Groups......Page 44 2.2.1 Deficiencies of Linear Models......Page 45 Binary Responses as Dichotomized Latent Variables......Page 46 Modeling the Common Distribution of a Binary and a Continuous Distribution......Page 47 Basic Form of Binary Regression Models......Page 48 2.3.2 Logit Model with Continuous Predictor......Page 49 Multivariate Predictor......Page 53 2.3.3 Logit Model with Binary Predictor......Page 54 Logit Model with (0-1)-Coding of Covariates......Page 55 Logit Model with Effect Coding......Page 56 Logit Model with (0-1)-Coding......Page 57 Logit Model with Effect Coding......Page 58 Logit Model with Several Categorical Predictors......Page 59 2.4 The Origins of the Logistic Function and the Logit Model......Page 60 2.5 Exercises......Page 61 3.1 Basic Structure......Page 63 3.2.2 Exponential Distribution......Page 65 3.2.3 Gamma-Distributed Responses......Page 66 3.2.4 Inverse Gaussian Distribution......Page 67 3.3.1 Models for Binary Data......Page 68 3.3.2 Models for Binomial Data......Page 69 3.3.3 Poisson Model for Count Data......Page 70 3.3.4 Negative Binomial Distribution......Page 71 3.4.1 Means and Variances......Page 72 3.4.2 Canonical Link......Page 73 3.5 Modeling of Grouped Data......Page 74 Log-Likelihood and Score Function......Page 75 Information Matrix......Page 77 3.7.1 The Deviance......Page 79 Analysis of Deviance......Page 81 3.7.3 Alternative Test Statistics for Linear Hypotheses......Page 83 3.8.1 The Deviance for Grouped Observations......Page 84 3.8.2 Pearson Statistic......Page 86 3.9 Computation of Maximum Likelihood Estimates......Page 87 3.10 Hat Matrix for Generalized Linear Models......Page 88 3.11 Quasi-Likelihood Modeling......Page 90 3.13 Exercises......Page 91 Chapter 4 Modeling of Binary Data......Page 93 4.1 Maximum Likelihood Estimation......Page 94 Single Binary Responses......Page 95 Grouped Data: Estimation for Binomially Distributed Responses......Page 96 Estimation Conditioned on Predictor Values......Page 97 4.2.1 The Deviance......Page 99 Deviance as Goodness-of-Fit Statistic......Page 101 4.2.2 Pearson Statistic......Page 102 Alternative Tests......Page 104 4.3.1 Residuals......Page 105 4.3.2 Hat Matrix and Influential Observations......Page 108 4.3.3 Case Deletion......Page 109 4.4.1 The Linear Predictor: Continuous Predictors, Factors, and Interactions......Page 113 The Crossing Operator......Page 115 4.4.2 Testing Components of the Linear Predictor......Page 116 4.4.3 Ordered Categorical Predictors......Page 120 4.5 Comparing Non-Nested Models......Page 125 4.6 Explanatory Value of Covariates......Page 126 4.6.1 Measures of Residual Variation......Page 127 4.6.3 Correlation-Based Measures......Page 129 4.7 Further Reading......Page 131 4.8 Exercises......Page 132 Probit Model......Page 135 Complementary Log-Log Model and Log-Log Model......Page 136 Exponential Model......Page 137 Cauchy Model......Page 138 5.1.2 Comparing Link Functions......Page 139 5.1.3 Choice between Models and Advantages of Logit Models......Page 141 Parametric Families of Link Functions......Page 142 Non-Parametric Fitting of Link Functions......Page 143 5.3 Overdispersion......Page 144 Unobserved Heterogeneity......Page 145 5.3.2 Beta-Binomial Model......Page 146 5.3.3 Generalized Estimation Functions and Quasi-Likelihood......Page 147 The Hypergeometric Distribution......Page 150 5.6 Exercises......Page 152 Chapter 6 Regularization and Variable Selection for Parametric Models......Page 155 6.1 Classical Subset Selection......Page 156 6.2 Regularization by Penalization......Page 157 6.2.1 Ridge Regression......Page 159 6.2.2 L1-Penalty: The Lasso......Page 161 The Adaptive Lasso......Page 164 Categorical Predictors and the Group Lasso......Page 165 6.2.3 The Elastic Net......Page 166 6.2.4 Alternative Estimators with Grouping Effect......Page 167 OSCAR......Page 168 Correlation-Based Penalties......Page 169 6.2.5 SCAD......Page 171 6.2.6 The Dantzig Selector......Page 174 6.3.1 Boosting for Linear Models......Page 175 6.3.2 Boosting for Generalized Linear Models......Page 177 Blockwise Boosting......Page 178 6.4 Simultaneous Selection of Link Function and Predictors......Page 182 6.5.1 Selection within Categorical Predictors......Page 185 Ordered Categories......Page 186 6.5.2 Selection of Variables Combined with Clustering of Categories......Page 187 6.5.3 Selection of Variables......Page 188 6.6 Bayesian Approach......Page 190 6.8 Exercises......Page 191 Chapter 7 Regression Analysis of Count Data......Page 193 7.1 The Poisson Distribution......Page 194 Poisson Distribution as the Law of Rare Events......Page 195 Poisson Process......Page 196 7.2 Poisson Regression Model......Page 197 Deviance and Goodness-of-Fit......Page 198 Testing of Hierarchical Models......Page 199 7.4 Poisson Regression with an Offset......Page 202 Model with Overdispersion Parameter......Page 204 Alternative Variance Functions......Page 205 7.6 Negative Binomial Model and Alternatives......Page 206 Negative Binomial Model as Gamma-Poisson-Model......Page 207 7.7 Zero-Inflated Counts......Page 210 7.8 Hurdle Models......Page 212 7.9 Further Reading......Page 215 7.10 Exercises......Page 216 Chapter 8 Multinomial Response Models......Page 219 8.1 The Multinomial Distribution......Page 221 8.2 The Multinomial Logit Model......Page 222 Side Constraints......Page 224 8.4 Structuring the Predictor......Page 227 8.5 Logit Model as Multivariate Generalized Linear Model......Page 229 8.6.1 Maximum Likelihood Estimation......Page 230 Separate Fitting of Binary Models......Page 231 Pearson Statistic......Page 232 Power-Divergence Family......Page 233 Residuals......Page 234 8.7 Multinomial Models with Hierarchically Structured Response......Page 235 Random Utility Models......Page 238 Independence from Irrelevant Alternatives......Page 240 Pair Comparison Models......Page 241 8.9 Nested Logit Model......Page 243 Ridge-Type Penalties......Page 245 8.11 Further Reading......Page 250 8.12 Exercises......Page 251 Chapter 9 Ordinal Response Models......Page 253 9.1.1 Simple Cumulative Model......Page 255 Cumulative Extreme Value Models......Page 258 Probit Model......Page 260 9.1.3 General Cumulative Models......Page 261 9.1.4 Testing the Proportional Odds Assumption......Page 263 9.2 Sequential Models......Page 264 9.2.1 Basic Model......Page 265 9.3.2 Equivalence of Cumulative and Sequential Models......Page 267 9.3.3 Cumulative Models versus Sequential Models......Page 268 9.4.2 Hierarchically Structured Models......Page 269 9.4.5 Adjacent Categories Logits......Page 272 9.5.1 Cumulative Models......Page 273 Maximum Likelihood Estimation by Binary Response Models......Page 275 9.7 Exercises......Page 277 10.1 Univariate Generalized Non-Parametric Regression......Page 281 10.1.1 Regression Splines and Basis Expansions......Page 282 Regression Splines: Truncated Power Series Basis......Page 283 Representation by B-Splines......Page 284 Alternative Basis Functions......Page 285 10.1.2 Smoothing Splines......Page 286 10.1.3 Penalized Estimation......Page 288 Penalized Splines......Page 289 Maximizing the Penalized Likelihood......Page 290 Effective Degrees of Freedom of a Smoother......Page 291 10.1.4 Local Regression......Page 293 Cross-Validation......Page 295 Likelihood-Based Approaches......Page 296 10.2.1 Expansion in Basis Functions......Page 297 10.2.2 Smoothing Splines......Page 298 10.2.3 Penalized Regression Splines......Page 299 10.2.4 Local Estimation in Two Dimensions......Page 300 10.3 Structured Additive Regression......Page 301 Penalized Regression Splines......Page 302 Selection of Smoothing Parameters......Page 303 Simultaneous Selection of Variables and Amount of Smoothing......Page 304 Univariate Smoothers and the Backfitting Algorithm......Page 306 10.3.2 Extension to Multicategorical Response......Page 310 Varying-Coefficients Models......Page 312 Continuous-by-Continuous Interactions......Page 313 Estimation......Page 314 10.3.4 Structured Additive Regression Modeling......Page 316 10.4 Functional Data and Signal Regression......Page 319 10.4.1 Functional Model for Univariate Response......Page 320 10.4.3 Penalized Signal Regression......Page 321 10.4.5 Feature Extraction in Signal Regression......Page 322 10.5 Further Reading......Page 325 10.6 Exercises......Page 326 11.1 Regression and Classification Trees......Page 329 Test-Based Splits......Page 331 Dichotomous Responses......Page 332 Multicategorical Response......Page 333 Splitting by Impurity Measures......Page 334 11.1.3 Size of a Tree......Page 336 11.1.4 Advantages and Disadvantages of Trees......Page 337 Random Forests......Page 338 Maximally Selected Statistics......Page 339 11.2 Multivariate Adaptive Regression Splines......Page 340 11.4 Exercises......Page 341 Chapter 12 The Analysis of Contingency Tables: Log-Linear and Graphical Models......Page 343 12.1 Types of Contingency Tables......Page 344 Type 3: Product-Multinomial Distribution (Treatment and Pain)......Page 345 Multinomial and Product-Multinomial Distributions......Page 346 12.2 Log-Linear Models for Two-Way Tables......Page 347 12.3 Log-Linear Models for Three-Way Tables......Page 350 Type 3: Product-Multinomial Distribution......Page 351 Type 0: Saturated Model......Page 353 Type 2: Only Two Two-Factor Interactions Contained......Page 354 Type 4: Main Effects Model......Page 356 12.5 Log-Linear and Graphical Models for Higher Dimensions......Page 357 Graphical Models......Page 358 12.6 Collapsibility......Page 360 Logit Models with Selected Response Variables......Page 361 12.8.1 Maximum Likelihood Estimates and Minimal Sufficient Statistics......Page 362 12.8.2 Testing and Goodness-of-Fit......Page 365 12.9 Model Selection and Regularization......Page 366 12.10 Mosaic Plots......Page 369 12.11 Further Reading......Page 370 12.12 Exercises......Page 371 Chapter 13 Multivariate Response Models......Page 375 Subject-Specific Models......Page 376 13.1.1 Transition Models and Response Variables with a Natural Order......Page 377 Estimation......Page 378 13.1.2 Symmetric Conditional Models......Page 380 13.2 Marginal Parametrization and Generalized Log-Linear Models......Page 382 13.3 General Marginal Models: Association as Nuisance and GEEs......Page 383 Working Correlation Matrices......Page 385 Specification by Odds Ratios......Page 387 13.3.1 Generalized Estimation Approach......Page 388 Structured Correlation Model......Page 389 Asymptotic Properties and Extensions......Page 390 Types of Covariates and Loss of Efficiency......Page 393 13.3.2 Marginal Models for Multinomial Responses......Page 394 13.3.3 Penalized GEE Approaches......Page 395 13.3.4 Generalized Additive Marginal Models......Page 396 13.4 Marginal Homogeneity......Page 397 13.4.1 Marginal Homogeneity for Dichotomous Outcome......Page 398 Likelihood Ratio Statistic and Alternatives......Page 399 13.4.2 Regression Approach to Marginal Homogeneity......Page 400 Conditional Logit Models......Page 401 Conditional Maximum Likelihood Estimation......Page 402 13.4.3 Marginal Homogeneity for Multicategorical Outcome......Page 403 13.5 Further Reading......Page 404 13.6 Exercises......Page 405 Chapter 14 Random Effects Models and Finite Mixtures......Page 407 14.1.1 Random Effects for Clustered Data......Page 408 Random Intercept Model......Page 409 14.1.2 General Linear Mixed Model......Page 410 Maximum Likelihood......Page 411 Best Linear Unbiased Prediction (BLUP)......Page 412 Logistic-Normal Model......Page 414 Probit-Normal Model and Alternatives......Page 416 14.2.3 Generalized Linear Mixed Models for Clustered Data......Page 417 Random Slopes......Page 418 14.3.1 Marginal Maximum Likelihood Estimation by Integration Techniques......Page 419 Indirect Maximization Based on the EM Algorithm......Page 421 Motivation as Posterior Mode Estimator......Page 423 Solution of the Penalized Likelihood Problem......Page 424 14.3.4 Estimation of Variances......Page 425 Error Approximation......Page 426 14.3.5 Bayesian Approaches......Page 427 Ordered Response Categories......Page 428 Mixed Multinomial Logit Model......Page 430 14.5 The Marginalized Random Effects Model......Page 431 14.7 Semiparametric Mixed Models......Page 432 14.8 Finite Mixture Models......Page 434 Estimation......Page 435 14.9 Further Reading......Page 438 14.10 Exercises......Page 439 Chapter 15 Prediction and Classification......Page 441 15.1 Basic Concepts of Prediction......Page 442 Estimated Prediction Rule......Page 443 15.1.1 Squared Error Loss......Page 444 15.1.2 Discrete Data......Page 445 Direct Prediction......Page 446 Prediction Based on Estimated Probabilities......Page 447 Loss for Univariate Discrete Response......Page 449 15.2.1 Bayes Rule and the Minimization of the Rate of Misclassification......Page 450 15.2.2 Classification with Discriminant Functions......Page 452 15.2.3 Discrimination with Normally Distributed Predictors......Page 454 15.2.4 Bayes Rule for General Loss Functions......Page 456 15.3.1 Samples and Error Rates......Page 457 Empirical Error Rates......Page 458 Receiver Operating Characteristic Curves (ROC Curves)......Page 460 Plug-In Rules......Page 463 Fisher’s Discriminant Analysis......Page 464 Quadratic Discrimination......Page 465 Regularized Discriminant Analysis......Page 466 15.4.3 Linear Separation and Support Vector Classifiers......Page 467 15.5.1 Nearest Neighborhood Methods......Page 469 15.5.2 Random Forests and Ensemble Methods......Page 470 Early Boosting Approaches: AdaBoost......Page 472 Functional Gradient Descent Boosting......Page 473 Likelihood-Based Boosting......Page 478 Multiple Category Case......Page 479 15.6.1 Feed-Forward Networks......Page 480 15.6.2 Radial Basis Function Networks......Page 482 15.7 Examples......Page 483 15.8 Variable Selection in Classification......Page 485 15.9 Prediction of Ordinal Outcomes......Page 486 15.9.1 Ordinal Response Models......Page 487 15.9.2 Aggregation over Binary Splits......Page 488 15.10 Model-Based Prediction......Page 492 15.11 Further Reading......Page 493 15.12 Exercises......Page 494 A.1.3 Negative Binomial Distribution......Page 497 A.1.6 Multinomial Distribution......Page 498 A.2.4 Gompertz or Minimum Extreme Value Distribution......Page 499 A.2.8 Dirichlet Distribution......Page 500 A.2.9 Beta Distribution......Page 501 Derivatives......Page 502 Univariate Version......Page 503 Taylor Approximation and the Asymptotic Covariance of ML Estimates......Page 504 B.3 Conditional Expectation, Distribution......Page 505 B.4 EM Algorithm......Page 506 C.1 Simplification of Penalties......Page 508 Simplifying the Penalty by Reparameterization......Page 509 C.2 Linear Constraints......Page 510 C.3 Fisher Scoring with Penalty Term......Page 511 D.1 Kullback-Leibler Distance......Page 512 D.1.2 Kullback-Leibler and Discrete Distributions......Page 513 D.1.3 Kullback-Leibler in Generalized Linear Models......Page 514 D.1.4 Decomposition......Page 515 E.1 Laplace Approximation......Page 516 E.2 Gauss-Hermite Integration......Page 517 E.2.1 Multivariate Gauss-Hermite Integration......Page 518 E.3 Inversion of Pseudo-Fisher Matrix......Page 519 List of Examples......Page 521 Bibliography......Page 525 Author Index......Page 557 Subject Index......Page 566 This book introduces basic and advanced concepts of categorical regression with a focus on the structuring constituents of regression, including regularization techniques to structure predictors. In addition to standard methods such as the logit and probit model and extensions to multivariate settings, the author presents more recent developments in flexible and high-dimensional regression, which allow weakening of assumptions on the structuring of the predictor and yield fits that are closer to the data. A generalized linear model is used as a unifying framework whenever possible in particular parametric models that are treated within this framework. Many topics not normally included in books on categorical data analysis are treated here, such as nonparametric regression; selection of predictors by regularized estimation procedures; ternative models like the hurdle model and zero-inflated regression models for count data; and non-standard tree-based ensemble methods, which provide excellent tools for prediction and the handling of both nominal and ordered categorical predictors. The book is accompanied an R package that contains data sets and code for all the examples. "Categorical data play an important role in many statistical analyses. They appear whenever the outcomes of one or more categorical variables are observed. A categorical variable can be seen as a variable for which the possible values form a set of categories, which can be finite or, in the case of count data, infinite. These categories can be records of answers (yes/no) in a questionnaire, diagnoses like normal/abnormal resulting from a medical examination or choices of brands in consumer behaviour. Data of this type are common in all sciences that use quantitative research tools, for example social sciences, economics, biology, genetics and medicine, but also engineering and agriculture. In some applications all of the observed variables are categorical and the resulting data can be summarized in contingency tables which contain the counts for combinations of possible outcomes. In other applications categorical data are collected together with continuous variables and one wants to investigate the dependence of one or more categorical variables on continuous and/or categorical variables"-- Provided by publisher "Categorical data play an important role in many statistical analyses. They appear whenever the outcomes of one or more categorical variables are observed. A categorical variable can be seen as a variable for which the possible values form a set of categories, which can be finite or, in the case of count data, infinite. These categories can be records of answers (yes/no) in a questionnaire, diagnoses like normal/abnormal resulting from a medical examination or choices of brands in consumer behaviour. Data of this type are common in all sciences that use quantitative research tools, for example social sciences, economics, biology, genetics and medicine, but also engineering and agriculture. In some applications all of the observed variables are categorical and the resulting data can be summarized in contingency tables which contain the counts for combinations of possible outcomes. In other applications categorical data are collected together with continuous variables and one wants to investigate the dependence of one or more categorical variables on continuous and/or categorical variables"-- Résumé de l'éditeur