Learn Polish: A Comprehensive Guide to Learning Polish for Beginners, Including Grammar, Short Stories and 1000 Popular Phrases

The dramatic growth in practical applications for machine learning over the last ten years has been accompanied by many important developments in the underlying algorithms and techniques. For example, Bayesian methods have grown from a specialist niche to become mainstream, while graphical models have emerged as a general framework for describing and applying probabilistic techniques. The practical applicability of Bayesian methods has been greatly enhanced by the development of a range of approximate inference algorithms such as variational Bayes and expectation propagation, while new models based on kernels have had a significant impact on both algorithms and applications. This completely new textbook reflects these recent developments while providing a comprehensive introduction to the fields of pattern recognition and machine learning. It is aimed at advanced undergraduates or first-year PhD students, as well as researchers and practitioners. No previous knowledge of pattern recognition or machine learning concepts is assumed. Familiarity with multivariate calculus and basic linear algebra is required, and some experience in the use of probabilities would be helpful though not essential as the book includes a self-contained introduction to basic probability theory. The book is suitable for courses on machine learning, statistics, computer science, signal processing, computer vision, data mining, and bioinformatics. Extensive support is provided for course instructors, including more than 400 exercises, graded according to difficulty. Example solutions for a subset of the exercises are available from the book web site, while solutions for the remainder can be obtained by instructors from the publisher. The book is supported by a great deal of additional material, and the reader is encouraged to visit the book web site for the latest information. Christopher M. Bishop is Deputy Director of Microsoft Research Cambridge, and holds a Chair in Computer Science at the University of Edinburgh. He is a Fellow of Darwin College Cambridge, a Fellow of the Royal Academy of Engineering, and a Fellow of the Royal Society of Edinburgh. His previous textbook "Neural Networks for Pattern Recognition" has been widely adopted COVER 1 Preface 7 Mathematical notation 11 Contents 13 1. Introduction 21 1.1. Example: Polynomial Curve Fitting 24 1.2. Probability Theory 32 1.2.1 Probability densities 37 1.2.2 Expectations and covariances 39 1.2.3 Bayesian probabilities 41 1.2.4 The Gaussian distribution 44 1.2.5 Curve fitting re-visited 48 1.2.6 Bayesian curve fitting 50 1.3. Model Selection 52 1.4. The Curse of Dimensionality 53 1.5. Decision Theory 58 1.5.1 Minimizing the misclassification rate 59 1.5.2 Minimizing the expected loss 61 1.5.3 The reject option 62 1.5.4 Inference and decision 62 1.5.5 Loss functions for regression 66 1.6. Information Theory 68 1.6.1 Relative entropy and mutual information 75 Exercises 78 2. Probability Distributions 87 2.1. Binary Variables 88 2.1.1 The beta distribution 91 2.2. Multinomial Variables 94 2.2.1 The Dirichlet distribution 96 2.3. The Gaussian Distribution 98 2.3.1 Conditional Gaussian distributions 105 2.3.2 Marginal Gaussian distributions 108 2.3.3 Bayes’ theorem for Gaussian variables 110 2.3.4 Maximum likelihood for the Gaussian 113 2.3.5 Sequential estimation 114 2.3.6 Bayesian inference for the Gaussian 117 2.3.7 Student’s t-distribution 122 2.3.8 Periodic variables 125 2.3.9 Mixtures of Gaussians 130 2.4. The Exponential Family 133 2.4.1 Maximum likelihood and sufficient statistics 136 2.4.2 Conjugate priors 137 2.4.3 Noninformative priors 137 2.5. Nonparametric Methods 140 2.5.1 Kernel density estimators 142 2.5.2 Nearest-neighbour methods 144 Exercises 147 3. Linear Models for Regression 157 3.1. Linear Basis Function Models 158 3.1.1 Maximum likelihood and least squares 160 3.1.2 Geometry of least squares 163 3.1.3 Sequential learning 163 3.1.4 Regularized least squares 164 3.1.5 Multiple outputs 166 3.2. The Bias-Variance Decomposition 167 3.3. Bayesian Linear Regression 172 3.3.1 Parameter distribution 172 3.3.2 Predictive distribution 176 3.3.3 Equivalent kernel 179 3.4. Bayesian Model Comparison 181 3.5. The Evidence Approximation 185 3.5.1 Evaluation of the evidence function 186 3.5.2 Maximizing the evidence function 188 3.5.3 Effective number of parameters 190 3.6. Limitations of Fixed Basis Functions 192 Exercises 193 4. Linear Models for Classification 199 4.1. Discriminant Functions 201 4.1.1 Two classes 201 4.1.2 Multiple classes 202 4.1.3 Least squares for classification 204 4.1.4 Fisher’s linear discriminant 206 4.1.5 Relation to least squares 209 4.1.6 Fisher’s discriminant for multiple classes 211 4.1.7 The perceptron algorithm 212 4.2. Probabilistic Generative Models 216 4.2.1 Continuous inputs 218 4.2.2 Maximum likelihood solution 220 4.2.3 Discrete features 222 4.2.4 Exponential family 222 4.3. Probabilistic Discriminative Models 223 4.3.1 Fixed basis functions 224 4.3.2 Logistic regression 225 4.3.3 Iterative reweighted least squares 227 4.3.4 Multiclass logistic regression 229 4.3.5 Probit regression 230 4.3.6 Canonical link functions 232 4.4. The Laplace Approximation 233 4.4.1 Model comparison and BIC 236 4.5. Bayesian Logistic Regression 237 4.5.1 Laplace approximation 237 4.5.2 Predictive distribution 238 Exercises 240 5. Neural Networks 245 5.1. Feed-forward Network Functions 247 5.1.1 Weight-space symmetries 251 5.2. Network Training 252 5.2.1 Parameter optimization 256 5.2.2 Local quadratic approximation 257 5.2.3 Use of gradient information 259 5.2.4 Gradient descent optimization 260 5.3. Error Backpropagation 261 5.3.1 Evaluation of error-function derivatives 262 5.3.2 A simple example 265 5.3.3 Efficiency of backpropagation 266 5.3.4 The Jacobian matrix 267 5.4. The Hessian Matrix 269 5.4.1 Diagonal approximation 270 5.4.2 Outer product approximation 271 5.4.3 Inverse Hessian 272 5.4.4 Finite differences 272 5.4.5 Exact evaluation of the Hessian 273 5.4.6 Fast multiplication by the Hessian 274 5.5. Regularization in Neural Networks 276 5.5.1 Consistent Gaussian priors 277 5.5.2 Early stopping 279 5.5.3 Invariances 281 5.5.4 Tangent propagation 283 5.5.5 Training with transformed data 285 5.5.6 Convolutional networks 287 5.5.7 Soft weight sharing 289 5.6. Mixture Density Networks 292 5.7. Bayesian Neural Networks 297 5.7.1 Posterior parameter distribution 298 5.7.2 Hyperparameter optimization 300 5.7.3 Bayesian neural networks for classification 301 Exercises 304 6. Kernel Methods 311 6.1. Dual Representations 313 6.2. Constructing Kernels 314 6.3. Radial Basis Function Networks 319 6.3.1 Nadaraya-Watson model 321 6.4. Gaussian Processes 323 6.4.1 Linear regression revisited 324 6.4.2 Gaussian processes for regression 326 6.4.3 Learning the hyperparameters 331 6.4.4 Automatic relevance determination 332 6.4.5 Gaussian processes for classification 333 6.4.6 Laplace approximation 335 6.4.7 Connection to neural networks 339 Exercises 340 7. Sparse Kernel Machines 345 7.1. Maximum Margin Classifiers 346 7.1.1 Overlapping class distributions 351 7.1.2 Relation to logistic regression 356 7.1.3 Multiclass SVMs 358 7.1.4 SVMs for regression 359 7.1.5 Computational learning theory 364 7.2. Relevance Vector Machines 365 7.2.1 RVM for regression 365 7.2.2 Analysis of sparsity 369 7.2.3 RVM for classification 373 8. Graphical Models 379 8.1. Bayesian Networks 380 8.1.1 Example: Polynomial regression 382 8.1.2 Generative models 385 8.1.3 Discrete variables 386 8.1.4 Linear-Gaussian models 390 8.2. Conditional Independence 392 8.2.1 Three example graphs 393 8.2.2 D-separation 398 8.3. Markov Random Fields 403 8.3.1 Conditional independence properties 403 8.3.2 Factorization properties 404 8.3.3 Illustration: Image de-noising 407 8.3.4 Relation to directed graphs 410 8.4. Inference in Graphical Models 413 8.4.1 Inference on a chain 414 8.4.2 Trees 418 8.4.3 Factor graphs 419 8.4.4 The sum-product algorithm 422 8.4.5 The max-sum algorithm 431 8.4.6 Exact inference in general graphs 436 8.4.7 Loopy belief propagation 437 8.4.8 Learning the graph structure 438 Exercises 438 9. Mixture Models and EM 443 9.1. K-means Clustering 444 9.1.1 Image segmentation and compression 448 9.2. Mixtures of Gaussians 450 9.2.1 Maximum likelihood 452 9.2.2 EM for Gaussian mixtures 455 9.3. An Alternative View of EM 459 9.3.1 Gaussian mixtures revisited 461 9.3.2 Relation to K-means 463 9.3.3 Mixtures of Bernoulli distributions 464 9.3.4 EM for Bayesian linear regression 468 9.4. The EM Algorithm in General 470 Exercises 475 10. Approximate Inference 481 10.1. Variational Inference 482 10.1.1 Factorized distributions 484 10.1.2 Properties of factorized approximations 486 10.1.3 Example: The univariate Gaussian 490 10.1.4 Model comparison 493 10.2. Illustration: Variational Mixture of Gaussians 494 10.2.1 Variational distribution 495 10.2.2 Variational lower bound 501 10.2.3 Predictive density 502 10.2.4 Determining the number of components 503 10.2.5 Induced factorizations 505 10.3. Variational Linear Regression 506 10.3.1 Variational distribution 506 10.3.2 Predictive distribution 508 10.3.3 Lower bound 509 10.4. Exponential Family Distributions 510 10.4.1 Variational message passing 511 10.5. Local Variational Methods 513 10.6. Variational Logistic Regression 518 10.6.1 Variational posterior distribution 518 10.6.2 Optimizing the variational parameters 520 10.6.3 Inference of hyperparameters 522 10.7. Expectation Propagation 525 10.7.1 Example: The clutter problem 531 10.7.2 Expectation propagation on graphs 533 Exercises 537 11. Sampling Methods 543 11.1. Basic Sampling Algorithms 546 11.1.1 Standard distributions 546 11.1.2 Rejection sampling 548 11.1.3 Adaptive rejection sampling 550 11.1.4 Importance sampling 552 11.1.5 Sampling-importance-resampling 554 11.1.6 Sampling and the EM algorithm 556 11.2. Markov Chain Monte Carlo 557 11.2.1 Markov chains 559 11.2.2 The Metropolis-Hastings algorithm 561 11.3. Gibbs Sampling 562 11.4. Slice Sampling 566 11.5. The Hybrid Monte Carlo Algorithm 568 11.5.1 Dynamical systems 568 11.5.2 Hybrid Monte Carlo 572 11.6. Estimating the Partition Function 574 Exercises 576 12. Continuous Latent Variables 579 12.1. Principal Component Analysis 581 12.1.1 Maximum variance formulation 581 12.1.2 Minimum-error formulation 583 12.1.3 Applications of PCA 585 12.1.4 PCA for high-dimensional data 589 12.2. Probabilistic PCA 590 12.2.1 Maximum likelihood PCA 594 12.2.2 EM algorithm for PCA 597 12.2.3 Bayesian PCA 600 12.2.4 Factor analysis 603 12.3. Kernel PCA 606 12.4. Nonlinear Latent Variable Models 611 12.4.1 Independent component analysis 611 12.4.2 Autoassociative neural networks 612 12.4.3 Modelling nonlinear manifolds 615 Exercises 619 13. Sequential Data 625 13.1. Markov Models 627 13.2. Hidden Markov Models 630 13.2.1 Maximum likelihood for the HMM 635 13.2.2 The forward-backward algorithm 638 13.2.3 The sum-product algorithm for the HMM 645 13.2.4 Scaling factors 647 13.2.5 The Viterbi algorithm 649 13.2.6 Extensions of the hidden Markov model 651 13.3. Linear Dynamical Systems 655 13.3.1 Inference in LDS 658 13.3.2 Learning in LDS 662 13.3.3 Extensions of LDS 664 13.3.4 Particle filters 665 Exercises 666 14. Combining Models 673 14.1. Bayesian Model Averaging 674 14.2. Committees 675 14.3. Boosting 677 14.3.1 Minimizing exponential error 679 14.3.2 Error functions for boosting 681 14.4. Tree-based Models 683 14.5. Conditional Mixture Models 686 14.5.1 Mixtures of linear regression models 687 14.5.2 Mixtures of logistic models 690 14.5.3 Mixtures of experts 692 Exercises 694 Appendix A. Data Sets 697 Appendix B. Probability Distributions 705 Appendix C. Properties of Matrices 715 Appendix D. Calculus of Variations 723 Appendix E. Lagrange Multipliers 727 References 731 Index 749 Introduction. Example : Polynomial Curve Fitting ; Probability Theory ; Model Selection ; The Curse Of Dimensionality Decision Theory ; Information Theory -- Probability Distributions. Binary Vehicles ; Multinomial Variables ; The Gaussian Distribution ; The Exponential Family ; Nonparametric Methods -- Linear Models For Regression. Linear Basis Function Models ; The Bias-variance Decomposition ; Bayesian Linear Regression ; Bayesian Model Comparison ; The Evidence Approximation ; Limitations Of Fixed Basis Functions -- Linear Models For Classification. Discriminant Functions ; Probabilistic Generative Models ; Probabilistic Discrimitive Models ; The Laplace Approximation ; Bayesian Logistic Regression -- Neural Networks. Feed-forward Network Functions ; Network Training ; Error Backpropagation ; The Hessian Matrix ; Regularization In Neural Networks ; Mixture Density Networks ; Bayesian Neural Networks. Kernel Methods. Dual Representations ; Constructing Kernals ; Radial Basis Function Networks ; Gaussian Processes -- Sparse Kernel Machines. Maximum Margin Classifiers ; Relevance Vector Machines -- Graphical Models. Bayesian Networks ; Conditional Independence ; Markov Random Fields ; Inference In Graphical Models -- Mixture Models And Em. K-means Clustering ; Mixtures Of Gaussians ; An Alternative View Of Em ; The Em Algorithm In General -- Approximate Inference. Variational Inference ; Illustration : Variational Mixture Of Gaussians ; Variational Linear Regression ; Exponential Family Distributions ; Local Variational Methods ; Variational Logistic Regression ; Expectation Propagation -- Sampling Methods. Basic Sampling Algorithms ; Markov Chain Monte Carlo ; Gibbs Sampling ; Slice Sampling ; The Hybrid Monte Carlo Algorithm ; Estimating The Partition Function. Continuous Latent Variables. Principal Component Analysis ; Probabilistic Pca ; Kernel Pca ; Nonlinear Latent Variable Models -- Sequential Data. Markoc Models ; Hidden Markov Models ; Linear Dynamical Systems -- Combining Models. Bayesian Model Averaging ; Committees ; Boosting ; Tree-based Models ; Conditional Mixture Models -- Data Sets -- Probability Distributions -- Properties Of Matrices -- Calculus Of Variations -- Lagrange Multipliers. Christopher M. Bishop. Includes Bibliographical References (p. 711-728) And Index. Pattern recognition has its origins in engineering, whereas machine learning grew out of computer science. However, these activities can be viewed as two facets of the same field, and together they have undergone substantial development over the past ten years. In particular, Bayesian methods have grown from a specialist niche to become mainstream, while graphical models have emerged as a general framework for describing and applying probabilistic models. Also, the practical applicability of Bayesian methods has been greatly enhanced through the development of a range of approximate inference algorithms such as variational Bayes and expectation propagation. Similarly, new models based on kernels have had a significant impact on both algorithms and applications. This new textbook reflects these recent developments while providing a comprehensive introduction to the fields of pattern recognition and machine learning. It is aimed at advanced undergraduates or first-year PhD students, as well as researchers and practitioners, and assumes no previous knowledge of pattern recognition or machine learning concepts. Knowledge of multivariate calculus and basic linear algebra is required, and some familiarity with probabilities would be helpful though not essential as the book includes a self-contained introduction to basic probability theory. This is the first textbook on pattern recognition to present the Bayesian viewpoint. The book presents approximate inference algorithms that permit fast approximate answers in situations where exact answers are not feasible. It uses graphical models to describe probability distributions when no other books apply graphical models to machine learning. No previous knowledge of pattern recognition or machine learning concepts is assumed. Familiarity with multivariate calculus and basic linear algebra is required, and some experience in the use of probabilities would be helpful though not essential as the book includes a self-contained introduction to basic probability theory.