Exponential families are parametric sets of probability distributions that arise in many applications. These include well-known univariate distributions (such as the binomial, Poisson, geometric, exponential, and normal distributions), but also multi-variate distributions like probabilistic graphical models and stationary distributions of several queueing models. In this presentation, we will first recall the definition of exponential families and motivate their study. In a second time, we will present a generic method for approximating the normalization constant of these distributions, as the exact calculation of this constant is practically infeasible in high dimension.
References:
- M. J. Wainwright and M. I. Jordan. Graphical Models, Exponential Families, and Variational Inference. Foundations and Trends® in Machine Learning 1.1 (2008), https://people.eecs.berkeley.edu/~wainwrig/Papers/WaiJor08_FTML.pdf. Chapters 1, 2, and 3 and Appendix A.
- S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press (2004), https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf. Section 3.3.
- Wikipedia pages Exponential family, Maximum-entropy probability distribution, Principle of maximum entropy, Lagrange multiplier, Convex conjugate.