Multivariate distributions are the natural extension of univariate distributions, but are inevitably significantly more complex — see Kotz and Johnson (1972 [JOH1]), and Kotz, Balakrishnan and Johnson (2000 [KOT1]) for a complete treatment of such distributions. In order to illustrate the concept of multivariate distributions we start with a simple extension to the Normal distribution, as this is probably the most important of the many possible distributions of this type.
In the discussion of the Normal distribution we have already mentioned its extension to the bivariate case as the Bivariate Normal. The simplest example of this is the case in which there are two random variables, x and y say, that are independently distributed with the same mean (typically 0) and standard deviations σx and σy. The joint probability of x and y, assuming they are independent, is simply the product of their separate Normal pdfs, so we have:
This distribution is a particular instance of the Bivariate Normal and can be used to obtain so-called circular error probabilities. Strictly speaking this model is for elliptical regions, since the two standard deviations may not be equal. With equal standard deviations a true circular model is obtained:
with cumulative distribution (obtained by integration in polar coordinates):
Now r can be seen to be the radius of a circle, so the distribution effectively is giving the probability that a point will be observed within a radius r of the center. This kind of problem arises in military ballistics, where the target is the center of the distribution and shots fired are affected by random factors that cause some vertical and horizontal deviation. For example, the probability that a shot will land within 2 meters of the middle of the target when the standard deviation is 1 meter is simply 1-e-2 = 86%. To illustrate this example we have generated 100 random (x,y) pairs using a unit Normal distribution and plotted these on a polar coordinate graph, as shown below (Plot A, left hand diagram). Notice that the choice of a random Normal distribution results in more points near the center of the plotted region than elsewhere — a random Uniform distribution would be more evenly distributed, but would not represented typical firing patterns although it may be suitable for other data types, such as airborne particles or gamma radiation landing at random on a sampling disc.
The circles in plot A are shown at 1,2 and 3 units from the center, which corresponds to 1,2 and 3 standard deviations in this case. There are 0 points outside the outer circle and 12 in the ring from 2 to 3 units, i.e. 88% in this sample lie within the inner ring. Plot B, on the right hand side, shows the same mechanism but this time with 1000 random points and variation in the horizontal direction being twice that of the vertical. The result is an extension of the range to 6 units for the outermost ring and a broadly elliptical pattern of points. Of the 1000 random points, only 4 lie outside the 6 unit ring, giving 99.6% lying within this range (the exact probability is 99.96%). Computation of probability values for the Bivariate Normal and, by extension, the Multivariate Normal and other multivariate distributions is typically by a callable program function (e.g. bvnor() in SPSS, dmnorm() in package mnormt for R users, mvnpdf() in MATLab etc.) rather than by applying special tables as was the case in the past (e.g. see Beyer, 1966 p146-148, [BEY1]).
If the two variables, x and y, are not independent and have non-equal standard deviations, the form of the distribution becomes more complicated. And as the number of dimensions is increased, a different form of notation, using matrices, is needed to enable the functions involved to be written in reasonably compact form.
Before addressing this issue some general comments about multivariate distributions are required. The first is that such distributions may be discrete or continuous, but typically it is the continuous versions that are most widely used, and in particular the Multivariate Normal distribution. Other multivariate distributions that are encountered include the multivariate versions of the Exponential distribution and the Chi-square distribution, which is known as the Wishart distribution. Multivariate versions of many of the other distributions described in our earlier sections on univariate distributions have also been the subject of much research — readers are referred to Kotz et al. (2000, [KOT1]) for more details.
Earlier we noted that for the continuous case, the basic form of a multivariate distribution function (or joint density function) was of the form:
with probability integral:
The cumulative distribution function, probability density function and moments follow directly from this definition. The marginal distribution of a subset of the variables is obtained by integrating out all those variables not included in the subset but which do appear in the joint distribution function. The conditional distribution of a subset is the joint distribution of the subset under the condition that the remaining variables are given (i.e. take specific values).
The Multivariate Normal distribution (MVN) is a straightforward extension of the univariate and bivariate Normal distribution cases previously described. In the bivariate case, with correlated rather than independent variables x and y, correlation coefficient ρ, and zero mean values, we have:
The form of this function is a bell-shape for which any horizontal plane intersection will produce an elliptical contour and any vertical plane will form a curve with the shape of a Normal distribution. An example is illustrated below, with ρ=0.5, σx=1, and σy=2 . The result in this case is an elliptical bell shape with major axis at 45 degrees to the axis, i.e. the correlation between the x and y variables corresponds to a rotation of the distribution through 45 degrees.
3D view of Bivariate Normal distribution
In order to obtain a compact form for the more general case, with n variables rather than 2, we need to introduce the variance-covariance matrix, Σ. This is an n x n matrix whose entries are the covariances of every pair of variables from the set of n we are considering. Thus Σ is symmetric with diagonal entries that are the variances of each variable. Now we assume that Σ is a positive definite matrix (and therefore has a determinant |Σ| and an inverse, Σ-1) with entries σij that are constants not variables, and we define μ as an n x 1 vector of mean values, μi, which again are constants. Then the MVN can be written as:
The notation used for the MVN is generally of the form ~N(μ,Σ). If n=1 the matrix Σ reduces to a 1x1 array, this being the variance σ2 and the expression for t reduces to (x-μ)2/σ2 , so we have:
which is the univariate form of the Normal distribution. The bivariate case produces the formula cited earlier in this section, but optionally including non-zero means. In the bivariate and multivariate cases the correlation between pairs of variables becomes relevant, as it appears in the matrix Σ — for two variables the matrix is of the form:
The marginal and conditional distributions of an MVN are also Normally distributed. Because the MVN underlies almost of all of classical multivariate statistical analysis, software packages include the MVN embedded in the formulas and reporting for these methods. However, separate computation of the MVN is not universally available, although it is relatively straightforward to generate programmatically if required. What is important to note is that the MVN involves a whole series of very specific assumptions: that the marginal distributions are all Normal; that the mean values are a set of constants; that the variances and covariances are also a set of constants. This in turn implies that one undertaking multivariate analysis that relies on the MVN for parameter estimation and/or for inferential analysis each and every variable must satisfy these assumptions. The MVN is to the Multinomial as the univariate Normal is to the Binomial, i.e. the MVN can be regarded as the limiting case for these distributions as n →∞.
Wikipedia: Multivariate Normal: https://en.wikipedia.org/wiki/Multivariate_normal