Conditioning a multivariate normal vector on partial, noisy gaussian evidence

'cause I'm tired of re-deriving this stuff!

Lemma 1

Suppose we partition the mean and covariance of a gaussian vector \(y\) as $$ y = \left[ \begin{array}{c} y_1 \\ y_2 \end{array} \right] \sim N \left( \left[ \begin{array}{c} u_1 \\ u_2 \end{array} \right] , \left[ \begin{array}{cc} C_{11} & C_{12} \\ C_{21} & C_{22} \end{array} \right] \right) $$ then the distribution of the latter part of the vector \(y_2\) conditioned on the former taking a known value \(y_1=a_1\) is multivariate gaussian with mean and variance indicated below: $$ y_2 \mid y_1=a_1 \sim N \left( u_2 + C_{21} C_{11}^{-1}(a_1-u_1), C_{22}-C_{21}C_{11}^{-1}C_{12} \right) $$ We note this here for convenience. See Wikipedia notes on conditional multivariate gaussian, for example.

Lemma 2a

Suppose \(x\) is partitioned as $$ x = \left[ \begin{array}{c} x_1 \\ x_2 \end{array} \right] \sim N \left( \left[ \begin{array}{c} \mu_1 \\ \mu_2 \end{array} \right] , \left[ \begin{array}{cc} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22} \end{array} \right] \right) $$ If we hit \(x\) with a linear transformation and also add another gaussian random variable, viz: $$ y = \left[ \begin{array}{cc} I & 0 \\ I & 0 \\ 0 & I \end{array} \right] x - \left[ \begin{array}{c} \eta \\ 0 \\ 0 \end{array} \right] $$ where \(\eta\) is the multivariate normal random vector $$ \eta \sim N \left(\tilde{\mu}_1 - \tilde{a}_1, \tilde{\Sigma}_{11} \right) $$ with offset mean parameter \( \tilde{\mu}_1\) and variance \(\tilde{\Sigma}_{11}\) (and the use of the offset \(\tilde{a}_1\) will become apparent), then by properties of affine transformation (see Wikipedia again) and addition of multivariate random variables we have $$ y \sim N\left( \left[ \begin{array}{c} \mu_1 - \tilde{\mu}_1 + \tilde{a}_1 \\ \mu_1 \\ \mu_2 \end{array} \right], \left[ \begin{array}{ccc} \Sigma_{11} + \tilde{\Sigma}_{11} & \Sigma_{11} & \Sigma_{12} \\ \Sigma_{11} & \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{12} & \Sigma_{22} \end{array} \right] \right) $$

Lemma 2b

With \(x,y\) as in Lemma 2a, let us reuse notation for the second and third blocks of \(y\), i.e. \(x_1\) and \(x_2\), since the transformation evidently leaves them untouched. We write \(\tilde{x}_1\) for the part that is changed: $$ y = \left[ \begin{array}{c} \tilde{x}_1 \\ x_1 \\ x_2 \end{array} \right] $$ But now if we condition on \(\tilde{x}_1=\tilde{a}_1\) then the conditional distribution of the second and third parts of \(y\) will of course change. It is gaussian by application of Lemma 1 and has mean, covariance given by $$ \left[ \begin{array}{c} x_1^{+} \\ x_2^{+} \end{array} \right] \sim N\left( \left[ \begin{array}{c} \mu_1 \\ \mu_2 \end{array} \right] + \left[ \begin{array}{c} \Sigma_{11} \\ \Sigma_{21} \end{array} \right] \left( \Sigma_{11} + \tilde{\Sigma}_{11}^{-1} \right) \left( \tilde{\mu}_1 - \mu_1 \right), \Sigma - \left[\begin{array}{c} \Sigma_{11} \\ \Sigma_{21} \end{array} \right] \left( \Sigma_{11} + \tilde{\Sigma}_{11} \right)^{-1} \left[ \Sigma_{11} \Sigma_{21} \right] \right) $$

Interpretation of Lemmas 2a and 2b

The formulas have the following interpretation. Suppose we assume \(x\) has some prior distribution $$ x = \left[ \begin{array}{c} x_1 \\ x_2 \end{array} \right] \sim N \left( \left[ \begin{array}{c} \mu_1 \\ \mu_2 \end{array} \right] , \left[ \begin{array}{cc} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22} \end{array} \right] \right) $$ and we now condition on \(x_1 -\nu = 0\) where \( \nu \sim N \left(\tilde{\mu}_1, \tilde{\Sigma}_{11} \right) \). Then the posterior distribution of \(x = [x_1\ x_2]'\) is precisely the multivariate gaussian vector \(x^{+} = [x^{+}_1 \ x^{+}_2]'\) given above. It has the interpretation of a posterior distribution when one part of the vector is conditioned not on a precise vector of values, but a noisy one as with Kalman filtering. In the calculation above we conditioned on \(x_1 - \eta = \tilde{a}_1\) where \( \eta \sim N \left(\tilde{\mu}_1 - \tilde{a}_1, \tilde{\Sigma}_{11} \right) \), but that is evidently the same thing as conditioning on \(x_1 = \nu \) where \( \nu \sim N \left(\tilde{\mu}_1 , \tilde{\Sigma}_{11} \right) \).
We remark that if \(\tilde{\Sigma}_{11}\) is relatively small with respect to \(\Sigma_{11}\) then this more or less forces one part of \(x\) to adopt a new mean and variance \(\tilde{\mu}_1,\tilde{\Sigma}_{11}\). For example in the scalar case with \( \Sigma_{11} = \sigma_{11} \) and \( \tilde{\Sigma}_{11} = \tilde{\sigma}_{11} \) the posterior mean of \(x_1\) is \(\tilde{\mu}_1- \frac{1}{1+\tilde{\sigma_{11}}/\sigma_{11}}(\tilde{\mu_1}-\mu_1) \) and this tends to the new, prescribed mean \(\tilde{\mu}_1\) as \( \tilde{\sigma}_{11}/\sigma_{11}\) tends to zero.