Crossrail Place Footbridge, October 2020
Data normalisation, transforming the variables in a regression to be mean zero with variance one (sometimes called standardisation), comes up frequently in work so it felt worth writing some notes on it.
Variables in regressions are often on different scales e.g. if we are predicting shops’ revenue using inputs of the local population count in 1000s and their online rating between 1 and 5. In this context the regression coefficients of the larger scale variables will generally be smaller as they have more variation. A direct comparison of the coefficients will suggest the variable on the smaller scale is more important, when in practice it may vary relatively little e.g online ratings can cluster at certain levels like four. Scales can also be abitrary and affect coefficients of impact without affecting the fundamental reality such as when we express a variable in units of 100s rather than 000s.
We therefore often standardise the data by subtracting each variable’s mean and dividing by its variance to get a sense of what the effect of a typical change in the variable might be. There are though other, but often related, motivations for data normalisation such as in a:
• ridge regression as you want to avoid it penalising the variables with the smallest scales and so the largest coefficients
• principal component analysis to prevent it prioritising dimensions on larger scales which account for more of the variance
• panel data regression as you want to remove individual unit specific effects without using dummy variables
• Bayesian regression as you want the simulation of the posterior distribution to converge better
• neural network’s inputs and in intermediate layers as this has been found to improve performance
In these notes we work through the effects of normalisation’s two standard components where data is scaled by:
- dividing by the variance
- subtracting the mean
This is done in the context of a linear model of the form:
where we have k variables on n observations and y is a (n x 1) vector, X an (n x k) matrix, beta a (k x 1) vector and eta a (n x 1) vector of random error terms. I assume it is typically not possible to make general statements on the effects of normalisation outside this basic linear case, or at least not easily.
1. Scaling the data by dividing by the variance
Scaling the X variables
If we have a transform that is scaling the X variables by a variable specific factor e.g. dividing each variable by its own variance with n observations on k variables then this corresponds to multiplying the X matrix by a (k x k) diagonal matrix D where each diagonal element is the inverse of the variable’s variance. If a regression has a constant then the associated X column will have zero variance and so the normalisation by variance is not possible. As we usually want to estimate a constant we leave it unchanged in the transform with a corresponding diagonal 1 term. For example with 3 variables and a constant the transformation would be:
Applying this transformation matrix D we obtain the Ordinary Least Squares (OLS) estimator on the transformed variables XD as:
Using the relationship of an inverse of a product of square matrices:
and the fact that D is symmmetric we get:
We see that scaling the X variables changes the beta estimates by the inverse of the scaling factor. In the case of normalising by the variance, this multiplies the estimates of beta by the variance. The constant is unchanged.
Geometrically as the transformation is a diagonal matrix it scales the basis vectors that the X data represents but does not change their orientation and the space spanned by them. As the original y vector remains unchanged the coefficients that define the corresponding projection of the y data we are trying to represent in this basis scales up accordingly if the transform shrinks the size of the basis vectors as normalising by the variance does.
Scaling the y variables
If we scale the y vector by a factor alpha e.g. dividing by the variance of y then this scalar commutes with everything in the equation and we obtain:
Geometrically if the transformation is dividing by the variance then the y vector shrinks and so correspondingly does the magnitude of the coefficients that are needed to describe it.
In general the scaling of the y variable is not done for ridge regression, but does happen in some Bayesian regression applications to assist convergence. If we have transformed the y data then, if we want to predict or evaluate the model’s y values we will have to reverse the y transformation to get the y values back in their original scale, something that is not necessary if only the X data is transformed.
The effect on the variance and test statistics
Calculating the effect on the variance of the betas:
The variance therefore changes by the square of the scaling factors. However the hypothesis tests are unchanged as in calculating the test statistic we normalise the estimated coefficients by the square root of their variances to allow us to use standard distributions. This removes the effect of the transformation on the test statistic. For example, specialising to one of the variable’s test statistics shows that the effects of the scaling factors cancel out.
2. Scaling the data by subtracting the mean
If we transform a matrix by subtracting the corresponding mean from each column then this is equivalent to the OLS regression of each variable on a vector of ones. The transformation on the X variables to subtract the mean is therefore:
If we transform the X variables we obtain the following OLS estimate:
If we transform both the X and the y variables we obtain the same beta estimates as if we ran the regression with just the X variables de-meaned:
This hides an important implication relating the values of the betas to the constant term in the regression. If we subtract the mean of the constant vector from itself then we get a vector of zeros. If we include this vector of zeros in the X matrix then it is not possible to calculate the inverse in the OLS estimation. This is because with a column of zeros there are an infinite number of values the constant can take so its value is indeterminate, matrix inversion fails and we cannot estimate the betas.
Therefore we either keep the vector of ones representing the constant in the regression as is or we drop that vector from X and estimate the constant separately.
We unpack this by looking at the effect of subtracting the means in a more general way. The Frisch - Waugh - Lovell theorem says that in a linear regression if you:
- regress the dependent variable y on a subset of the independent variables
- regress the remaining independent variables on the first set
- regress the residuals of the first regression on the residuals of the second regression
then the effects of the second set of independent variables is obtained.
The projection matrix is:
Effects on the betas, variance and test statistics (and the constant)
Having looked at the transformation in a more general way, the implications of subtracting the mean become clear. When X_1 is just the vector of 1s then the regression of y and X_2 on it corresponds to the transformation of subtracting the means of the y and X variables. As a result we obtain the same estimate of beta_2 as we do without subtracting the means. The betas and variance are therefore unchanged by de-meaning and test statistics are as before.
The equation that specifies OLS is solvable if we have the X data de-meaned, but there is no constant. The estimates of the betas (which will not include a constant term in this case) are also unaffected by de-meaning the y variable.
From the previous discussion we can see that in the transformed regression we do not need to include the constant to obtain unbiased estimates of the variables’ coefficients. In a ridge regression the constant is typically not included in estimation as it cannot be normalised on a consistent basis as it has zero variance and so will always have a different scale. However, in both cases we still want to estimate a constant if we want to make any predictions.
If we have estimated beta_2 we can substitute this in to the untransformed data and calculate the residual.
If we then regress the residuals on the i vector (which is a fancy way of saying average the residuals) we get an unbiased estimate of the constant back.
This is also the estimate of beta_1 we would obtain from running an OLS regression on the de-meaned X variables with a constant keeping the y variables untransformed.
If we also de-mean the y variables then this removes the variation which is left behind after we adjust for the variation in the de-meaned X variables i.e. the effect of the constant is removed and our estimate of it will be the average of the random noise which will be zero in large samples. De-meaning the y variable in a regression with a constant and de-meaned X variables effectively gives us no estimate of the constant, but we can recover one using the approach just outlined.
3. Both subtracting the mean and dividing by the variance
Intuitively one should be able to subtract the mean and/or divide the variance and the effects are separate. If we apply the variance normalisation D and the mean subtracting transform M to the X matrix then in the OLS estimator we see that we can factor out the effect of the variance transformation making it independent of the mean subtraction transformation and vice versa.
We can therefore combine both effects. Subtracting the X variables’ means while dividing by their variances (omitting the constant from the transformations) therefore increases the estimates of the coefficients by their variances. The estimate of the constant is left unchanged by the transformation of the X variables.
If the y variable is also transformed by subtracting its mean and dividing by its variance then the estimate of the constant will tend to 0 and the estimate of the slope coefficients will also be scaled by the variance of the y variable. The hypothesis tests are unchanged when both X and y variables are transformed.
References
Davidson and Mackinnon ‘Econometric Theory and Methods’