Notes on the Bias Variance trade-off and Ridge regression

Old Millfun, Shanghai, March 2026

Introduction

It felt useful to revisit some of the basics of the Bias Variance trade-off and ridge regression covering:

1. The relationship between a model’s forecast errors and its bias and variance
2. The trade-off between bias and variance
3. Ridge regression
4. Example of the bias and variance trade-off

The theory parts of these notes are based on Stanley H. Chan’s, ‘Introduction to Probability for Data Science’ and Wessel N. van Wieringen’s, ‘Lecture notes on ridge regression’

1. The relationship between a model’s forecast errors and its bias and variance

We assume that a value y is generated from an input variable x by a ‘true’ model f(x) with a random error term ε. f(x) is a general function of x.

ridge0

We do not know f(x) but can estimate a model of it f_hat. To estimate how well this predictor will do we calculate its expected mean squared error: ridge1a The error variance term is constant so we focus on the first term. We want to rewrite it in terms of the bias and variance of the estimator. To do this we use the trick of adding and subtracting the expectation of the estimated model to the equation. This allows us to express the expected mean squared errors in terms of the model’s bias squared, its variance and the variance of the error term along with a remainder term that we show has an expectation of zero. ridge1b Applying the expectations operator to the function f(x) returns the function f(x) as it is non-stochastic.
ridge1c The expectation of an expectation returns the original expectation as that is a constant. ridge1d The remaining terms therefore cancel to zero and we obtain the expected forecast error decomposed into three terms: ridge1e

Although the function we use is quite general in a regression context, this decomposition is not true for classification problems, where the error function that is minimised typically a form of cross entropy that cannot be expressed in terms of separate model bias and variance terms.

2. The trade-off between bias and variance

The previous section has rewritten the model’s expected forecast errors in terms of bias and variance. It has not shown any trade-off between bias and variance. It suggests two ways to increase a model’s forecast performance by reducing its:

Bias
Variance

However, the bias and variance can be related to each other which makes it harder to reduce both simultaneously.

In general one would expect a simple model to have a lower variance across training samples as it has fewer parameters and so there is less scope for it to be affected by new training data. For example a function that was a constant independent of x would have no variance and only the bias term would remain. If the underlying reality is though that the y values are a complicated function of x then a simple model will not be able to fit the true model and be more likely to be biased.

If we had a more complicated model then one might expect there to be less bias. However, even if the model was unbiased there will still be variance caused by the model estimates being sensitive to the data that it is estimated on. A model with more parameters is therefore likely to have a higher variance.

If we have an unbiased model then one might expect the bias term should go to 0 as the sample size increases. However there would still be error due to the model variance and the random noise in the data.

3. Ridge regression

A way to address the bias variance trade-off is the ridge regression. If we have a linear model written in matrix form where y is a (n x 1) dependent variable vector, X an (n x k) matrix of explanatory variables, β a (k x 1) vector of coefficients and ε a (n x 1) vector of random error terms. ridge3a We can extend the standard optimisation of choosing β to minimise the total of the sum of squared errors plus the sum of the squared coefficient values multiplied by a parameter λ. The latter term penalises choosing larger values of model coefficients reducing the variance of the estimator and thus helping improve forecast performance. ridge3b This is a generalisation of the Ordinary Least Squares (OLS) solution which is obtained when lambda is equal to zero. As the OLS estimator is an unbiased estimator, we know that the ridge regression which deviates from this is a biased estimator of β. As larger coefficient values of β are penalised the regulator term pushes the coefficient estimates towards 0.

Calculating the variance of the estimator ridge3d It is possible to show (see the Appendix) that ridge regression estimators, while biased, systematically have a lower level of variance than a normal OLS one. Although it is feasible, for a given data set, to estimate what an optimal value of lambda might using cross-validation over hold-out samples, there is no general theory of what the optimal lambda should be and no necessary guarantee that the trade-off of reducing the variance at the expense of the bias will improve forecast performance.

The connection to Bayesian estimation and Lasso

Another perspective on ridge regression is that we can obtain it by imposing certain restrictions on the priors of a Bayesian model. If we estimated a Bayesian model where the likelihood and prior are normally distributed and then solve out for the values of the beta coefficients that maximises the value of the resulting posterior then we would get the following estimator of the betas: ridge3e Ridge regression is therefore equivalent to a Bayesian regression with a prior of 0 slope coefficients and homoscedasticity across variables.

In the case of a lasso regulator where we have the sum of the betas as the regulator term: ridge3c then specific individual coefficient estimates among the variables are set to zero (unlike ridge regression this does not have an analytical solution).

4. Example of the bias and variance trade-off

To make the bias and variance trade-off more concrete we write a Python class BiasVariance that generates a simple example.

The class has the following methods:

_create_test_train_data Creates test and train datasets. As our ground truth we specify a seventh order polynomial wiggly_polynomial to which random noise is added in both the test and train datasets. We generate the corresponding y values at 51 evenly spaced x values between 0 and 100.
_create_estimate_regression Estimates a regression on this data where we can specify the order of the polynomial and the value of the regularisation parameter (giving a normal regression if lambda = 0)
_calculate_errors Calculates the:
• errors squared metric
• the bias squared metric
• the variance metric

In the class we approximate the expectations with averages over the test and training samples, with the expected:
• sum of squared errors for each x value we calculate the sum of squared errors for each test sample relative to the predictions from the training sample. We then average that over the test samples at a given x value and then average the resulting values over the x values.
• bias squared for each x value we calculate the average of the predictions over the training samples. We square the difference between the true model value and the average at each of the x points. This value is the same across all the training samples so we calculate it once rather than average it over the training samples which would give exactly the same number. We then average the resulting squared deviation of the average model across the x values.
• variance squared for each x value we calculate the difference between a forecast from the training sample and the average forecast over the training samples and the square that. We then average that across the training samples at each x value and then average the resulting values over the x values.

import numpy as np
from numpy.polynomial import Polynomial
import pandas as pd
import numpy as np
from sklearn.linear_model import Ridge
import matplotlib
from pprint import pprint

matplotlib.use("Agg")
import matplotlib.pyplot as plt
from matplotlib.ticker import FuncFormatter
import matplotlib.ticker as ticker


def wiggly_polynomial(x):
    z = (x - 50) / 50

    return 30 * z ** 7 - 5 * z ** 5 + 5 * z ** 3 - 4 * z


class BiasVariance:
    """A class that demonstrates the bias variance trade-off
    Args:
        truth_func: The underlying true function that generates the data.
        train_polynomial_degree (int): The degree of the polynomial used to fit the training data.
        sample_size (int): The number of training and test samples to generate
    Methods:
        _create_test_train_data(self): Creates the test and training datasets based on the true function.
        _create_estimate_ridge_regression(self, lambda_param): Estimates the model using ridge regression with the specified lambda parameter.
        _calculate_errors(self, model_forecast): Calculates the bias, variance, and sum of squared errors from the forecast. 
    """

    def __init__(self, truth_func, train_polynomial_degree, sample_size):
        """Class that illustrates the BiasVariance trade-off"""
        # The function that we pass as the ground truth
        self.truth_func = truth_func
        # The x values we are evaluating the function over
        self.x_inputs = np.linspace(0, 100, 51)
        self.truth_func_values = self.truth_func(self.x_inputs)
        # The order of the polynomial we are trying to fit
        self.poly_train_deg = train_polynomial_degree
        # The value of the lambda parameter
        self.lambda_param = None
        # The number of test and training samples we generate. For simplicity we assume these are the same
        self.sample = sample_size
        self.train_data = None
        # collection of error metrics
        self.error_metrics = {}

    def _create_test_train_data(self):
        """Create test and train datasets based off an underlying true model"""
        # Creates two sets of noise for the test and train data with mean 0 and standard deviation 2
        samples = np.random.normal(size=(self.sample, 51), scale=10)
        samples2 = np.random.normal(size=(self.sample, 51), scale=10)
        # Adds the random noise to the true model to create the test and train datasets
        self.test_data = self.truth_func_values + samples
        self.train_data = self.truth_func_values + samples2
        return self

    def _create_estimate_ridge_regression(self, lambda_param):
        """Create an estimated model predicting yfrom the training data"""
        # initalises the ridge regression
        self.lambda_param = lambda_param
        ridge = Ridge(lambda_param, fit_intercept=True)
        # Create a matrix with multiple variables 1, x, x^2, x^3 etc in ascending order for the polynomial we specify the degree of
        # We drop the first column as this is this will be fitted separately
        V = np.vander(self.x_inputs, increasing=True, N=self.poly_train_deg + 1)[:, 1:]
        # Fit the ridge regression on the training data
        ridge.fit(V, self.train_data.T)
        # Extract the estimated coefficients and intercept and make a prediction
        coeffs = ridge.coef_
        intercept = ridge.intercept_
        coeff_pred = V @ coeffs.T
        self.estimated_regression_ridge = intercept + coeff_pred
        return self.estimated_regression_ridge

    def _calculate_errors(self, model_forecast):
        """Calculate the bias, variance, and sum of squared errors from the input forecast"""
        # Average model forecasts across the training samples (axis=1) giving an average forecast for each of the x values (axis1)
        self.mean = model_forecast.mean(axis=1)

        # Average the squared errors across the training datasets at each x value, average over the different x values
        self.sse = ((model_forecast - self.test_data.T) ** 2).mean(axis=1).mean(axis=0)

        # Calculate the variance of the estimator from across the training datasets at each x value, average over the different x values
        # , None is used to broadcast the mean across the different training samples for each x value
        self.estimator_var = (
            ((model_forecast - self.mean[:, None]) ** 2).mean(axis=1).mean(axis=0)
        )

        # Calculate the bias squared of the estimator as the:
        # ground truth minus the average of the estimator from the training datasets, average over the different x values
        self.estimator_bias_sq = ((self.truth_func_values - self.mean) ** 2).mean(
            axis=0
        )

        # Return the metrics as a dictionary
        self.error_metrics = {
            "sum of errors squared": self.sse.round(1),
            "bias squared": self.estimator_bias_sq.round(1),
            "estimator variance": self.estimator_var.round(1),
        }
        return self

    def _chart_it(self):
        """chart the data"""
        ax = plt.gca()
        x = self.x_inputs
        # plot the different regressions estimated from the training data
        plt.plot(x, self.estimated_regression_ridge, alpha=0.2)
        # plot the average model output from the training data
        plt.plot(x, self.mean, alpha=1, color="black")
        # plot the ground truth as a blue curve
        plt.plot(x, self.truth_func_values, color="blue")
        ax.yaxis.set_major_formatter(ticker.StrMethodFormatter("{x:,.2f}"))
        plt.xlabel("x")
        plt.ylabel("y")
        plt.title(
            f"Estimated polynomial of order {self.poly_train_deg}, lambda {self.lambda_param}, sample size {self.sample}"
        )
        filepath = f"plot_{self.poly_train_deg}_{self.lambda_param}_{self.sample}.png"
        plt.savefig(filepath)
        plt.close()
        return self

To apply the class

# Apply the class
BV_example = BiasVariance(
    truth_func=wiggly_polynomial, train_polynomial_degree=10, sample_size=50
)
BV_example._create_test_train_data()
BV_example._create_estimate_ridge_regression(lambda_param=1)
BV_example._calculate_errors(BV_example.estimated_regression_ridge)
BV_example._chart_it()

print(BV_example.error_metrics)

The chart below shows a series of curves estimated using OLS on 50 random training samples. The blue curve shows the underlying true model that with the random noise is generating the data. The black curve shows the average of the model forecasts across the training data. The spread of the lighter individual training run curves around this gives a sense of the model variance. ridge4a The variance of the errors added to any given point is 100 and so when added to the bias squared and the estimator variance we approximate the sum of errors squared as suggested by the bias variance decomposition.

{'sum of errors squared': 120.4, 'bias squared': 0.3, 'variance': 21.8}

If we then apply the ridge regression penalising the variation setting lambda to one then the curves estimated from the different training samples vary less. ridge4b .

{'sum of errors squared': 119.2, 'bias squared': 0.1, 'variance': 20.6}

References

Stanley H. Chan, ‘Introduction to Probability for Data Science’
Efron and Hastie, ‘Computer Age statistical inference’
Kevin P. Murphy, ‘Machine Learning a Probabilistic Perspective’
Wessel N. van Wieringen, ‘Lecture notes on ridge regression’

Appendix 1: comparing the variance of the OLS and ridge regressions

We want to show that: ridgea1 ridgea2 Expanding the term in the central bracket gives: ridgea3 Bringing in the outer brackets: ridgea4 And we reduce the expression to: ridgea5 With lambda greater than 0 then the diagonal elements of all the matrices are positive in the middle and we have that the matrix is positive semi-definite. The ridge regression therefore has a lower variance than the standard OLS estimator.