Ridge regression is just linear regression with L2 regularization. Let me explain.

A few days ago I implemented an example of lasso regression from scratch. Lasso regression (“least absolute shrinkage and selection operator”) is just linear regression with L1 regularization applied to the coefficients during training.

Note: A revised version of the code presented in this blog post, where the code is wrapped up in a class definition, is at https://jamesmccaffreyblog.com/2026/05/05/ridge-regression-from-scratch-using-python-with-sgd-training/

A close cousin to lasso regression is ridge regression. Ridge regression is just linear regression with L2 regularization applied during training. So, I couldn’t resist implementing ridge regression from scratch to see how it compares to lasso regression from scratch.

An ordinary linear regression problem is one where the goal is to predict a single numeric value from one or more numeric predictor values. For example, you might want to predict the murder rate in a city based on x0 = city population, x1 = percent poor families in city, and x2 percent unemployed in city. A regression equation might look like:

y = (0.32 * x0) + (0.55 * x1) + (0.38 * x2) – 0.21

Here y is the predicted murder rate, the (0.32, 0.55, 0.38) are the coefficients/weights, and the -0.21 is the constant/bias.

Ridge regression is just a slight modification of standard linear regression. Ridge applies L2 regularization to the regression coefficients. This means the coefficients are nudged towards zero during training to avoid huge coefficient values. Large coefficients often lead to a prediction model that is overfitted — the model predicts well on the data used to create the model but predicts poorly on new, previously unseen data (poor generalization).

I found a nice, simple dataset at people.sc.fsu.edu/~jburkardt/datasets/regression/x08.txt. There are four columns: city population, city poverty, city unemployment, murder rate. I normalized the data so that all values are between 0.0 and 1.0 by dividing population by 10,000,000, poverty by 100, unemployment by 10, and murder rate by 100. The goal is to predict murder rate.

L2 regularization is somewhat of a meta-algorithm, meaning that there are several ways to implement it. In theory, you calculate a gradient for each coefficient and then modify each weight gradient based on mean squared error. Then you use the modified gradients to update each associated coefficient. The math is quite tricky.

It turns out that this is mathematically equivalent (subject to multiplication by a constant) to computing each gradient as usual, updating each associated coefficient as usual, and then moving the new weight towards zero. This second approach is easier to implement. (Note: there are several minor variations here too; for example, you can move weights towards zero first, then update).

The key code for ridge regression training looks like:

. . .
for j in range(3):          # each coeff
  a[j] += lr * x[j] * (t - y)
  a[j] *= (1.0 - alpha)     # apply L2
  b += lr * (t - y)         # the constant
  b *= (1.0 - alpha)        # apply L2
. . .

Weight decay is another closely related technique. See https://jamesmccaffreyblog.com/2019/05/09/the-difference-between-neural-network-l2-regularization-and-weight-decay/.

I used stochastic gradient descent for my ridge regression demo because I’ve used L2 regularization for neural networks and understand the mechanism.

Ridge regression is a classical statistics technique. Many of these old techniques are being forgotten in a sense because systems based on neural networks are so much more powerful. But classical statistics techniques can still be useful, especially in situations with small datasets.

Sadly, the violent crime rate in a city is highly correlated with many bad things: low education, high reliance on public assistance, single parent households, drug use, and so on. But correlation isn’t causation and I’m optimistic that technology, including machine learning, can solve a lot of problems and improve people’s lives.

Demo code. Replace “lt” with Boolean operator symbol.

# ridge_regression.py
# data: https://people.sc.fsu.edu/~jburkardt/+
#   datasets/regression/x08.txt

import numpy as np

def compute(x, a, b):
  # x inputs, a coefficients, b constant
  result = 0.0
  for i in range(3):
    result += x[i] * a[i]
  result += b
  return result

def error(data, a, b):
  n = len(data)
  sum = 0.0
  for i in range(n):
    y = compute(data[i], a, b)  # predicted
    t = data[i][3]              # target at [3]
    sum += (y - t) * (y - t)
  return sum / n  # mean squared error

def accuracy(data, a, b, pct):
  n = len(data)
  n_correct = 0; n_wrong = 0
  for i in range(n):
    y = compute(data[i], a, b)  # predicted
    t = data[i][3]              # target at [3]
    if np.abs(y - t) "lt" np.abs(pct * t):
      n_correct += 1
    else:
      n_wrong += 1
  return n_correct / (n_correct + n_wrong)

def train(data, epochs, lr, alpha):
  rnd = np.random.RandomState(0)
  n = len(data)  # 20
  a = rnd.rand(3)  # predictors: pop., pov., unemp.
  b = 0.0
  indices = np.arange(n)  # [0, 1, 2, .. 19]

  for ep in range(epochs):
    rnd.shuffle(indices)
    for ii in range(n):
      i = indices[ii]
      x = data[i]                 # 3 inputs
      y = compute(data[i], a, b)  # predicted
      t = data[i][3]              # target
      for j in range(3):          # each coeff
        a[j] += lr * x[j] * (t - y)
        a[j] *= (1.0 - alpha)     # L2
      b += lr * (t - y)
      b *= (1.0 - alpha)          # L2

    err = error(data, a, b)
    print("epoch = %4d | error = %0.4f " % (ep, err))
  return (a, b)
  

def main():
  print("\nBegin ridge regression demo ")
  print("Predict city murder rate ")

  # population, low_income, unemployed, murder_rate
  data = np.array(
   [[0.0587, 0.165, 0.62, 0.112], 
    [0.0643, 0.205, 0.64, 0.134], 
    [0.0635, 0.263, 0.93, 0.407], 
    [0.0692, 0.165, 0.53, 0.053], 
    [0.1248, 0.192, 0.73, 0.248], 
    [0.0643, 0.165, 0.59, 0.127], 
    [0.1964, 0.202, 0.64, 0.209], 
    [0.1531, 0.213, 0.76, 0.357], 
    [0.0713, 0.172, 0.49, 0.087], 
    [0.0749, 0.143, 0.64, 0.096], 
    [0.7895, 0.181, 0.60, 0.145], 
    [0.0762, 0.231, 0.74, 0.269], 
    [0.2793, 0.191, 0.58, 0.157], 
    [0.0741, 0.247, 0.86, 0.362], 
    [0.0625, 0.186, 0.65, 0.181], 
    [0.0854, 0.249, 0.83, 0.289], 
    [0.0716, 0.179, 0.67, 0.149], 
    [0.0921, 0.224, 0.86, 0.258], 
    [0.0595, 0.202, 0.84, 0.217], 
    [0.3353, 0.169, 0.67, 0.257]])

  epochs = 10
  lr = 0.05
  alpha = 0.001  # ridge
  print("\nStarting training ")
  (a, b) = train(data, epochs, lr, alpha)
  print("Done ")

  print("\nCoefficients: ")
  print(a)
  print("Bias: ")
  print(b)

  acc = accuracy(data, a, b, 0.25)
  print("\nModel accuracy = %0.4f " % acc)

  print("\nEnd ridge regression demo ")

if __name__ == "__main__":
  main()