I try to learn something new every day. That usually means writing some computer code. Over the past few years, I have run into LARS (least angle regression) a few times, but never took the time to try and understand it. So, one evening after work, I figured I’d start an exploration using the scikit-learn LassoLars module.

Least Angle Regression (LARS) is a variation of linear regression. It uses stepwise regression — a technique that adds one predictor variable at a time, skipping over variables that don’t help. It is intended for use with high-dimensional datasets, where the number of features outnumbers the observations.

The “lasso” part of the scikit LassoLars, adds L1 regularization.

I see two immediate problems with LARS. First, LARS is just a linear model, which means it cannot handle complex data. Second, scenarios where the number of predictor variables is greater than the number of training items, are very rare.

After a few hours of experimentation, I started to understand LARS, but I still have more things to figure out. My initial thoughts are that LARS with lasso (aka L1 regularization) is very similar to standard linear regression with lasso except that the model coefficients and bias are computed in a different way. Of course, there’s a lot more than that but I don’t want to put down possibly incorrect information in this post.

The output of one run (of many) of my experiments is:

train_X:
[[ 2.10 -0.64  0.19 -0.19  1.69  0.05 -0.75 -0.89]
 [ 0.59 -0.35 -0.21  0.12 -0.35 -1.14  0.30  0.62]
 [-1.10 -0.38  1.13  0.32 -2.06 -0.32  1.46 -0.25]
 [-0.30  1.25  0.51  0.84  0.89 -0.75  0.29  0.93]
 [-0.50 -1.40 -1.44  0.49  1.52  2.19  1.13 -0.08]
 [ 0.76  0.83  0.23  0.16 -2.02 -0.31  0.32  0.88]
 [ 0.50  1.14  0.90 -0.17  0.58 -1.10  0.04 -0.88]]

train_y:
[  26.51   18.68 -137.94  200.07  242.04  -92.10  -24.09]

test_X:
[[ 0.74  0.23  1.66 -0.69 -0.01 -1.12 -0.67 -0.85]
 [-0.40  0.53 -0.69  0.90 -0.94 -0.27 -0.12 -0.68]
 [-0.76 -2.30  1.74  1.62 -1.07  0.87 -0.53 -0.61]]

test_y:
[-143.33 -161.28 -214.44]

===========================

Creating LARS model alpha = 0.0100
Done

LARS model coefficients:
[-39.92   7.95 -12.94   0.00  87.53  20.06   0.00  84.51]
LARS model bias = 43.7840

LARS model R2 on train data = 1.0000
LARS model acc (0.10) train = 1.0000

LARS model R2 on test data = -7.7915
LARS model acc (0.10) test = 0.0000

===========================

Creating Lasso model alpha = 0.0100
Done

Lasso model coefficients:
[-39.92   7.97 -12.99   0.00  87.52  20.05   0.00  84.49]
Lasso model bias = 43.7917

Lasso model R2 on train data = 1.0000
Lasso model accuracy (0.10) train = 1.0000

Lasso model R2 on test data = -7.7875
Lasso model accuracy (0.10) test = 0.0000

End demo

There is a lot going on here. The synthetic data has 8 predictors but only 2 contribute to the target y values. Notice both LARS with lasso and regular linear regression with lasso drive two of the model coefficients to 0.

OK. An interesting start but there’s a lot more exploring to do before I can say that I understand least angle regression.

Some linear regression results need no explanation.

Demo program. Replace the “lt” in the accuracy() function with the Boolean less-than operator (my blog editor chokes on symbols).

# lars_scikit_explore.py

import numpy as np
from sklearn.datasets import make_regression
from sklearn import linear_model

np.set_printoptions(precision=2, suppress=True,
  floatmode='fixed')

# -----------------------------------------------------------

def accuracy(model, data_X, data_y, pct_close):
  n = len(data_X)
  n_correct = 0; n_wrong = 0
  for i in range(n):
    x = data_X[i].reshape(1,-1)
    y = data_y[i]
    y_pred = model.predict(x)[0]

    if np.abs(y - y_pred) < np.abs(y * pct_close):
      n_correct += 1
    else: 
      n_wrong += 1
  # print("Correct = " + str(n_correct))
  # print("Wrong   = " + str(n_wrong))
  return n_correct / (n_correct + n_wrong)

# -----------------------------------------------------------

X, y = make_regression(n_samples=10,
  n_features=8, n_informative=6, noise=0.01,
  random_state=1)

train_X = X[0:7,:]
train_y = y[0:7]
test_X = X[7:10,:]
test_y = y[7:10]

print("\ntrain_X: ")
print(train_X)

print("\ntrain_y: ")
print(train_y)

print("\ntest_X: ")
print(test_X)

print("\ntest_y: ")
print(test_y)

# LassoLars(alpha=1.0, *, fit_intercept=True,
# verbose=False, precompute='auto', max_iter=500,
# eps=np.float64(2.220446049250313e-16), copy_X=True,
# fit_path=True, positive=False, jitter=None,
# random_state=None)

print("\n=========================== ")

alpha = 0.01
print("\nCreating LARS model alpha = %0.4f " % alpha)
lars_model = linear_model.LassoLars(alpha=alpha)
lars_model.fit(train_X, train_y)
print("Done")

print("\nLARS model coefficients: ")
print(lars_model.coef_)
print("LARS model bias = %0.4f " % lars_model.intercept_)

train_r2 = lars_model.score(train_X, train_y)
print("\nLARS model R2 on train data = %0.4f " % train_r2)

train_acc = accuracy(lars_model, train_X, train_y, 0.10)
print("LARS model acc (0.10) train = %0.4f " % train_acc)

test_r2 = lars_model.score(test_X, test_y)
print("\nLARS model R2 on test data = %0.4f " % test_r2)

test_acc = accuracy(lars_model, test_X, test_y, 0.10)
print("LARS model acc (0.10) test = %0.4f " % test_acc)

print("\n=========================== ")

print("\nCreating Lasso model alpha = %0.4f " % alpha)
lasso_model = linear_model.Lasso(alpha=alpha, max_iter=10000)
lasso_model.fit(train_X, train_y)
print("Done")

print("\nLasso model coefficients: ")
print(lasso_model.coef_)
print("Lasso model bias = %0.4f " % lasso_model.intercept_)

train_r2 = lasso_model.score(train_X, train_y)
print("\nLasso model R2 on train data = %0.4f " % train_r2)

train_acc = accuracy(lasso_model, train_X, train_y, 0.10)
print("Lasso model accuracy (0.10) train = %0.4f " % train_acc)

test_r2 = lasso_model.score(test_X, test_y)
print("\nLasso model R2 on test data = %0.4f " % test_r2)

test_acc = accuracy(lasso_model, test_X, test_y, 0.10)
print("Lasso model accuracy (0.10) test = %0.4f " % test_acc)

print("\nEnd demo ")