I write code almost every day. Like many skills, writing code is something that must be practiced, and anyway, I just enjoy writing code. One morning, I figured I’d run the well-known Diabetes Dataset through a from-scratch C# implementation of gradient boost regression. I realized that this was going to be a non-trivial exploration. So I decided to run the Diabetes Dataset through the scikit GradientBoostingRegressor module, so that I could get some baseline sanity-check results to see if my future scratch C# system is on the right track.

Based on previous experiments with linear regression, quadratic regression, neural network regression, kernel ridge regression, random forest regression, and AdaBoost regression, I was pretty sure the scikit GradientBoostingRegressor model would give poor prediction accuracy, and that’s what happened.

The raw Diabetes Dataset looks like:

59, 2, 32.1, 101.00, 157,  93.2, 38, 4.00, 4.8598, 87, 151
48, 1, 21.6,  87.00, 183, 103.2, 70, 3.00, 3.8918, 69,  75
72, 2, 30.5,  93.00, 156,  93.6, 41, 4.00, 4.6728, 85, 141
. . .

Each line represents a patient. The first 10 values on each line are predictors. The last value on each line is the target value (a diabetes metric) to predict. The predictors are: age, sex, body mass index, blood pressure, serum cholesterol, low-density lipoproteins, high-density lipoproteins, total cholesterol, triglycerides, blood sugar. There are 442 data items.

The sex encoding isn’t explained anywhere but I suspect male = 1, female = 2 because there are 235 1 values and 206 2 values).

Note that this Diabetes Dataset, which is included as an example dataset in the Python language scikit-learn library, is not the same as the Pima Diabetes Dataset from the UCI dataset repository.

For the sanity-check scikit version, I just use the built-in scikit dataset, with the model_selection train_test_split() function which will scale the predictors, and use 75% of the 442 items for training (331 items) and 25% for test (111 items). To keep the target y values in roughly the same range as the predictors, I divide the y values by 1000, but this doesn’t affect model accuracy.

The output of the demo is:

Begin scikit Gradient Boost demo

First three X predictors:
[[-0.0491 -0.0446 -0.0569 -0.0435 -0.0456 -0.0433  0.0008
  -0.0395 -0.0119  0.0155]

 [-0.0527 -0.0446 -0.0558 -0.0367  0.0892 -0.0032  0.0081
   0.0343  0.1324  0.0031]

 [-0.0927  0.0507 -0.0903 -0.0573 -0.0250 -0.0304 -0.0066
  -0.0026  0.0241  0.0031]]

First three y targets:
0.0680
0.1090
0.0940

Setting n_estimators = 100
Setting max_depth = 3
Setting learning rate = 0.1000

Training Gradient Boost model
Done

Accuracy train (within 0.10): 0.4079
Accuracy test (within 0.10): 0.1982

MSE train: 0.0008
MSE test: 0.0039

R2 train: 0.8785
R2 test: 0.2165

Predicting for x =
[-0.0491 -0.0446 -0.0569 -0.0435 -0.0456
 -0.0433  0.0008 -0.0395 -0.0119 0.0155]
Predicted y = 0.0961

End demo

These poor results were similar to the results I got using other regression models. I have done many experiments with the Diabetes Dataset and I’ve concluded the the default target value in the last column (a patient diabetes score) simply cannot be predicted well. But the variables in columns [4], [5], [6], [7], and [8] can be meaningfully predicted from the other columns.

Gradient boost regression was new and exciting when it was developed in 1999-2001. Twenty-five years later, it’s reaching “vintage machine learning technique” status.

I have always been a big fan of vintage space art from the 1950s. Here are a couple of examples by two of my favorite artists. Left: A lunar outpost by Alexander Leydenfrost (1888-1961). Right: An orbiting space station in operation by Alex Schomburg (1905-1998).

Demo program. Replace “lt” in the accuracy() function with the Boolean less-than operator. (My blog editor chokes on symbols).

# scikit_gradient_boost.py
# demo using Diabetes Dataset

import numpy as np

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor

# -----------------------------------------------------------

def accuracy(model, data_X, data_y, pct_close):
  # assumes model has a predict(X)
  n = len(data_X)
  n_correct = 0; n_wrong = 0
  for i in range(n):
    x = data_X[i].reshape(1,-1)  # make it a matrix
    y = data_y[i]
    y_pred = model.predict(x)[0]  # predict() expects 2D

    if np.abs(y - y_pred) "lt" np.abs(y * pct_close):
      n_correct += 1
    else: 
      n_wrong += 1
  # print("Correct = " + str(n_correct))
  # print("Wrong   = " + str(n_wrong))
  return n_correct / (n_correct + n_wrong)

# -----------------------------------------------------------

def MSE(model, data_X, data_y):
  # scratch. could use sklearn.metrics.mean_squared_error()
  n = len(data_X)
  sum = 0.0
  for i in range(n):
    x = data_X[i].reshape(1,-1)
    y = data_y[i]
    y_pred = model.predict(x)[0]
    sum += (y - y_pred) * (y - y_pred)

  return sum / n

# -----------------------------------------------------------

def main():
  print("\nBegin scikit Gradient Boost demo ")

  np.set_printoptions(precision=4, suppress=True,
    floatmode='fixed')
  np.random.seed(0)  # not used this version

  # use built-in scikit diabetes data
  X, y = load_diabetes(return_X_y=True, scaled=True)
  train_X, test_X, train_y, test_y = \
    train_test_split(X, y, random_state=0)  # 25% test

  # scale y to keep MSE values more interpretable
  train_y /= 1000
  test_y /= 1000

  print("\nFirst three X predictors: ")
  print(train_X[0:3,:])
  print("\nFirst three y targets: ")
  for i in range(3):
    print("%0.4f" % train_y[i])

  n_ests = 100  # these are default values
  max_d = 3
  lrn_rate = 0.10

  print("\nSetting n_estimators = " + \
    str(n_ests))
  print("Setting max_depth = " + \
    str(max_d))
  print("Setting learning rate = %0.4f" % lrn_rate)

  # GradientBoostingRegressor(*, loss='squared_error',
  # learning_rate=0.1, n_estimators=100, subsample=1.0,
  # criterion='friedman_mse', min_samples_split=2,
  # min_samples_leaf=1, min_weight_fraction_leaf=0.0,
  # max_depth=3, min_impurity_decrease=0.0, init=None,
  # random_state=None, max_features=None, alpha=0.9,
  # verbose=0, max_leaf_nodes=None, warm_start=False,
  # validation_fraction=0.1, n_iter_no_change=None,
  # tol=0.0001, ccp_alpha=0.0)
  # SHEESH!

  model = \
    GradientBoostingRegressor(
    n_estimators=n_ests,
    max_depth=max_d,
    learning_rate=lrn_rate,
    random_state=0)

  print("\nTraining Gradient Boost model ")
  model.fit(train_X, train_y)
  print("Done " )

  acc_train = accuracy(model, train_X, train_y, 0.10)
  print("\nAccuracy train (within 0.10): %0.4f " % acc_train)
  acc_test = accuracy(model, test_X, test_y, 0.10)
  print("Accuracy test (within 0.10): %0.4f " % acc_test)

  mse_train = MSE(model, train_X, train_y)
  print("\nMSE train: %0.4f " % mse_train)
  mse_test = MSE(model, test_X, test_y)
  print("MSE test: %0.4f " % mse_test)

  r2_train = model.score(train_X, train_y)
  print("\nR2 train: %0.4f " % r2_train)
  r2_test = model.score(test_X, test_y)
  print("R2 test: %0.4f " % r2_test)

  print("\nPredicting for x = ")
  print(train_X[0])
  pred_y = model.predict(train_X[0].reshape(1,-1))[0]
  print("Predicted y = %0.4f " % pred_y)

  print("\nEnd demo ")

if __name__ == "__main__":
  main()