Bottom line: I compared my from-scratch C# implementation of gradient boosting regression on training data with categorical predictor variables with the scikit-learn GradientBoostingRegression library module and got essentially the same results.
The goal of a machine learning regression problem is to predict a single numeric value. For example, you might want to predict the Income of a person based on their Sex, Age, City, Height, and Psychology (timid, alert, brave). There are many different regression techniques. One of the most common techniques is gradient boosting regression.
Gradient boosting regression (GBR) is a tree-based technique. GBR creates a collection of small decision trees (typically about 100) where each tree learns how to predict training items that were poorly predicted by the previous tree.
I wanted to validate my from-scratch C# implementation of GBR against the Python language scikit-learn library GradientBoostingRegressor module. I generated a set of synthetic training data that looks like:
M, 29, Austin, 68.5, timid, 54600.00 F, 51, Juneau, 60.0, brave, 25700.00 M, 32 ,Boston, 73.5, alert, 74900.00 . . .
The goal is to predict Income (in the last column) from Sex, Age, City, Height (in inches), and Psychology (timid, alert, brave). There are 200 training items and 40 test items.
The Sex variable is encoded as M = 0, F = 1. There are 10 possible City values: Austin, Boston, Casper, Denver, Eugene, Fresno, Gallup, Helena, Irvine, Juneau.
I used sorted-target-mean ordinal encoding, where the city that has the smallest average Income (Boston = $30,600.00) is encoded as 0, the next larger average Income city (Casper = $33,100.00) is encoded as 1, and so on to the largest average income city (Helena = $64,194.74) is 9.
I used one-hot encoding for Psychology: timid = 1,0,0 alert = 0,1,0 brave = 0,0,1.
When using tree-based regression, it’s not necessary to normalize numeric predictor variables, but it doesn’t hurt and can be helpful in some scenarios. I divided all Age values by 100, and all Height values by 100. I normalized the target Income variable by dividing all values by 100,000.
The resulting normalized and encoded data looks like:
0, 0.29, 2, 0.685, 1,0,0, 0.54600 1, 0.51, 8, 0.600, 0,0,1, 0.25700 0, 0.32, 0, 0.735, 0,1,0, 0.74900 . . .
The key parts of the output of my C# implementation of gradient boosting regression are:
Begin C# Gradient Boosting regression demo Predict Income from Sex, Age, City Height, Psychology Loading people train (200) and test (40) data Done First three raw train X: M,29,Austin,68.5,timid,54600.00 F,51,Juneau,60.0,brave,25700.00 M,32,Boston,73.5,alert,74900.00 First three train X: 0.00000 0.29000 2.00000 0.68500 1.00000 0.00000 0.00000 1.00000 0.51000 8.00000 0.60000 0.00000 0.00000 1.00000 0.00000 0.32000 0.00000 0.73500 0.00000 1.00000 0.00000 First three train y: 0.54600 0.25700 0.74900 Setting numTrees = 200 Setting maxDepth = 4 Setting minSamples = 2 Setting minLeaf = 1 Setting lrnRate = 0.0500 Creating and training GradientBoostRegression model Done Evaluating model Accuracy train (within 0.10) = 1.0000 Accuracy test (within 0.10) = 0.7750 MSE train = 0.0000 MSE test = 0.0022 R2 train = 0.9993 R2 test = 0.9344 Predicting for x = 0.0000 0.2900 2.0000 0.6850 1.0000 0.0000 0.0000 Predicted y = 0.5391 Explaining for x = 0.0000 0.2900 2.0000 0.6850 1.0000 0.0000 0.0000 Initial prediction: 0.4685 t = 0 pred_res = -0.0342 delta = -0.0017 pred = 0.4668 t = 1 pred_res = 0.0079 delta = 0.0004 pred = 0.4672 t = 2 pred_res = -0.0315 delta = -0.0016 pred = 0.4656 t = 3 pred_res = -0.0361 delta = -0.0018 pred = 0.4638 t = 4 pred_res = -0.0136 delta = -0.0007 pred = 0.4632 . . . t = 198 pred_res = -0.0000 delta = -0.0000 pred = 0.5391 t = 199 pred_res = 0.0000 delta = 0.0000 pred = 0.5391 Predicted y = 0.5391 End demo
I ran the data through the scikit GradientBoostingRegression module and got essentially the same results:
Begin scikit gradient boost regression demo Predict Income from Sex, Age, City, Height, Psych Setting n_estimators = 200 Setting max_depth = 4 Setting min_samples_split = 2 Setting min_samples_leaf = 1 Setting lrn_rate = 0.0500 Training gradient boost regression model Done Evaluating model Acc train (within 0.10): 1.0000 Acc test (within 0.10): 0.8000 MSE train: 0.0000 MSE test: 0.0022 R2 train: 0.9993 R2 test: 0.9347 Predicting for x = [0.0000 0.2900 2.0000 0.6850 1.0000 0.0000 0.0000] Predicted y = 0.5391 End demo
A big advantage of my from-scratch C# implementation of GBR is that it was easy for me to add statements that show exactly how a prediction is made.
Machine learning regression models are often beautiful to my eye. But I’m not a good judge of other kinds of models. I’ve noticed that fashion models often look like they walk with an exaggerated limp. Here are three images that were returned by an Internet search for “ugly fashion model on runway”.
Left: She doesn’t look ugly to me. Maybe she is too blonde or maybe the slightly vacuous look on her face isn’t attractive in some model-sense way.
Center: Again, this model isn’t ugly to my eye but who am I to disagree with the Internet? She does kind of have a resting-witch-face maybe.
Right: OK, there isn’t much to disagree about “ugly fashion model on runway” here. I don’t want to sound mean, but she doesn’t belong on a fashion runway. An airport runway maybe, but not a fashion runway.
Demo C# program. Very long and complex. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols (my lame blog editor consistently chokes on symbols).
using System;
using System.IO;
using System.Collections.Generic;
namespace GradientBoostRegression
{
internal class GradientBoostRegressionProgram
{
static void Main(string[] args)
{
Console.WriteLine("\nBegin C# Gradient Boosting" +
" regression demo ");
Console.WriteLine("Predict Income from Sex, Age," +
" City Height, Psychology ");
// 1. load data
Console.WriteLine("\nLoading people train" +
" (200) and test (40) data");
string trainFile =
"..\\..\\..\\Data\\People\\people_train_200.txt";
int[] colsX = new int[] { 0, 1, 2, 3, 4,5,6 };
double[][] trainX =
MatLoad(trainFile, colsX, ',', "#");
double[] trainY =
MatToVec(MatLoad(trainFile,
[7], ',', "#"));
string testFile =
"..\\..\\..\\Data\\People\\people_test_40.txt";
double[][] testX =
MatLoad(testFile, colsX, ',', "#");
double[] testY =
MatToVec(MatLoad(testFile,
[7], ',', "#"));
Console.WriteLine("Done ");
Console.WriteLine("\nFirst three raw train X: ");
Console.WriteLine("M,29,Austin,68.5,timid,54600.00 ");
Console.WriteLine("F,51,Juneau,60.0,brave,25700.00 ");
Console.WriteLine("M,32,Boston,73.5,alert,74900.00 ");
Console.WriteLine("\nFirst three train X: ");
for (int i = 0; i "lt" 3; ++i)
VecShow(trainX[i], 4, 8);
Console.WriteLine("\nFirst three train y: ");
for (int i = 0; i "lt" 3; ++i)
Console.WriteLine(trainY[i].ToString("F4").
PadLeft(8));
// 2. create and train model
int numTrees = 200;
int maxDepth = 4;
int minSamples = 2;
int minLeaf = 1;
int numSplitCols = -1; // use all columns
double lrnRate = 0.10;
Console.WriteLine("\nSetting numTrees = " +
numTrees);
Console.WriteLine("Setting maxDepth = " +
maxDepth);
Console.WriteLine("Setting minSamples = " +
minSamples);
Console.WriteLine("Setting minLeaf = " +
minLeaf);
Console.WriteLine("Setting lrnRate = " +
lrnRate.ToString("F4"));
Console.WriteLine("\nCreating and training" +
" GradientBoostRegression model ");
GradientBoostRegressor gbr =
new GradientBoostRegressor(numTrees, maxDepth,
minSamples, minLeaf, numSplitCols, lrnRate);
gbr.Train(trainX, trainY);
Console.WriteLine("Done ");
// 3. evaluate model
Console.WriteLine("\nEvaluating model ");
double accTrain = gbr.Accuracy(trainX, trainY, 0.10);
Console.WriteLine("\nAccuracy train (within 0.10) = " +
accTrain.ToString("F4"));
double accTest = gbr.Accuracy(testX, testY, 0.10);
Console.WriteLine("Accuracy test (within 0.10) = " +
accTest.ToString("F4"));
double mseTrain = gbr.MSE(trainX, trainY);
Console.WriteLine("\nMSE train = " +
mseTrain.ToString("F4"));
double mseTest = gbr.MSE(testX, testY);
Console.WriteLine("MSE test = " +
mseTest.ToString("F4"));
double r2Train = gbr.R2(trainX, trainY);
Console.WriteLine("\nR2 train = " +
r2Train.ToString("F4"));
double r2Test = gbr.R2(testX, testY);
Console.WriteLine("R2 test = " +
r2Test.ToString("F4"));
// 4. use model
double[] x = trainX[0];
Console.WriteLine("\nPredicting for x = ");
VecShow(x, 4, 9);
double predY = gbr.Predict(x, verbose: true);
Console.WriteLine("Predicted y = " +
predY.ToString("F4"));
Console.WriteLine("\nEnd demo ");
Console.ReadLine();
} // Main()
// ------------------------------------------------------
// helpers for Main()
// ------------------------------------------------------
static double[][] MatLoad(string fn, int[] usecols,
char sep, string comment)
{
List"lt"double[]"gt" result =
new List"lt"double[]"gt"();
string line = "";
FileStream ifs = new FileStream(fn, FileMode.Open);
StreamReader sr = new StreamReader(ifs);
while ((line = sr.ReadLine()) != null)
{
if (line.StartsWith(comment) == true)
continue;
string[] tokens = line.Split(sep);
List"lt"double"gt" lst = new List"lt"double"gt"();
for (int j = 0; j "lt" usecols.Length; ++j)
lst.Add(double.Parse(tokens[usecols[j]]));
double[] row = lst.ToArray();
result.Add(row);
}
sr.Close(); ifs.Close();
return result.ToArray();
}
static double[] MatToVec(double[][] mat)
{
int nRows = mat.Length;
int nCols = mat[0].Length;
double[] result = new double[nRows * nCols];
int k = 0;
for (int i = 0; i "lt" nRows; ++i)
for (int j = 0; j "lt" nCols; ++j)
result[k++] = mat[i][j];
return result;
}
static void VecShow(double[] vec, int dec, int wid)
{
for (int i = 0; i "lt" vec.Length; ++i)
Console.Write(vec[i].ToString("F" + dec).
PadLeft(wid));
Console.WriteLine("");
}
} // class Program
// ========================================================
public class GradientBoostRegressor
{
public double lrnRate;
public int nTrees;
public int maxDepth;
public int minSamples;
public int minLeaf;
public int numSplitCols;
public List"lt"DecisionTreeRegressor"gt" trees;
public double pred0; // initial prediction
private Random rnd;
public GradientBoostRegressor(int nTrees, int maxDepth,
int minSamples, int minLeaf, int numSplitCols,
double lrnRate, int seed=0)
{
this.nTrees = nTrees;
this.maxDepth = maxDepth;
this.minSamples = minSamples;
this.minLeaf = minLeaf;
this.numSplitCols = numSplitCols;
this.trees = new List"lt"DecisionTreeRegressor"gt"();
this.lrnRate = lrnRate;
this.rnd = new Random(seed);
}
public void Train(double[][] trainX, double[] trainY)
{
int n = trainX.Length;
this.pred0 = Mean(trainY);
double[] preds = new double[n]; //each data item
for (int i = 0; i "lt" n; ++i)
preds[i] = this.pred0;
double[] residuals = new double[n]; // for curr tree
for (int t = 0; t "lt" this.nTrees; ++t) // each tree
{
// double[] residuals = new double[n]; // curr tree
for (int i = 0; i "lt" n; ++i)
residuals[i] = trainY[i] - preds[i];
DecisionTreeRegressor dtr =
new DecisionTreeRegressor(this.maxDepth,
this.minSamples, this.minLeaf, this.numSplitCols,
false, this.rnd.Next(0,1_000_000));
dtr.Train(trainX, residuals); // predict residuals
for (int i = 0; i "lt" n; ++i)
{
double predResidual = dtr.Predict(trainX[i]);
preds[i] += this.lrnRate * predResidual;
}
this.trees.Add(dtr);
}
} // Train
public double Predict(double[] x, bool verbose = false)
{
double result = this.pred0;
if (verbose == true)
{
Console.WriteLine("\nInitial prediction: " +
result.ToString("F4"));
}
for (int t = 0; t "lt" this.nTrees; ++t)
{
double predResidual = this.trees[t].Predict(x);
double delta = this.lrnRate * predResidual;
result += delta;
if (verbose == true)
{
if (t "gte" 0 && t "lte" 4 ||
t "gte" this.nTrees - 2 &&
t "lte" this.nTrees - 1)
{
Console.Write("t = " + t.ToString().PadLeft(3) +
" pred_res = " + predResidual.ToString("F4").
PadLeft(10));
Console.Write(" delta = " + delta.ToString("F4").
PadLeft(10));
Console.WriteLine(" pred = " +
result.ToString("F4").PadLeft(7));
}
if (t == 5)
Console.WriteLine(". . . ");
}
}
return result;
}
public double Accuracy(double[][] dataX, double[] dataY,
double pctClose)
{
int numCorrect = 0; int numWrong = 0;
for (int i = 0; i "lt" dataX.Length; ++i)
{
double actualY = dataY[i];
double predY = Predict(dataX[i]);
if (Math.Abs(predY - actualY) "lt"
(pctClose * Math.Abs(actualY)))
++numCorrect;
else
++numWrong;
}
return (numCorrect * 1.0) / (numWrong + numCorrect);
}
public double MSE(double[][] dataX,
double[] dataY)
{
int n = dataX.Length;
double sum = 0.0;
for (int i = 0; i "lt" n; ++i)
{
double actualY = dataY[i];
double predY = this.Predict(dataX[i]);
sum += (actualY - predY) * (actualY - predY);
}
return sum / n;
}
public double R2(double[][] dataX, double[] dataY)
{
// coefficient of determination
int n = dataX.Length;
double sum = 0.0;
for (int i = 0; i "lt" n; ++i)
sum += dataY[i];
double meanActual = sum / n;
double ssRes = 0.0;
double ssTot = 0.0;
for (int i = 0; i "lt" n; ++i)
{
double predY = this.Predict(dataX[i]);
ssRes += (dataY[i] - predY) * (dataY[i] - predY);
ssTot += (dataY[i] - meanActual) *
(dataY[i] - meanActual);
}
if (Math.Abs(ssTot) "lt" 1.0e-12)
return 0.0;
else
return 1.0 - (ssRes / ssTot);
}
private static double Mean(double[] data)
{
int n = data.Length;
double sum = 0.0;
for (int i = 0; i "lt" n; ++i)
sum += data[i];
return sum / n;
}
} // class GradientBoostRegressor
// ========================================================
public class DecisionTreeRegressor
{
// ChatGPT: "I don't see any fundamental tree-building or
// indexing bugs. If the inputs are guaranteed to be
// finite doubles and the parameters are validated
// elsewhere, the core regression algorithm is correct."
public int maxDepth;
public int minSamples;
public int minLeaf;
public int numSplitCols;
public List"lt"Node"gt" tree;
public Random rnd;
public bool saveRows;
public double[][] trainX;
public double[] trainY;
// ............................................
public class Node
{
public int id;
public int colIdx = -1;
public double thresh = 0.0;
public int left = -1;
public int right = -1;
public double value = 0.0;
public bool isLeaf = false;
public List"lt"int"gt" rows;
}
// ............................................
public DecisionTreeRegressor(int maxDepth = 3,
int minSamples = 2, int minLeaf = 1,
int numSplitCols = -1, bool saveRows = false,
int seed = 0)
{
this.maxDepth = maxDepth;
this.minSamples = minSamples;
this.minLeaf = minLeaf;
this.numSplitCols = numSplitCols;
this.saveRows = saveRows;
this.tree = new List"lt"Node"gt"();
int numNodes = (int)Math.Pow(2, (maxDepth + 1)) - 1;
for (int i = 0; i "lt" numNodes; ++i)
this.tree.Add(null);
this.rnd = new Random(seed);
}
// ------------------------------------------------------
// ------------------------------------------------------
// public: ctor(), Train(), Predict()
// private: BestSplit(), TreeTargetMean()
// ------------------------------------------------------
public void Train(double[][] trainX, double[] trainY)
{
this.trainX = trainX;
this.trainY = trainY;
int maxParentID = (int)Math.Pow(2, this.maxDepth) - 2;
List"lt"int"gt" allRows =
new List"lt"int"gt"(this.trainX.Length);
for (int i = 0; i "lt" this.trainX.Length; ++i)
allRows.Add(i);
// construct root node
Node root = new Node()
{
id = 0,
value = this.TreeTargetMean(allRows),
rows = allRows
};
this.tree[0] = root;
for (int i = 0; i "lt" this.tree.Count; ++i)
{
Node currNode = this.tree[i];
if (currNode == null) continue;
int leftID = currNode.id * 2 + 1;
int rightID = currNode.id * 2 + 2;
// 1. check before expensive split calcs
if (currNode.id "gt" maxParentID ||
currNode.rows.Count "lt" this.minSamples ||
rightID "gte" this.tree.Count)
{
currNode.isLeaf = true;
continue;
}
// 2. find best split
double[] splitInfo = this.BestSplit(currNode.rows);
int colIdx = (int)splitInfo[0];
double splitVal = splitInfo[1];
if (colIdx == -1) // unable to split
{
currNode.isLeaf = true;
continue;
}
// 3. apply split properties
currNode.colIdx = colIdx; // aka split col
currNode.thresh = splitVal;
List"lt"int"gt" leftIdxs = new List"lt"int"gt"();
List"lt"int"gt" rightIdxs = new List"lt"int"gt"();
for (int k = 0; k "lt" currNode.rows.Count; ++k)
{
int r = currNode.rows[k];
if (this.trainX[r][colIdx] "lte" splitVal)
leftIdxs.Add(r);
else
rightIdxs.Add(r);
}
// 4. create and attach children
currNode.left = leftID;
this.tree[leftID] = new Node()
{
id = leftID,
rows = leftIdxs,
value = this.TreeTargetMean(leftIdxs)
};
currNode.right = rightID;
this.tree[rightID] = new Node()
{
id = rightID,
rows = rightIdxs,
value = this.TreeTargetMean(rightIdxs)
};
} // main loop
// delete row info (for ensembles)
if (this.saveRows == false)
{
for (int i = 0; i "lt" this.tree.Count; ++i)
if (this.tree[i] != null)
this.tree[i].rows = null;
}
} // Train()
// ------------------------------------------------------
public double Predict(double[] x)
{
int p = 0;
double lastValidValue = 0.0;
while (p != -1 && p "lt" this.tree.Count)
{
Node currNode = this.tree[p];
if (currNode == null) break;
lastValidValue = currNode.value;
if (currNode.isLeaf ||
currNode.colIdx == -1) break;
if (x[currNode.colIdx] "lte" currNode.thresh)
p = currNode.left;
else
p = currNode.right;
}
return lastValidValue;
}
// ------------------------------------------------------
private double[] BestSplit(List"lt"int"gt" rows)
{
int bestColIdx = -1;
double bestThresh = 0.0;
double bestGain = -1.0;
int nRows = rows.Count;
int nCols = this.trainX[0].Length;
int[] colIndices = new int[nCols];
for (int k = 0; k "lt" nCols; ++k)
colIndices[k] = k;
// feature sampling
for (int i = 0; i "lt" nCols; ++i)
{
int ri = rnd.Next(i, nCols);
int tmp = colIndices[i];
colIndices[i] = colIndices[ri];
colIndices[ri] = tmp;
}
int numColsToUse;
if (this.numSplitCols == -1)
numColsToUse = nCols;
else
numColsToUse = Math.Min(this.numSplitCols, nCols);
int[] sortedRows = new int[nRows];
double[] featureKeys = new double[nRows];
double totalSum = 0.0;
for (int i = 0; i "lt" nRows; ++i)
totalSum += this.trainY[rows[i]];
for (int j = 0; j "lt" numColsToUse; ++j)
{
int colIdx = colIndices[j];
for (int i = 0; i "lt" nRows; ++i)
{
int r = rows[i];
sortedRows[i] = r;
featureKeys[i] = this.trainX[r][colIdx];
}
Array.Sort(featureKeys, sortedRows);
double leftSum = 0.0;
for (int i = 0; i "lt" nRows - 1; ++i)
{
leftSum += this.trainY[sortedRows[i]];
int leftCount = i + 1;
int rightCount = nRows - leftCount;
if (featureKeys[i] == featureKeys[i + 1])
continue; // duplicate values
if (leftCount "lt" this.minLeaf ||
rightCount "lt" this.minLeaf)
continue;
double rightSum = totalSum - leftSum;
// maximizing Gain = (G_L^2 / N_L) + (G_R^2 / N_R)
// equivalent to maximizing explicit weighted
// variance reduction
double gain = (leftSum * leftSum / leftCount) +
(rightSum * rightSum / rightCount);
if (gain "gt" bestGain)
{
bestGain = gain;
bestColIdx = colIdx;
// bestThresh =
// (featureKeys[i] + featureKeys[i + 1]) / 2.0;
bestThresh = featureKeys[i] +
(featureKeys[i + 1] - featureKeys[i]) / 2.0;
}
}
}
// using a tuple would create version dependency
double[] result = new double[2];
result[0] = 1.0 * bestColIdx;
result[1] = bestThresh;
return result;
}
private double TreeTargetMean(List"lt"int"gt" rows)
{
if (rows == null || rows.Count == 0)
return 0.0;
double sum = 0.0;
for (int i = 0; i "lt" rows.Count; ++i)
sum += this.trainY[rows[i]];
return sum / rows.Count;
}
} // DecisionTreeRegressor
// ========================================================
Demo scikit program.
# scikit_gradient_boost_regression.py
# on custom People dataset
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
# -----------------------------------------------------------
def accuracy(model, data_X, data_y, pct_close):
# assumes model has a predict(X)
n = len(data_X)
n_correct = 0; n_wrong = 0
for i in range(n):
x = data_X[i].reshape(1,-1) # make it a matrix
y = data_y[i]
y_pred = model.predict(x)[0] # predict() expects 2D
if np.abs(y - y_pred) "lt" np.abs(y) * pct_close:
n_correct += 1
else:
n_wrong += 1
# print("Correct = " + str(n_correct))
# print("Wrong = " + str(n_wrong))
return n_correct / (n_correct + n_wrong)
# -----------------------------------------------------------
def MSE(model, data_X, data_y):
n = len(data_X)
sum = 0.0
for i in range(n):
x = data_X[i].reshape(1,-1)
y = data_y[i]
y_pred = model.predict(x)[0]
sum += (y - y_pred) * (y - y_pred)
return sum / n
# -----------------------------------------------------------
def main():
print("\nBegin scikit gradient boost regression demo ")
print("Predict Income from Sex, Age, City, Height, Psych ")
np.set_printoptions(precision=4, suppress=True,
floatmode='fixed')
np.random.seed(0) # not used this version
# 1. load data
print("\nLoading People train (200), test (400) data ")
train_file = ".\\Data\\People\\people_train_200.txt"
test_file = ".\\Data\\People\\people_test_40.txt"
# sex, age, city, height, psych(3)
x_cols=[0,1,2,3, 4,5,6]
target_col = 7 # 1.0000 .6413
train_X = np.loadtxt(train_file, comments="#",
usecols=x_cols, delimiter=",", dtype=np.float64)
train_y = np.loadtxt(train_file, comments="#",
usecols=target_col, delimiter=",", dtype=np.float64)
test_X = np.loadtxt(test_file, comments="#",
usecols=x_cols, delimiter=",", dtype=np.float64)
test_y = np.loadtxt(test_file, comments="#",
usecols=target_col, delimiter=",", dtype=np.float64)
print("Done ")
print("\nFirst three raw train X: ")
print("M,29,Austin,68.5,timid,54600.00 ")
print("F,51,Juneau,60.0,brave,25700.00 ")
print("M,32,Boston,73.5,alert,74900.00 ")
print("\nFirst three X predictors: ")
print(train_X[0:3,:])
print("\nFirst three y targets: ")
for i in range(3):
print("%0.4f" % train_y[i])
n_estimators = 200
max_depth = 4
min_samples_split = 2
min_samples_leaf = 1
lrn_rate = 0.05
seed = 0
print("\nSetting n_estimators = " + \
str(n_estimators))
print("Setting max_depth = " + \
str(max_depth))
print("Setting min_samples_split = " + \
str(min_samples_split))
print("Setting min_samples_leaf = " + \
str(min_samples_leaf))
print("Setting lrn_rate = %0.4f " % lrn_rate)
# GradientBoostingRegressor(*, loss='squared_error',
# learning_rate=0.1, n_estimators=100, subsample=1.0,
# criterion='friedman_mse', min_samples_split=2,
# min_samples_leaf=1, min_weight_fraction_leaf=0.0,
# max_depth=3, min_impurity_decrease=0.0, init=None,
# random_state=None, max_features=None, alpha=0.9,
# verbose=0, max_leaf_nodes=None, warm_start=False,
# validation_fraction=0.1, n_iter_no_change=None,
# tol=0.0001, ccp_alpha=0.0)
model = \
GradientBoostingRegressor(max_depth=max_depth,
min_samples_split=min_samples_split,
n_estimators=n_estimators,
min_samples_leaf=min_samples_leaf,
learning_rate=lrn_rate, random_state=seed)
print("\nTraining gradient boost regression model ")
model.fit(train_X, train_y)
print("Done " )
print("\nEvaluating model ")
acc_train = accuracy(model, train_X, train_y, 0.10)
print("\nAcc train (within 0.10): %0.4f " % acc_train)
acc_test = accuracy(model, test_X, test_y, 0.10)
print("Acc test (within 0.10): %0.4f " % acc_test)
mse_train = MSE(model, train_X, train_y)
print("\nMSE train: %0.4f " % mse_train)
mse_test = MSE(model, test_X, test_y)
print("MSE test: %0.4f " % mse_test)
r2_train = model.score(train_X, train_y)
print("\nR2 train: %0.4f " % r2_train)
r2_test = model.score(test_X, test_y)
print("R2 test: %0.4f " % r2_test)
print("\nPredicting for x = ")
print(train_X[0])
pred_y = model.predict(train_X[0].reshape(1,-1))[0]
print("Predicted y = %0.4f " % pred_y)
print("\nEnd demo ")
if __name__ == "__main__":
main()
Training data:
# people_train_200.txt
#
# M = 0, F = 1
# city_target_encoding = {
# "Boston": 30600.00, 0
# "Casper": 33100.00, 1
# "Austin": 33961.90, 2
# "Denver": 35535.00, 3
# "Eugene": 44090.00, 4
# "Irvine": 51840.00, 5
# "Fresno": 52950.00, 6
# "Gallup": 52960.00, 7
# "Juneau": 58842.86, 8
# "Helena": 64194.74, 9
# }
# timid = 1,0,0 alert = 0,1,0 brave = 0,0,1
#
# Sex Age City Height Psych Income
#
0, 0.29, 2, 0.685, 1,0,0, 0.54600
1, 0.51, 8, 0.600, 0,0,1, 0.25700
0, 0.32, 0, 0.735, 0,1,0, 0.74900
0, 0.19, 5, 0.610, 0,0,1, 0.30300
1, 0.45, 4, 0.645, 0,1,0, 0.35600
0, 0.58, 6, 0.720, 1,0,0, 0.52100
1, 0.22, 2, 0.630, 0,1,0, 0.27800
0, 0.64, 8, 0.775, 0,0,1, 0.88200
1, 0.38, 3, 0.660, 1,0,0, 0.32400
0, 0.41, 7, 0.700, 0,1,0, 0.51200
1, 0.27, 1, 0.625, 0,0,1, 0.28900
0, 0.53, 9, 0.740, 1,0,0, 0.62300
1, 0.34, 0, 0.650, 0,1,0, 0.33100
0, 0.60, 5, 0.765, 0,0,1, 0.79800
1, 0.21, 4, 0.605, 1,0,0, 0.22400
0, 0.48, 6, 0.690, 0,1,0, 0.48700
1, 0.62, 2, 0.670, 0,0,1, 0.53600
0, 0.25, 8, 0.715, 1,0,0, 0.45200
1, 0.39, 3, 0.640, 0,1,0, 0.31800
0, 0.55, 7, 0.750, 0,0,1, 0.72100
1, 0.31, 1, 0.615, 1,0,0, 0.24500
0, 0.43, 9, 0.705, 0,1,0, 0.52400
1, 0.50, 0, 0.635, 0,0,1, 0.38900
0, 0.28, 5, 0.730, 0,1,0, 0.58100
1, 0.56, 4, 0.665, 1,0,0, 0.47800
0, 0.20, 6, 0.620, 0,0,1, 0.29100
1, 0.42, 2, 0.655, 0,1,0, 0.37200
0, 0.66, 8, 0.780, 1,0,0, 0.85400
1, 0.36, 3, 0.620, 0,0,1, 0.29800
0, 0.49, 7, 0.725, 0,1,0, 0.59300
1, 0.24, 1, 0.600, 0,1,0, 0.21200
0, 0.57, 9, 0.755, 0,0,1, 0.76400
1, 0.46, 0, 0.640, 1,0,0, 0.36100
0, 0.33, 5, 0.680, 0,1,0, 0.44300
1, 0.61, 4, 0.675, 0,0,1, 0.57900
0, 0.23, 6, 0.710, 1,0,0, 0.42800
1, 0.52, 2, 0.630, 0,1,0, 0.39400
0, 0.65, 8, 0.760, 0,0,1, 0.81200
1, 0.30, 3, 0.610, 1,0,0, 0.23100
0, 0.44, 7, 0.695, 0,1,0, 0.49600
1, 0.59, 1, 0.680, 0,0,1, 0.58300
0, 0.35, 9, 0.720, 1,0,0, 0.49800
1, 0.18, 0, 0.605, 0,1,0, 0.21900
0, 0.51, 5, 0.745, 0,0,1, 0.66700
1, 0.40, 4, 0.650, 1,0,0, 0.33500
0, 0.67, 6, 0.785, 0,1,0, 0.94200
1, 0.26, 2, 0.625, 0,0,1, 0.27400
0, 0.37, 8, 0.690, 0,1,0, 0.45100
1, 0.54, 3, 0.660, 1,0,0, 0.43600
0, 0.30, 7, 0.730, 0,0,1, 0.59800
1, 0.47, 1, 0.645, 0,1,0, 0.38200
0, 0.63, 9, 0.770, 1,0,0, 0.83100
1, 0.28, 0, 0.615, 0,0,1, 0.26300
0, 0.42, 5, 0.700, 0,1,0, 0.48900
1, 0.35, 4, 0.635, 1,0,0, 0.28700
0, 0.59, 6, 0.750, 0,0,1, 0.74600
1, 0.20, 2, 0.610, 0,1,0, 0.22800
0, 0.46, 8, 0.720, 1,0,0, 0.53200
1, 0.58, 3, 0.670, 0,1,0, 0.51400
0, 0.31, 7, 0.715, 0,0,1, 0.52900
1, 0.33, 1, 0.620, 1,0,0, 0.25800
0, 0.54, 9, 0.735, 0,1,0, 0.64500
1, 0.41, 0, 0.655, 0,0,1, 0.36700
0, 0.22, 5, 0.630, 1,0,0, 0.26400
1, 0.67, 4, 0.685, 0,1,0, 0.69100
0, 0.38, 6, 0.695, 0,0,1, 0.47900
1, 0.25, 2, 0.600, 1,0,0, 0.19800
0, 0.61, 8, 0.760, 0,1,0, 0.77400
1, 0.43, 3, 0.640, 0,0,1, 0.35200
0, 0.26, 7, 0.700, 1,0,0, 0.43100
1, 0.48, 1, 0.650, 0,1,0, 0.39700
0, 0.50, 9, 0.725, 0,0,1, 0.60200
1, 0.29, 0, 0.620, 1,0,0, 0.24900
0, 0.68, 5, 0.785, 0,1,0, 0.96800
1, 0.37, 4, 0.630, 0,0,1, 0.30600
0, 0.18, 6, 0.615, 1,0,0, 0.25300
1, 0.65, 2, 0.680, 0,1,0, 0.63200
0, 0.34, 8, 0.685, 0,0,1, 0.43800
1, 0.53, 3, 0.655, 1,0,0, 0.41900
0, 0.45, 7, 0.710, 0,1,0, 0.53700
1, 0.23, 1, 0.610, 0,0,1, 0.23600
0, 0.52, 9, 0.740, 1,0,0, 0.61100
1, 0.32, 0, 0.630, 0,1,0, 0.28400
0, 0.62, 5, 0.770, 0,0,1, 0.82500
1, 0.19, 4, 0.600, 1,0,0, 0.20100
0, 0.40, 6, 0.700, 0,1,0, 0.47300
1, 0.60, 2, 0.675, 0,0,1, 0.56800
0, 0.27, 8, 0.695, 1,0,0, 0.41200
1, 0.44, 3, 0.645, 0,1,0, 0.36900
0, 0.56, 7, 0.755, 0,0,1, 0.75300
1, 0.36, 1, 0.635, 1,0,0, 0.29100
0, 0.47, 9, 0.715, 0,1,0, 0.55800
1, 0.55, 0, 0.660, 0,0,1, 0.46200
0, 0.21, 5, 0.625, 0,1,0, 0.28900
1, 0.63, 4, 0.680, 1,0,0, 0.58400
0, 0.39, 6, 0.690, 0,0,1, 0.46100
1, 0.22, 2, 0.615, 1,0,0, 0.21700
0, 0.64, 8, 0.770, 0,1,0, 0.85900
1, 0.38, 3, 0.635, 0,0,1, 0.33800
0, 0.29, 7, 0.705, 1,0,0, 0.45800
1, 0.51, 1, 0.655, 0,1,0, 0.43100
0, 0.58, 9, 0.760, 0,0,1, 0.78200
1, 0.27, 0, 0.610, 0,1,0, 0.24300
0, 0.36, 5, 0.685, 1,0,0, 0.42100
1, 0.66, 4, 0.685, 0,0,1, 0.67500
0, 0.24, 6, 0.640, 0,1,0, 0.31200
1, 0.49, 2, 0.650, 1,0,0, 0.38400
0, 0.60, 8, 0.755, 0,0,1, 0.76900
1, 0.31, 3, 0.625, 0,1,0, 0.26800
0, 0.43, 7, 0.700, 1,0,0, 0.48200
1, 0.57, 1, 0.665, 0,0,1, 0.51900
0, 0.32, 9, 0.690, 0,1,0, 0.43900
1, 0.20, 0, 0.600, 1,0,0, 0.19200
0, 0.53, 5, 0.750, 0,1,0, 0.68400
1, 0.42, 4, 0.640, 0,0,1, 0.36100
0, 0.65, 6, 0.775, 1,0,0, 0.84700
1, 0.28, 2, 0.620, 0,1,0, 0.26200
0, 0.35, 8, 0.680, 1,0,0, 0.39600
1, 0.50, 3, 0.650, 0,0,1, 0.41800
0, 0.25, 7, 0.690, 0,1,0, 0.41300
1, 0.45, 1, 0.640, 1,0,0, 0.34800
0, 0.59, 9, 0.765, 0,1,0, 0.80100
1, 0.33, 0, 0.625, 0,0,1, 0.30200
0, 0.23, 5, 0.635, 0,1,0, 0.29700
1, 0.64, 4, 0.680, 0,0,1, 0.62100
0, 0.41, 6, 0.705, 1,0,0, 0.47100
1, 0.21, 2, 0.605, 0,0,1, 0.21800
0, 0.67, 8, 0.785, 0,1,0, 0.95600
1, 0.37, 3, 0.630, 1,0,0, 0.29200
0, 0.48, 7, 0.720, 0,0,1, 0.59100
1, 0.26, 1, 0.615, 0,1,0, 0.26100
0, 0.55, 9, 0.745, 1,0,0, 0.65800
1, 0.40, 0, 0.645, 0,1,0, 0.34900
0, 0.19, 5, 0.615, 0,0,1, 0.28400
1, 0.68, 4, 0.690, 1,0,0, 0.70200
0, 0.36, 6, 0.680, 0,1,0, 0.42600
1, 0.29, 2, 0.625, 0,0,1, 0.29300
0, 0.62, 8, 0.765, 1,0,0, 0.79300
1, 0.44, 3, 0.640, 0,1,0, 0.35700
0, 0.28, 7, 0.700, 0,0,1, 0.46900
1, 0.52, 1, 0.660, 1,0,0, 0.42100
0, 0.46, 9, 0.710, 0,1,0, 0.53900
1, 0.24, 0, 0.605, 0,0,1, 0.20700
0, 0.57, 5, 0.755, 1,0,0, 0.75100
1, 0.35, 4, 0.630, 0,1,0, 0.29600
0, 0.66, 6, 0.780, 0,0,1, 0.88900
1, 0.30, 2, 0.620, 0,0,1, 0.29700
0, 0.38, 8, 0.690, 1,0,0, 0.42400
1, 0.56, 3, 0.665, 0,1,0, 0.48300
0, 0.27, 7, 0.695, 0,0,1, 0.44800
1, 0.43, 1, 0.635, 1,0,0, 0.32800
0, 0.61, 9, 0.770, 0,1,0, 0.83600
1, 0.31, 0, 0.620, 0,0,1, 0.28100
0, 0.20, 5, 0.620, 1,0,0, 0.24600
1, 0.65, 4, 0.685, 0,1,0, 0.66400
0, 0.42, 6, 0.700, 0,0,1, 0.51200
1, 0.18, 2, 0.600, 0,1,0, 0.18900
0, 0.63, 8, 0.775, 1,0,0, 0.82900
1, 0.39, 3, 0.635, 0,1,0, 0.32600
0, 0.50, 7, 0.725, 0,0,1, 0.63400
1, 0.25, 1, 0.610, 1,0,0, 0.22300
0, 0.54, 9, 0.740, 0,1,0, 0.65200
1, 0.36, 0, 0.635, 1,0,0, 0.29800
0, 0.31, 5, 0.675, 0,0,1, 0.42900
1, 0.58, 4, 0.670, 0,1,0, 0.52800
0, 0.25, 6, 0.645, 1,0,0, 0.31900
1, 0.47, 2, 0.645, 0,0,1, 0.41100
0, 0.59, 8, 0.750, 0,1,0, 0.73800
1, 0.28, 3, 0.615, 1,0,0, 0.23700
0, 0.40, 7, 0.690, 0,1,0, 0.45600
1, 0.54, 1, 0.650, 0,0,1, 0.46300
0, 0.34, 9, 0.695, 1,0,0, 0.43200
1, 0.23, 0, 0.600, 0,1,0, 0.20800
0, 0.52, 5, 0.740, 0,0,1, 0.66100
1, 0.41, 4, 0.640, 1,0,0, 0.32900
0, 0.68, 6, 0.785, 0,1,0, 0.97800
1, 0.26, 2, 0.610, 0,0,1, 0.23800
0, 0.37, 8, 0.685, 0,1,0, 0.40900
1, 0.50, 3, 0.655, 1,0,0, 0.42700
0, 0.29, 7, 0.700, 0,0,1, 0.46300
1, 0.46, 1, 0.640, 0,1,0, 0.35400
0, 0.60, 9, 0.760, 1,0,0, 0.80800
1, 0.28, 0, 0.610, 0,0,1, 0.26800
0, 0.43, 5, 0.705, 0,1,0, 0.47600
1, 0.34, 4, 0.625, 1,0,0, 0.28300
0, 0.56, 6, 0.745, 0,0,1, 0.70400
1, 0.19, 2, 0.600, 0,1,0, 0.17400
0, 0.65, 8, 0.770, 1,0,0, 0.86800
1, 0.42, 3, 0.635, 0,0,1, 0.35900
0, 0.30, 7, 0.695, 0,1,0, 0.42200
1, 0.53, 1, 0.655, 1,0,0, 0.44800
0, 0.47, 9, 0.720, 0,0,1, 0.59400
1, 0.22, 0, 0.600, 0,1,0, 0.19800
0, 0.66, 5, 0.780, 1,0,0, 0.89200
1, 0.38, 4, 0.630, 0,0,1, 0.31800
0, 0.21, 6, 0.620, 0,1,0, 0.27100
1, 0.62, 2, 0.665, 1,0,0, 0.50300
0, 0.33, 8, 0.675, 0,0,1, 0.41200
1, 0.51, 3, 0.650, 0,1,0, 0.41500
0, 0.44, 7, 0.705, 1,0,0, 0.48800
Test data:
# people_test_40.txt # 1, 0.27, 1, 0.620, 0,0,1, 0.28100 0, 0.58, 9, 0.750, 0,1,0, 0.75900 1, 0.32, 0, 0.625, 1,0,0, 0.25900 0, 0.24, 5, 0.640, 0,0,1, 0.32400 1, 0.60, 4, 0.660, 0,1,0, 0.52100 0, 0.39, 6, 0.685, 1,0,0, 0.42700 0, 0.26, 2, 0.690, 0,1,0, 0.40200 1, 0.49, 8, 0.645, 0,0,1, 0.38800 0, 0.53, 3, 0.730, 1,0,0, 0.61200 1, 0.30, 7, 0.615, 0,1,0, 0.23400 0, 0.61, 1, 0.760, 0,0,1, 0.78900 1, 0.22, 9, 0.610, 1,0,0, 0.21500 0, 0.45, 0, 0.700, 0,1,0, 0.49300 1, 0.57, 5, 0.670, 0,0,1, 0.51100 0, 0.28, 4, 0.695, 1,0,0, 0.40800 1, 0.39, 6, 0.630, 0,1,0, 0.31100 0, 0.51, 2, 0.720, 0,0,1, 0.57400 1, 0.33, 8, 0.625, 1,0,0, 0.26700 0, 0.64, 3, 0.770, 0,1,0, 0.83500 1, 0.18, 7, 0.600, 0,0,1, 0.16200 0, 0.42, 1, 0.695, 1,0,0, 0.46100 1, 0.59, 9, 0.680, 0,1,0, 0.56300 0, 0.36, 0, 0.685, 0,0,1, 0.43400 1, 0.23, 5, 0.610, 1,0,0, 0.22100 0, 0.67, 4, 0.780, 0,1,0, 0.92400 1, 0.45, 6, 0.645, 0,0,1, 0.37800 0, 0.31, 2, 0.700, 1,0,0, 0.43500 1, 0.52, 8, 0.655, 0,1,0, 0.43900 0, 0.20, 3, 0.620, 0,0,1, 0.26200 1, 0.48, 7, 0.650, 1,0,0, 0.37100 0, 0.55, 1, 0.740, 0,1,0, 0.63800 1, 0.29, 9, 0.620, 0,0,1, 0.27300 0, 0.63, 0, 0.765, 1,0,0, 0.80100 1, 0.41, 5, 0.640, 0,1,0, 0.34200 0, 0.22, 4, 0.630, 0,0,1, 0.28600 1, 0.64, 6, 0.675, 1,0,0, 0.61400 0, 0.37, 2, 0.690, 0,1,0, 0.44100 1, 0.50, 8, 0.650, 0,0,1, 0.42200 0, 0.27, 3, 0.695, 1,0,0, 0.39800 1, 0.44, 7, 0.640, 0,1,0, 0.33900
.NET Test Automation Recipes
Software Testing
SciPy Programming Succinctly
Keras Succinctly
R Programming
Visual Studio Live
Microsoft MLADS Conference
DevIntersection Conference
Machine Learning Week
Ai4 Conference
G2E Conference
iSC West Conference
You must be logged in to post a comment.