James D. McCaffrey

Years ago, when I was first exposed to tree-based regression techniques, most of the resources talked about bagging tree regression and random forest regression as if they were two distinct techniques. But in fact, bagging tree regression is just a special case of random forest regression.

I suspect the cause of the confusion is the fact that bagging (“bootstrap aggregation”) tree regression was first described in 1994 paper by a fellow named Leo Breiman. Then seven years later, Breiman slightly modified bagging tree regression and called the new technique random forest regression. So, even though bagging tree regression is just a special case of random forest regression, the term bagging tree regression had been in use for seven years so there was no getting rid of that term.

A simple decision tree for regression will almost always overfit the training data where the model accuracy on the training data is near 100% but accuracy on new, previously unseen data is very poor.

The simple idea of bagging tree regression is to create a collection/ensemble of many simple decision trees, where each tree is trained on a different subset of the rows of the training data. For example, if the source set of training data has 200 rows, you could create 50 simple trees, where each tree is trained on 180 randomly selected rows. The selection is done “with replacement” so for a random subset of training data, some rows will be used more than once and some rows won’t be used at all.

After all the trees are trained, a prediction for an input vector x is just the average of the predictions of the trees in the collection. A very simple and crude idea that sometimes works well, and sometimes doesn’t work very well.

Random forest regression exapnds the idea to include randomly selected columns too, but in a subtle way. Instead of creating subsets of training data that are fixed, random forest regression selects random columns during each split in the split process that constructs the decision trees. For example, if a set of training data has 8 columns of predictors, you might specify to use just 6 randomly selected columns when computing each split.

Because there are many splits computed when a decision tree is built, there is quite a bit of variability introduced into each tree, which acts as a kind of regularization, which discourages model overfitting.

Put another way, bagging tree regression is just random forest regression where all columns are used when computing each split.

The implication is that there’s no need to implement a separate BaggingTreeRegression model — a RandomForestRegression model, with number columns set to the number of columns in the training data has the exact same functionality as bagging tree regression.

Just for fun, I implemented a dedicated BaggingTreeRegession model from scratch, using C#, and a dedicated RandomForestRegression model. I used some synthetic training and test data that has 5 columns of predictor variables. When the random forest model was trained using nCols = 5, the results were the same as the begging tree regression model.

The output of the random forest model:

Begin C# Random Forest regression demo

Loading synthetic train (200) and test (40) data
Done

First three train X:
 -0.1660  0.4406 -0.9998 -0.3953 -0.7065
  0.0776 -0.1616  0.3704 -0.5911  0.7562
 -0.9452  0.3409 -0.1654  0.1174 -0.7192

First three train y:
  0.4840
  0.1568
  0.8054

Setting nTrees = 100
Setting maxDepth = 6
Setting minSamples = 2
Setting minLeaf = 1
Setting nCols = 5
Setting nRows = 150

Creating and training random forest regression model
Done

Accuracy train (within 0.10) = 0.8050
Accuracy test (within 0.10) = 0.5750

MSE train = 0.0006
MSE test = 0.0016

Predicting for x =
  -0.1660   0.4406  -0.9998  -0.3953  -0.7065
y = 0.4728

End demo

The key output statements of the dedicated bagging tree regression program are identical, as expected:

Begin C# Bagging Tree regression demo

Setting nTrees = 100
Setting maxDepth = 6
Setting minSamples = 2
Setting minLeaf = 1
Setting nRows = 150
(Using all columns)

Creating and training BaggingTreeRegression model
Done

Accuracy train (within 0.10) = 0.8050
Accuracy test (within 0.10) = 0.5750

MSE train = 0.0006
MSE test = 0.0016

Predicting for x =
  -0.1660   0.4406  -0.9998  -0.3953  -0.7065
y = 0.4728

Notice that the model still overfits quite a bit. This is because the synthetic data is more or less spead evenly. Because of this weakness, the more powerful gradient boosting regression model is often used instead of bagging tree regression or random forest regression. That said however, for some problem scenarios, bagging tree and random forest regression work quite well.

Machine learning trees are good. Some trees in old science fiction movies are not good.

Right: “Amazons” (1986) – A group of reptile men kidnap virgins from neighboring villages to sacrifce them to a demonic tree. The tree has a very creepy face, but it’s not clear if he intends to eat the sacrificial virgins or not. My personal grade = C-.

Demo program. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols (my blog editor often chokes on symbols).

using System;
using System.IO;
using System.Collections.Generic;

namespace RandomForestRegression
{
  internal class RandomForestRegressionProgram
  {
    static void Main(string[] args)
    {
      Console.WriteLine("\nBegin C# Random Forest" +
        " regression demo ");

      // 1. load data
      Console.WriteLine("\nLoading synthetic train (200)" +
        " and test (40) data");
      string trainFile =
        "..\\..\\..\\Data\\synthetic_train_200.txt";
      int[] colsX = new int[] { 0, 1, 2, 3, 4 };
      double[][] trainX =
        MatLoad(trainFile, colsX, ',', "#");
      double[] trainY =
        MatToVec(MatLoad(trainFile,
        new int[] { 5 }, ',', "#"));

      string testFile =
        "..\\..\\..\\Data\\synthetic_test_40.txt";
      double[][] testX =
        MatLoad(testFile, colsX, ',', "#");
      double[] testY =
        MatToVec(MatLoad(testFile,
        new int[] { 5 }, ',', "#"));
      Console.WriteLine("Done ");

      Console.WriteLine("\nFirst three train X: ");
      for (int i = 0; i "lt" 3; ++i)
        VecShow(trainX[i], 4, 8);

      Console.WriteLine("\nFirst three train y: ");
      for (int i = 0; i "lt" 3; ++i)
        Console.WriteLine(trainY[i].ToString("F4").
          PadLeft(8));

      // 2. create and train model
      //int nTrees = 150;  // aka n_estimators
      //int maxDepth = 8;
      //int minSamples = 2;
      //int minLeaf = 1;
      //int nCols = 4;  // aka max_features
      //int nRows = 190;  // aka max_samples

      int nTrees = 100;
      int maxDepth = 6;
      int minSamples = 2;
      int minLeaf = 1;
      int nCols = 5;  // use all
      int nRows = 150;  // train data for each tree

      Console.WriteLine("\nSetting nTrees = " + nTrees);
      Console.WriteLine("Setting maxDepth = " + maxDepth);
      Console.WriteLine("Setting minSamples = " + 
        minSamples);
      Console.WriteLine("Setting minLeaf = " + minLeaf);
      Console.WriteLine("Setting nCols = " + nCols);
      Console.WriteLine("Setting nRows = " + nRows);

      Console.WriteLine("\nCreating and training" +
        " random forest regression model ");
      RandomForestRegressor rfr =
        new RandomForestRegressor(nTrees, maxDepth,
        minSamples, minLeaf, nCols, nRows, seed: 0);
      rfr.Train(trainX, trainY);
      Console.WriteLine("Done ");

      // 3. evaluate model
      double accTrain = rfr.Accuracy(trainX, trainY, 0.10);
      Console.WriteLine("\nAccuracy train (within 0.10) = " +
        accTrain.ToString("F4"));
      double accTest = rfr.Accuracy(testX, testY, 0.10);
      Console.WriteLine("Accuracy test (within 0.10) = " +
        accTest.ToString("F4"));

      double mseTrain = rfr.MSE(trainX, trainY);
      Console.WriteLine("\nMSE train = " +
        mseTrain.ToString("F4"));
      double mseTest = rfr.MSE(testX, testY);
      Console.WriteLine("MSE test = " +
        mseTest.ToString("F4"));

      // 4. use model
      double[] x = trainX[0];
      Console.WriteLine("\nPredicting for x = ");
      VecShow(x, 4, 9);
      double predY = rfr.Predict(x);
      Console.WriteLine("y = " + predY.ToString("F4"));

      Console.WriteLine("\nEnd demo ");
      Console.ReadLine();
    } // Main()

    // ------------------------------------------------------
    // helpers for Main()
    // ------------------------------------------------------

    static double[][] MatLoad(string fn, int[] usecols,
      char sep, string comment)
    {
      List"lt"double[]"gt" result =
        new List"lt"double[]"gt"();
      string line = "";
      FileStream ifs = new FileStream(fn, FileMode.Open);
      StreamReader sr = new StreamReader(ifs);
      while ((line = sr.ReadLine()) != null)
      {
        if (line.StartsWith(comment) == true)
          continue;
        string[] tokens = line.Split(sep);
        List"lt"double"gt" lst = new List"lt"double"gt"();
        for (int j = 0; j "lt" usecols.Length; ++j)
          lst.Add(double.Parse(tokens[usecols[j]]));
        double[] row = lst.ToArray();
        result.Add(row);
      }
      sr.Close(); ifs.Close();
      return result.ToArray();
    }

    static double[] MatToVec(double[][] M)
    {
      int nRows = M.Length;
      int nCols = M[0].Length;
      double[] result = new double[nRows * nCols];
      int k = 0;
      for (int i = 0; i "lt" nRows; ++i)
        for (int j = 0; j "lt" nCols; ++j)
          result[k++] = M[i][j];
      return result;
    }

    static void VecShow(double[] vec, int dec, int wid)
    {
      for (int i = 0; i "lt" vec.Length; ++i)
        Console.Write(vec[i].ToString("F" + dec).
          PadLeft(wid));
      Console.WriteLine("");
    }

  } // class Program

  // ========================================================

  public class RandomForestRegressor
  {
    public int nTrees;
    public int maxDepth;
    public int minSamples;
    public int minLeaf;
    public int nSplitCols;  // num cols each split
    public int nRows;
    public List"lt"DecisionTreeRegressor"gt" trees;
    public Random rnd;

    public RandomForestRegressor(int nTrees, int maxDepth,
      int minSamples, int minLeaf, int nCols, int nRows,
      int seed = 0)
    {
      this.nTrees = nTrees;
      this.maxDepth = maxDepth;
      this.minSamples = minSamples;
      this.minLeaf = minLeaf;
      this.nRows = nRows;
      this.nSplitCols = nCols;  // pass to DecisionTree
      this.trees = 
        new List"lt"DecisionTreeRegressor"gt"();
      this.rnd = new Random(seed);
    }

    // ------------------------------------------------------

    public void Train(double[][] trainX, double[] trainY)
    {
      int nr = trainX.Length;
      // int nc = trainX[0].Length;

      for (int i = 0; i "lt" this.nTrees; ++i)
      {
        // construct data rows-subset 
        int[] rndRows =
          GetRandomRowIdxs(nr, this.nRows, this.rnd);
        double[][] subsetX =
          MakeRowsSubsetX(trainX, rndRows);
        double[] subsetY =
          MakeSubsetY(trainY, rndRows);

        // make and train curr tree
        DecisionTreeRegressor dtr =
          new DecisionTreeRegressor(this.maxDepth,
          this.minSamples, this.minLeaf, this.nSplitCols);
        // random cols will be used during splits
        dtr.Train(subsetX, subsetY);
        this.trees.Add(dtr);
      } // i
    }

    // ------------------------------------------------------

    public double Predict(double[] x)
    {
      double sum = 0.0;
      for (int t = 0; t "lt" this.nTrees; ++t)
        sum += this.trees[t].Predict(x);
      return sum / this.nTrees;
    }

    // ------------------------------------------------------

    public double Accuracy(double[][] dataX, double[] dataY,
      double pctClose)
    {
      int numCorrect = 0; int numWrong = 0;
      for (int i = 0; i "lt" dataX.Length; ++i)
      {
        double actualY = dataY[i];
        double predY = this.Predict(dataX[i]);
        if (Math.Abs(predY - actualY) "lt"
          Math.Abs(pctClose * actualY))
          ++numCorrect;
        else
          ++numWrong;
      }
      return (numCorrect * 1.0) / (numWrong + numCorrect);
    }

    // ------------------------------------------------------

    public double MSE(double[][] dataX, double[] dataY)
    {
      int n = dataX.Length;
      double sum = 0.0;
      for (int i = 0; i "lt" n; ++i)
      {
        double actualY = dataY[i];
        double predY = this.Predict(dataX[i]);
        sum += (actualY - predY) * (actualY - predY);
      }
      return sum / n;
    }

    // ------------------------------------------------------

    private static int[] GetRandomRowIdxs(int N, int n,
      Random rnd)
    {
      // pick n rows from N with replacement
      int[] result = new int[n];
      for (int i = 0; i "lt" n; ++i)
        result[i] = rnd.Next(0, N);
      Array.Sort(result);
      return result;
    }

    private static double[][] MakeRowsSubsetX(double[][]
      trainX, int[] rndRows)
    {
      int nCols = trainX[0].Length;  // use all cols
      double[][] result = new double[rndRows.Length][];
      for (int i = 0; i "lt" rndRows.Length; ++i)
        result[i] = new double[nCols];

      for (int i = 0; i "lt" rndRows.Length; ++i)
      {
        for (int j = 0; j "lt" nCols; ++j)
        {
          int r = rndRows[i]; // random row in trainX
          result[i][j] = trainX[r][j];
        }
      }
      return result;
    }

    private static double[] MakeSubsetY(double[] trainY,
      int[] rndRows)
    {
      double[] result = new double[rndRows.Length];
      for (int i = 0; i "lt" rndRows.Length; ++i)
        result[i] = trainY[rndRows[i]];
      return result;
    }

  } // class RandomForestRegression

  // ========================================================

  // simplified list-based tree

  // ========================================================

  public class DecisionTreeRegressor
  {
    public int maxDepth;
    public int minSamples;  // aka min_samples_split
    public int minLeaf;  // min number of values in a leaf
    public int numSplitCols; // mostly for random forest
    public List"lt"Node"gt" tree = new List"lt"Node"gt"();
    public Random rnd;  // order in which cols are searched

    public double[][] trainX;  // store data by ref
    public double[] trainY;

    // ------------------------------------------------------

    public class Node
    {
      public int id;
      public int colIdx;  // aka featureIdx
      public double thresh;
      public int left;  // index into List
      public int right;
      public double value;
      public bool isLeaf;
      public List"lt"int"gt" rows;  // assoc rows in train data

      public Node()
      {
        this.id = -1;
        this.colIdx = -1;
        this.thresh = 0.0;  // aka split value
        this.left = -1;
        this.right = -1;
        this.value = 0.0;  // aka pred y
        this.isLeaf = false;
        this.rows = null;
      }
    } // class Node

    // --------------------------------------------

    public DecisionTreeRegressor(int maxDepth = 2,
      int minSamples = 2, int minLeaf = 1,
      int numSplitCols = -1, int seed = 0)
    {
      // if maxDepth = 0, tree has just a root node
      // if maxDepth = 1, at most 3 nodes (root, l, r)
      // if maxDepth = n, at most 2^(n+1) - 1 nodes
      this.maxDepth = maxDepth;
      this.minSamples = minSamples;
      this.minLeaf = minLeaf;
      this.numSplitCols = numSplitCols;  // for ran. forest

      // create full tree List with null nodes
      int numNodes = (int)Math.Pow(2, (maxDepth + 1)) - 1;
      for (int i = 0; i "lt" numNodes; ++i)
      {
        this.tree.Add(null);  // empty nodes
      }
      this.rnd = new Random(seed);
    }

    // ------------------------------------------------------
    // public: Train(), Predict(), Explain(), Accuracy(),
    //   MSE(), Display().
    // helpers: MakeTree(), BestSplit(), TreeTargetMean(),
    //   TreeTargetVariance().
    // ------------------------------------------------------

    public void Train(double[][] trainX, double[] trainY)
    {
      this.trainX = trainX; // 
      this.trainY = trainY;
      this.MakeTree();
    }

    // ------------------------------------------------------

    public double Predict(double[] x)
    {
      int p = 0;
      Node currNode = this.tree[p];
      while (currNode != null &&
        currNode.isLeaf == false &&
        p "lt" this.tree.Count)
      {
        if (x[currNode.colIdx] "lte" currNode.thresh)
          p = currNode.left;
        else
          p = currNode.right;
        currNode = this.tree[p];
      }
      return this.tree[p].value;
    }

    private void MakeTree()
    {
      // no recursion, no pointers, List storage, no stack
      if (this.numSplitCols == -1) // use all cols
        this.numSplitCols = this.trainX[0].Length;

      // prepare root node
      List"lt"int"gt" allRows = new List"lt"int"gt"();
      for (int i = 0; i "lt" this.trainX.Length; ++i)
        allRows.Add(i);
      double grandMean = this.TreeTargetMean(allRows);

      // wait to supply colIdx and thresh in loop
      Node root = new Node();
      root.id = 0;
      root.left = 1;
      root.right = 2;
      root.value = grandMean;
      root.isLeaf = false; // already set
      root.rows = allRows;
      this.tree[0] = root;

      for (int i = 0; i "lt" this.tree.Count; ++i)
      {
        Node currNode = this.tree[i];
        // curr node has values for everything
        // except colIdx and thresh

        // curr node too deep to have children OR
        // curr node not enough rows to split then
        // leave both children as null
        if (currNode == null ||
          currNode.rows.Count == 0) { continue; }

        // if parent cannot be split, make parent a leaf
        if (currNode.id "gte" (int)Math.Pow(2,
          (this.maxDepth)) - 1 ||
          currNode.rows.Count "lt" this.minSamples)
        {
          currNode.isLeaf = true;
          continue;
        }

        // parent has enough rows to try to split
        double[] splitInfo = this.BestSplit(currNode.rows);
        int colIdx = (int)splitInfo[0];
        double splitVal = splitInfo[1]; //split value

        if (colIdx == -1)  // unable split, is a leaf
        {
          currNode.isLeaf = true;
          continue;
        }

        // complete the fields for curr node
        currNode.colIdx = colIdx;
        currNode.thresh = splitVal;

        // construct the children, 
        // except for colIdx and thresh
        // which will be supplied in amin loop
        Node leftNode = new Node();
        Node rightNode = new Node();

        // construct children rows using split info
        // all info except colIdx and thresh
        List"lt"int"gt" leftIdxs = new List"lt"int"gt"();
        List"lt"int"gt" rightIdxs = new List"lt"int"gt"();
        for (int k = 0; k "lt" currNode.rows.Count; ++k)
        {
          int r = currNode.rows[k];
          if (this.trainX[r][colIdx] "lte" splitVal)
            leftIdxs.Add(r);
          else
            rightIdxs.Add(r);
        }

        leftNode.id = currNode.id * 2 + 1;
        if (leftNode.id "gt" (int)Math.Pow(2,
          (maxDepth + 1)) - 2) leftNode.id = -1;
        leftNode.left = leftNode.id * 2 + 1;
        if (leftNode.left "gt" (int)Math.Pow(2,
          (maxDepth + 1)) - 2) leftNode.left = -1;
        leftNode.right = leftNode.id * 2 + 2;
        if (leftNode.right "gt" (int)Math.Pow(2,
          (maxDepth + 1)) - 2) leftNode.right = -1;

        leftNode.rows = leftIdxs;
        leftNode.value =
          this.TreeTargetMean(leftNode.rows);
        this.tree[leftNode.id] = leftNode;

        rightNode.id = currNode.id * 2 + 2;
        if (rightNode.id "gt" (int)Math.Pow(2,
          (maxDepth + 1)) - 2) rightNode.id = -1;
        rightNode.left = rightNode.id * 2 + 1;
        if (rightNode.left "gt" (int)Math.Pow(2,
          (maxDepth + 1)) - 2) rightNode.left = -1;
        rightNode.right = rightNode.id * 2 + 2;
        if (rightNode.right "gt" (int)Math.Pow(2,
          (maxDepth + 1)) - 2) rightNode.right = -1;
        rightNode.rows = rightIdxs;
        rightNode.value =
          this.TreeTargetMean(rightNode.rows);
        this.tree[rightNode.id] = rightNode;

      } // i
      return;
    }

    // ------------------------------------------------------

    private double[] BestSplit(List"lt"int"gt" rows)
    {
      // implicit params numSplitCols, minLeaf, numsplitCols
      // result[0] = best col idx (as double)
      // result[1] = best split value
      rows.Sort();

      int bestColIdx = -1;  // indicates bad split
      double bestThresh = 0.0;
      double bestVar = double.MaxValue;  // smaller better

      int nRows = rows.Count;  // or dataY.Length
      int nCols = this.trainX[0].Length;

      if (nRows == 0)
      {
        throw new Exception("empty data in BestSplit()");
      }

      // process cols in scrambled order
      int[] colIndices = new int[nCols];
      for (int k = 0; k "lt" nCols; ++k)
        colIndices[k] = k;
      // shuffle, inline Fisher-Yates
      int n = colIndices.Length;
      for (int i = 0; i "lt" n; ++i)
      {
        int ri = rnd.Next(i, n);  // be careful
        int tmp = colIndices[i];
        colIndices[i] = colIndices[ri];
        colIndices[ri] = tmp;
      }

      // numSplitCols is usually all columns (-1)
      for (int j = 0; j "lt" this.numSplitCols; ++j)
      {
        int colIdx = colIndices[j];
        HashSet"lt"double"gt" examineds =
          new HashSet"lt"double"gt"();

        for (int i = 0; i "lt" nRows; ++i) // each row
        {
          // if curr thresh been seen, skip it
          double thresh = this.trainX[rows[i]][colIdx];
          if (examineds.Contains(thresh)) continue;
          examineds.Add(thresh);

          // get row idxs where x is lte, gt thresh
          List"lt"int"gt" leftIdxs = new List"lt"int"gt"();
          List"lt"int"gt" rightIdxs = new List"lt"int"gt"();
          for (int k = 0; k "lt" nRows; ++k)
          {
            if (this.trainX[rows[k]][colIdx] "lte" thresh)
              leftIdxs.Add(rows[k]);
            else
              rightIdxs.Add(rows[k]);
          }

          // Check if proposed split has too few values
          if (leftIdxs.Count "lt" this.minLeaf ||
            rightIdxs.Count "lt" this.minLeaf)
            continue;  // to next row

          double leftVar =
            this.TreeTargetVariance(leftIdxs);
          double rightVar =
            this.TreeTargetVariance(rightIdxs);
          double weightedVar = (leftIdxs.Count * leftVar +
            rightIdxs.Count * rightVar) / nRows;

          if (weightedVar "lt" bestVar)
          {
            // if this never happens, bestColIdx remains -1
            // which means a bad split. used in MakeTree()
            bestColIdx = colIdx;
            bestThresh = thresh;
            bestVar = weightedVar;
          }

        } // each row
      } // j each col

      double[] result = new double[2];  // out params ugly
      result[0] = 1.0 * bestColIdx;
      result[1] = bestThresh;
      return result;

    } // BestSplit()

    // ------------------------------------------------------

    private double TreeTargetMean(List"lt"int"gt" rows)
    {
      // mean of rows items in trainY: for node prediction
      double sum = 0.0;
      for (int i = 0; i "lt" rows.Count; ++i)
      {
        int r = rows[i];
        sum += this.trainY[r];
      }
      return sum / rows.Count;
    }

    // ------------------------------------------------------

    private double TreeTargetVariance(List"lt"int"gt" rows)
    {
      double mean = this.TreeTargetMean(rows);
      double sum = 0.0;
      for (int i = 0; i "lt" rows.Count; ++i)
      {
        int r = rows[i];
        sum += (this.trainY[r] - mean) *
          (this.trainY[r] - mean);
      }
      return sum / rows.Count;
    }

    // ------------------------------------------------------

  } // class DecisionTreeRegressor
} // ns

Training data:

# synthetic_train_200.txt
#
-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
-0.8299, -0.9219, -0.6603,  0.7563, -0.8033,  0.7955
 0.0663,  0.3838, -0.3690,  0.3730,  0.6693,  0.3206
-0.9634,  0.5003,  0.9777,  0.4963, -0.4391,  0.7377
-0.1042,  0.8172, -0.4128, -0.4244, -0.7399,  0.4801
-0.9613,  0.3577, -0.5767, -0.4689, -0.0169,  0.6861
-0.7065,  0.1786,  0.3995, -0.7953, -0.1719,  0.5569
 0.3888, -0.1716, -0.9001,  0.0718,  0.3276,  0.2500
 0.1731,  0.8068, -0.7251, -0.7214,  0.6148,  0.3297
-0.2046, -0.6693,  0.8550, -0.3045,  0.5016,  0.2129
 0.2473,  0.5019, -0.3022, -0.4601,  0.7918,  0.2613
-0.1438,  0.9297,  0.3269,  0.2434, -0.7705,  0.5171
 0.1568, -0.1837, -0.5259,  0.8068,  0.1474,  0.3307
-0.9943,  0.2343, -0.3467,  0.0541,  0.7719,  0.5581
 0.2467, -0.9684,  0.8589,  0.3818,  0.9946,  0.1092
-0.6553, -0.7257,  0.8652,  0.3936, -0.8680,  0.7018
 0.8460,  0.4230, -0.7515, -0.9602, -0.9476,  0.1996
-0.9434, -0.5076,  0.7201,  0.0777,  0.1056,  0.5664
 0.9392,  0.1221, -0.9627,  0.6013, -0.5341,  0.1533
 0.6142, -0.2243,  0.7271,  0.4942,  0.1125,  0.1661
 0.4260,  0.1194, -0.9749, -0.8561,  0.9346,  0.2230
 0.1362, -0.5934, -0.4953,  0.4877, -0.6091,  0.3810
 0.6937, -0.5203, -0.0125,  0.2399,  0.6580,  0.1460
-0.6864, -0.9628, -0.8600, -0.0273,  0.2127,  0.5387
 0.9772,  0.1595, -0.2397,  0.1019,  0.4907,  0.1611
 0.3385, -0.4702, -0.8673, -0.2598,  0.2594,  0.2270
-0.8669, -0.4794,  0.6095, -0.6131,  0.2789,  0.4700
 0.0493,  0.8496, -0.4734, -0.8681,  0.4701,  0.3516
 0.8639, -0.9721, -0.5313,  0.2336,  0.8980,  0.1412
 0.9004,  0.1133,  0.8312,  0.2831, -0.2200,  0.1782
 0.0991,  0.8524,  0.8375, -0.2102,  0.9265,  0.2150
-0.6521, -0.7473, -0.7298,  0.0113, -0.9570,  0.7422
 0.6190, -0.3105,  0.8802,  0.1640,  0.7577,  0.1056
 0.6895,  0.8108, -0.0802,  0.0927,  0.5972,  0.2214
 0.1982, -0.9689,  0.1870, -0.1326,  0.6147,  0.1310
-0.3695,  0.7858,  0.1557, -0.6320,  0.5759,  0.3773
-0.1596,  0.3581,  0.8372, -0.9992,  0.9535,  0.2071
-0.2468,  0.9476,  0.2094,  0.6577,  0.1494,  0.4132
 0.1737,  0.5000,  0.7166,  0.5102,  0.3961,  0.2611
 0.7290, -0.3546,  0.3416, -0.0983, -0.2358,  0.1332
-0.3652,  0.2438, -0.1395,  0.9476,  0.3556,  0.4170
-0.6029, -0.1466, -0.3133,  0.5953,  0.7600,  0.4334
-0.4596, -0.4953,  0.7098,  0.0554,  0.6043,  0.2775
 0.1450,  0.4663,  0.0380,  0.5418,  0.1377,  0.2931
-0.8636, -0.2442, -0.8407,  0.9656, -0.6368,  0.7429
 0.6237,  0.7499,  0.3768,  0.1390, -0.6781,  0.2185
-0.5499,  0.1850, -0.3755,  0.8326,  0.8193,  0.4399
-0.4858, -0.7782, -0.6141, -0.0008,  0.4572,  0.4197
 0.7033, -0.1683,  0.2334, -0.5327, -0.7961,  0.1776
 0.0317, -0.0457, -0.6947,  0.2436,  0.0880,  0.3345
 0.5031, -0.5559,  0.0387,  0.5706, -0.9553,  0.3107
-0.3513,  0.7458,  0.6894,  0.0769,  0.7332,  0.3170
 0.2205,  0.5992, -0.9309,  0.5405,  0.4635,  0.3532
-0.4806, -0.4859,  0.2646, -0.3094,  0.5932,  0.3202
 0.9809, -0.3995, -0.7140,  0.8026,  0.0831,  0.1600
 0.9495,  0.2732,  0.9878,  0.0921,  0.0529,  0.1289
-0.9476, -0.6792,  0.4913, -0.9392, -0.2669,  0.5966
 0.7247,  0.3854,  0.3819, -0.6227, -0.1162,  0.1550
-0.5922, -0.5045, -0.4757,  0.5003, -0.0860,  0.5863
-0.8861,  0.0170, -0.5761,  0.5972, -0.4053,  0.7301
 0.6877, -0.2380,  0.4997,  0.0223,  0.0819,  0.1404
 0.9189,  0.6079, -0.9354,  0.4188, -0.0700,  0.1907
-0.1428, -0.7820,  0.2676,  0.6059,  0.3936,  0.2790
 0.5324, -0.3151,  0.6917, -0.1425,  0.6480,  0.1071
-0.8432, -0.9633, -0.8666, -0.0828, -0.7733,  0.7784
-0.9444,  0.5097, -0.2103,  0.4939, -0.0952,  0.6787
-0.0520,  0.6063, -0.1952,  0.8094, -0.9259,  0.4836
 0.5477, -0.7487,  0.2370, -0.9793,  0.0773,  0.1241
 0.2450,  0.8116,  0.9799,  0.4222,  0.4636,  0.2355
 0.8186, -0.1983, -0.5003, -0.6531, -0.7611,  0.1511
-0.4714,  0.6382, -0.3788,  0.9648, -0.4667,  0.5950
 0.0673, -0.3711,  0.8215, -0.2669, -0.1328,  0.2677
-0.9381,  0.4338,  0.7820, -0.9454,  0.0441,  0.5518
-0.3480,  0.7190,  0.1170,  0.3805, -0.0943,  0.4724
-0.9813,  0.1535, -0.3771,  0.0345,  0.8328,  0.5438
-0.1471, -0.5052, -0.2574,  0.8637,  0.8737,  0.3042
-0.5454, -0.3712, -0.6505,  0.2142, -0.1728,  0.5783
 0.6327, -0.6297,  0.4038, -0.5193,  0.1484,  0.1153
-0.5424,  0.3282, -0.0055,  0.0380, -0.6506,  0.6613
 0.1414,  0.9935,  0.6337,  0.1887,  0.9520,  0.2540
-0.9351, -0.8128, -0.8693, -0.0965, -0.2491,  0.7353
 0.9507, -0.6640,  0.9456,  0.5349,  0.6485,  0.1059
-0.0462, -0.9737, -0.2940, -0.0159,  0.4602,  0.2606
-0.0627, -0.0852, -0.7247, -0.9782,  0.5166,  0.2977
 0.0478,  0.5098, -0.0723, -0.7504, -0.3750,  0.3335
 0.0090,  0.3477,  0.5403, -0.7393, -0.9542,  0.4415
-0.9748,  0.3449,  0.3736, -0.1015,  0.8296,  0.4358
 0.2887, -0.9895, -0.0311,  0.7186,  0.6608,  0.2057
 0.1570, -0.4518,  0.1211,  0.3435, -0.2951,  0.3244
 0.7117, -0.6099,  0.4946, -0.4208,  0.5476,  0.1096
-0.2929, -0.5726,  0.5346, -0.3827,  0.4665,  0.2465
 0.4889, -0.5572, -0.5718, -0.6021, -0.7150,  0.2163
-0.7782,  0.3491,  0.5996, -0.8389, -0.5366,  0.6516
-0.5847,  0.8347,  0.4226,  0.1078, -0.3910,  0.6134
 0.8469,  0.4121, -0.0439, -0.7476,  0.9521,  0.1571
-0.6803, -0.5948, -0.1376, -0.1916, -0.7065,  0.7156
 0.2878,  0.5086, -0.5785,  0.2019,  0.4979,  0.2980
 0.2764,  0.1943, -0.4090,  0.4632,  0.8906,  0.2960
-0.8877,  0.6705, -0.6155, -0.2098, -0.3998,  0.7107
-0.8398,  0.8093, -0.2597,  0.0614, -0.0118,  0.6502
-0.8476,  0.0158, -0.4769, -0.2859, -0.7839,  0.7715
 0.5751, -0.7868,  0.9714, -0.6457,  0.1448,  0.1175
 0.4802, -0.7001,  0.1022, -0.5668,  0.5184,  0.1090
 0.4458, -0.6469,  0.7239, -0.9604,  0.7205,  0.0779
 0.5175,  0.4339,  0.9747, -0.4438, -0.9924,  0.2879
 0.8678,  0.7158,  0.4577,  0.0334,  0.4139,  0.1678
 0.5406,  0.5012,  0.2264, -0.1963,  0.3946,  0.2088
-0.9938,  0.5498,  0.7928, -0.5214, -0.7585,  0.7687
 0.7661,  0.0863, -0.4266, -0.7233, -0.4197,  0.1466
 0.2277, -0.3517, -0.0853, -0.1118,  0.6563,  0.1767
 0.3499, -0.5570, -0.0655, -0.3705,  0.2537,  0.1632
 0.7547, -0.1046,  0.5689, -0.0861,  0.3125,  0.1257
 0.8186,  0.2110,  0.5335,  0.0094, -0.0039,  0.1391
 0.6858, -0.8644,  0.1465,  0.8855,  0.0357,  0.1845
-0.4967,  0.4015,  0.0805,  0.8977,  0.2487,  0.4663
 0.6760, -0.9841,  0.9787, -0.8446, -0.3557,  0.1509
-0.1203, -0.4885,  0.6054, -0.0443, -0.7313,  0.4854
 0.8557,  0.7919, -0.0169,  0.7134, -0.1628,  0.2002
 0.0115, -0.6209,  0.9300, -0.4116, -0.7931,  0.4052
-0.7114, -0.9718,  0.4319,  0.1290,  0.5892,  0.3661
 0.3915,  0.5557, -0.1870,  0.2955, -0.6404,  0.2954
-0.3564, -0.6548, -0.1827, -0.5172, -0.1862,  0.4622
 0.2392, -0.4959,  0.5857, -0.1341, -0.2850,  0.2470
-0.3394,  0.3947, -0.4627,  0.6166, -0.4094,  0.5325
 0.7107,  0.7768, -0.6312,  0.1707,  0.7964,  0.2757
-0.1078,  0.8437, -0.4420,  0.2177,  0.3649,  0.4028
-0.3139,  0.5595, -0.6505, -0.3161, -0.7108,  0.5546
 0.4335,  0.3986,  0.3770, -0.4932,  0.3847,  0.1810
-0.2562, -0.2894, -0.8847,  0.2633,  0.4146,  0.4036
 0.2272,  0.2966, -0.6601, -0.7011,  0.0284,  0.2778
-0.0743, -0.1421, -0.0054, -0.6770, -0.3151,  0.3597
-0.4762,  0.6891,  0.6007, -0.1467,  0.2140,  0.4266
-0.4061,  0.7193,  0.3432,  0.2669, -0.7505,  0.6147
-0.0588,  0.9731,  0.8966,  0.2902, -0.6966,  0.4955
-0.0627, -0.1439,  0.1985,  0.6999,  0.5022,  0.3077
 0.1587,  0.8494, -0.8705,  0.9827, -0.8940,  0.4263
-0.7850,  0.2473, -0.9040, -0.4308, -0.8779,  0.7199
 0.4070,  0.3369, -0.2428, -0.6236,  0.4940,  0.2215
-0.0242,  0.0513, -0.9430,  0.2885, -0.2987,  0.3947
-0.5416, -0.1322, -0.2351, -0.0604,  0.9590,  0.3683
 0.1055,  0.7783, -0.2901, -0.5090,  0.8220,  0.2984
-0.9129,  0.9015,  0.1128, -0.2473,  0.9901,  0.4776
-0.9378,  0.1424, -0.6391,  0.2619,  0.9618,  0.5368
 0.7498, -0.0963,  0.4169,  0.5549, -0.0103,  0.1614
-0.2612, -0.7156,  0.4538, -0.0460, -0.1022,  0.3717
 0.7720,  0.0552, -0.1818, -0.4622, -0.8560,  0.1685
-0.4177,  0.0070,  0.9319, -0.7812,  0.3461,  0.3052
-0.0001,  0.5542, -0.7128, -0.8336, -0.2016,  0.3803
 0.5356, -0.4194, -0.5662, -0.9666, -0.2027,  0.1776
-0.2378,  0.3187, -0.8582, -0.6948, -0.9668,  0.5474
-0.1947, -0.3579,  0.1158,  0.9869,  0.6690,  0.2992
 0.3992,  0.8365, -0.9205, -0.8593, -0.0520,  0.3154
-0.0209,  0.0793,  0.7905, -0.1067,  0.7541,  0.1864
-0.4928, -0.4524, -0.3433,  0.0951, -0.5597,  0.6261
-0.8118,  0.7404, -0.5263, -0.2280,  0.1431,  0.6349
 0.0516, -0.8480,  0.7483,  0.9023,  0.6250,  0.1959
-0.3212,  0.1093,  0.9488, -0.3766,  0.3376,  0.2735
-0.3481,  0.5490, -0.3484,  0.7797,  0.5034,  0.4379
-0.5785, -0.9170, -0.3563, -0.9258,  0.3877,  0.4121
 0.3407, -0.1391,  0.5356,  0.0720, -0.9203,  0.3458
-0.3287, -0.8954,  0.2102,  0.0241,  0.2349,  0.3247
-0.1353,  0.6954, -0.0919, -0.9692,  0.7461,  0.3338
 0.9036, -0.8982, -0.5299, -0.8733, -0.1567,  0.1187
 0.7277, -0.8368, -0.0538, -0.7489,  0.5458,  0.0830
 0.9049,  0.8878,  0.2279,  0.9470, -0.3103,  0.2194
 0.7957, -0.1308, -0.5284,  0.8817,  0.3684,  0.2172
 0.4647, -0.4931,  0.2010,  0.6292, -0.8918,  0.3371
-0.7390,  0.6849,  0.2367,  0.0626, -0.5034,  0.7039
-0.1567, -0.8711,  0.7940, -0.5932,  0.6525,  0.1710
 0.7635, -0.0265,  0.1969,  0.0545,  0.2496,  0.1445
 0.7675,  0.1354, -0.7698, -0.5460,  0.1920,  0.1728
-0.5211, -0.7372, -0.6763,  0.6897,  0.2044,  0.5217
 0.1913,  0.1980,  0.2314, -0.8816,  0.5006,  0.1998
 0.8964,  0.0694, -0.6149,  0.5059, -0.9854,  0.1825
 0.1767,  0.7104,  0.2093,  0.6452,  0.7590,  0.2832
-0.3580, -0.7541,  0.4426, -0.1193, -0.7465,  0.5657
-0.5996,  0.5766, -0.9758, -0.3933, -0.9572,  0.6800
 0.9950,  0.1641, -0.4132,  0.8579,  0.0142,  0.2003
-0.4717, -0.3894, -0.2567, -0.5111,  0.1691,  0.4266
 0.3917, -0.8561,  0.9422,  0.5061,  0.6123,  0.1212
-0.0366, -0.1087,  0.3449, -0.1025,  0.4086,  0.2475
 0.3633,  0.3943,  0.2372, -0.6980,  0.5216,  0.1925
-0.5325, -0.6466, -0.2178, -0.3589,  0.6310,  0.3568
 0.2271,  0.5200, -0.1447, -0.8011, -0.7699,  0.3128
 0.6415,  0.1993,  0.3777, -0.0178, -0.8237,  0.2181
-0.5298, -0.0768, -0.6028, -0.9490,  0.4588,  0.4356
 0.6870, -0.1431,  0.7294,  0.3141,  0.1621,  0.1632
-0.5985,  0.0591,  0.7889, -0.3900,  0.7419,  0.2945
 0.3661,  0.7984, -0.8486,  0.7572, -0.6183,  0.3449
 0.6995,  0.3342, -0.3113, -0.6972,  0.2707,  0.1712
 0.2565,  0.9126,  0.1798, -0.6043, -0.1413,  0.2893
-0.3265,  0.9839, -0.2395,  0.9854,  0.0376,  0.4770
 0.2690, -0.1722,  0.9818,  0.8599, -0.7015,  0.3954
-0.2102, -0.0768,  0.1219,  0.5607, -0.0256,  0.3949
 0.8216, -0.9555,  0.6422, -0.6231,  0.3715,  0.0801
-0.2896,  0.9484, -0.7545, -0.6249,  0.7789,  0.4370
-0.9985, -0.5448, -0.7092, -0.5931,  0.7926,  0.5402

Test data:

# synthetic_test_40.txt
#
 0.7462,  0.4006, -0.0590,  0.6543, -0.0083,  0.1935
 0.8495, -0.2260, -0.0142, -0.4911,  0.7699,  0.1078
-0.2335, -0.4049,  0.4352, -0.6183, -0.7636,  0.5088
 0.1810, -0.5142,  0.2465,  0.2767, -0.3449,  0.3136
-0.8650,  0.7611, -0.0801,  0.5277, -0.4922,  0.7140
-0.2358, -0.7466, -0.5115, -0.8413, -0.3943,  0.4533
 0.4834,  0.2300,  0.3448, -0.9832,  0.3568,  0.1360
-0.6502, -0.6300,  0.6885,  0.9652,  0.8275,  0.3046
-0.3053,  0.5604,  0.0929,  0.6329, -0.0325,  0.4756
-0.7995,  0.0740, -0.2680,  0.2086,  0.9176,  0.4565
-0.2144, -0.2141,  0.5813,  0.2902, -0.2122,  0.4119
-0.7278, -0.0987, -0.3312, -0.5641,  0.8515,  0.4438
 0.3793,  0.1976,  0.4933,  0.0839,  0.4011,  0.1905
-0.8568,  0.9573, -0.5272,  0.3212, -0.8207,  0.7415
-0.5785,  0.0056, -0.7901, -0.2223,  0.0760,  0.5551
 0.0735, -0.2188,  0.3925,  0.3570,  0.3746,  0.2191
 0.1230, -0.2838,  0.2262,  0.8715,  0.1938,  0.2878
 0.4792, -0.9248,  0.5295,  0.0366, -0.9894,  0.3149
-0.4456,  0.0697,  0.5359, -0.8938,  0.0981,  0.3879
 0.8629, -0.8505, -0.4464,  0.8385,  0.5300,  0.1769
 0.1995,  0.6659,  0.7921,  0.9454,  0.9970,  0.2330
-0.0249, -0.3066, -0.2927, -0.4923,  0.8220,  0.2437
 0.4513, -0.9481, -0.0770, -0.4374, -0.9421,  0.2879
-0.3405,  0.5931, -0.3507, -0.3842,  0.8562,  0.3987
 0.9538,  0.0471,  0.9039,  0.7760,  0.0361,  0.1706
-0.0887,  0.2104,  0.9808,  0.5478, -0.3314,  0.4128
-0.8220, -0.6302,  0.0537, -0.1658,  0.6013,  0.4306
-0.4123, -0.2880,  0.9074, -0.0461, -0.4435,  0.5144
 0.0060,  0.2867, -0.7775,  0.5161,  0.7039,  0.3599
-0.7968, -0.5484,  0.9426, -0.4308,  0.8148,  0.2979
 0.7811,  0.8450, -0.6877,  0.7594,  0.2640,  0.2362
-0.6802, -0.1113, -0.8325, -0.6694, -0.6056,  0.6544
 0.3821,  0.1476,  0.7466, -0.5107,  0.2592,  0.1648
 0.7265,  0.9683, -0.9803, -0.4943, -0.5523,  0.2454
-0.9049, -0.9797, -0.0196, -0.9090, -0.4433,  0.6447
-0.4607,  0.1811, -0.2389,  0.4050, -0.0078,  0.5229
 0.2664, -0.2932, -0.4259, -0.7336,  0.8742,  0.1834
-0.4507,  0.1029, -0.6294, -0.1158, -0.6294,  0.6081
 0.8948, -0.0124,  0.9278,  0.2899, -0.0314,  0.1534
-0.1323, -0.8813, -0.0146, -0.0697,  0.6135,  0.2386

Bottom line: The title of this post is unintentional click-bait. The default target-to-predict diabetes score in column [10] of the scikit Diabetes Dataset cannot be predicted meaningfully. But the variables in columns [4], [5], [6], [7], and [8] can be meaningfully predicted from the other columns.

I have worked with machine learning for decades. In 2010, the scikit-learn library was published and it contained several datasets for use in examples. One of these popular datasets is the Diabetes Dataset for regression models.

I was aware of the Diabetes Dataset but never looked at it until recently. I was more than just a little bit surprised to discover that the dataset is essentially useless — the target y values just cannot be predicted with any meaningful accuracy.

But, the dataset is useful if a different variable is specified as the target y to predict

The Diabetes Dataset looks like:

59, 2, 32.1, 101.00, 157,  93.2, 38, 4.00, 4.8598, 87, 151
48, 1, 21.6,  87.00, 183, 103.2, 70, 3.00, 3.8918, 69,  75
72, 2, 30.5,  93.00, 156,  93.6, 41, 4.00, 4.6728, 85, 141
. . . 

The dataset has 442 items. Each item represents a patient and has 10 predictor values followed by a target value to predict. The 10 predictor variables are age in column [0], sex [1], body mass index [2], blood pressure [3], serum cholesterol [4], low-density lipoproteins [5], high-density lipoproteins [6], total cholesterol [7], triglycerides [8], blood sugar [9]. The stated (in the documentation) target value to predict in the last column is a measure of diabetes [10].

Note: The sex encoding isn’t explained but I suspect male = 1, female = 2 because there are 235 1 values and 206 2 values).

I ran the Diabetes Dataset through six models: 1.) linear regression, 2.) k-nearest neighbors regression, 3.) kernel ridge regression, 4.) neural network (MLP) regression, 5.) random forest regression, and 6.) gradient boosting regression. None of the models gave any meaningful prediction results for the diabetes score. The output of my experiment:

Begin Diabetes Dataset exploration

Loading diabetes train (342), test (100) data
Done

First three X predictors:
[ 59.0000   2.0000  32.1000 . . . 87.0000]
[ 48.0000   1.0000  21.6000 . . . 69.0000]
[ 72.0000   2.0000  30.5000 . . . 85.0000]

First three y targets:
151.0000
 75.0000
141.0000

==========
Linear Regression

acc train (0.10) = 0.1813
acc test (0.10) = 0.2800
==========

==========
Nearest Neighbors k=4

acc train (0.10) = 0.1988
acc test (0.10) = 0.1400
==========

==========
Kernel Ridge Regression RBF gamma=0.005, alpha=0.01

acc train (0.10) = 0.9971
acc test (0.10) = 0.0700
==========

==========
MLPRegressor max_iter=2000

acc train (0.10) = 0.1520
acc test (0.10) = 0.2200
==========

==========
Random Forest Regression depth=5

acc train (0.10) = 0.2368
acc test (0.10) = 0.1700
==========

==========
Gradient Boost Regression depth=5, lrn_rate=0.10

acc train (0.10) = 0.7807
acc test (0.10) = 0.1800
==========

The models either severely underfit (linear regression, nearest neighbors, neural network, random forest), or severely overfit (kernel ridge regression, gradient boosting regression). For the accuracy values, a prediction is scored as correct if it is within 10% of the true target value.

Note: I re-encoded the sex predictor from 1,2 to 0,1. I normalized the other predictor values using divide-by-constant normalization (100, 1, 100, 1000, 1000, 1000, 100, 10, 10, 1000, 1000). I normalized the target y values by dividing by 1000. I used the first 342 data items for training and the remaining 100 items for testing. I experimented with other splitting ideas and normalization techniques, but none had any significant effect.

So, the scikit Diabetes Dataset is an unintentional hoax. The default diabetes score in the dataset cannot be meaningfully predicted.

However, after a bit of experimentation, I discovered that columns [4], [5], [6], [7], and [8] can be predicted meaningfully. If you use the built-in load_diabetes() function with a return_X_y=True parameter, column [10] is automatically the target to predict. But you can load the data as a DataFrame and then specify a different column as the target. Alternatively, you can use preprocessed training data.

Here’s partial output when column [4] (serum cholesterol) is set as the target-to-predict:

==========
Linear Regression

acc train (0.10) = 0.9942
acc test (0.10) = 0.9800
==========

==========
Nearest Neighbors k=4

acc train (0.10) = 0.8860
acc test (0.10) = 0.7900
==========

==========
Random Forest Regression depth=5

acc train (0.10) = 0.9766
acc test (0.10) = 0.9400
==========

So, the title of this post is somewhat click-bait. The scikit Diabetes Dataset is useful for experimenting with regression models, but only when the default diabetes score in column [10] is not used as the target y.

The inclusion of the Diabetes Dataset in the scikit library collection of datasets is an unintentional hoax. But financial fraud is a different story. Stereotypes exist for a reason.

Demo program. Replace the “lt” in the accuracy() function with the less-than operator.

# diabetes_scikit.py
# various techniques for the Diabetes Dataset

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn import neighbors
from sklearn.kernel_ridge import KernelRidge
from sklearn.neural_network import MLPRegressor
from sklearn import ensemble

np.set_printoptions(precision=4, suppress=True,
  floatmode='fixed', linewidth=120)

# -----------------------------------------------------------

def accuracy(model, data_X, data_y, pct_close):
  n = len(data_X)
  n_correct = 0; n_wrong = 0
  for i in range(n):
    x = data_X[i].reshape(1,-1)
    y = data_y[i]
    y_pred = model.predict(x)

    # replace the "lt" with less-than symbol
    if np.abs(y - y_pred) "lt" np.abs(y * pct_close):
      n_correct += 1
    else: 
      n_wrong += 1
  # print("Correct = " + str(n_correct))
  # print("Wrong   = " + str(n_wrong))
  return n_correct / (n_correct + n_wrong)

# -----------------------------------------------------------

def MSE(model, data_X, data_y):
  n = len(data_X)
  sum = 0.0
  for i in range(n):
    x = data_X[i].reshape(1,-1)
    y = data_y[i]
    y_pred = model.predict(x)[0]
    # print(y_pred); input()
    sum += (y - y_pred) * (y - y_pred)

  return sum / n

# -----------------------------------------------------------

print("\nBegin Diabetes Dataset exploration ")

# use pre-normalized data
print("\nLoading diabetes train (342), test (100) data ")

cols_X = [0,1,2,3,4,5,6,7,8,9]  # predictors
col_y = 10  # target. cols 4 5 6 7 8 are much better
train_X = np.loadtxt(train_file, comments="#",
  usecols=cols_X,
  delimiter=",",  dtype=np.float64)
train_y = np.loadtxt(train_file, comments="#", usecols=col_y,
  delimiter=",",  dtype=np.float64)

test_file = ".\\Data\\diabetes_test_100.txt"
test_X = np.loadtxt(test_file, comments="#",
  usecols=cols_X,
  delimiter=",",  dtype=np.float64)
test_y = np.loadtxt(test_file, comments="#", usecols=col_y,
  delimiter=",",  dtype=np.float64)
print("Done ")

# use built-in diabetes data for
#   alternative normalization and split.
# from sklearn.datasets import load_diabetes
# from sklearn.model_selection import train_test_split
# X, y = load_diabetes(return_X_y=True, scaled=True)
# train_X, test_X, train_y, test_y = \
#   train_test_split(X, y, random_state=0)  # 25% test

print("\nFirst three X predictors: ")
for i in range(3):
  print(train_X[i])
print("\nFirst three y targets: ")
for i in range(3):
  print("%0.4f" % train_y[i])

print("\n========== ")
print("Linear Regression ")
model = LinearRegression()
model.fit(train_X, train_y)
acc_train = accuracy(model, train_X, train_y, 0.10)
print("\nacc train (0.10) = %0.4f " % acc_train)
acc_test = accuracy(model, test_X, test_y, 0.10)
print("acc test (0.10) = %0.4f " % acc_test)
print("========== ")

print("\n========== ")
print("Nearest Neighbors k=4")
model = neighbors.KNeighborsRegressor(4)
model.fit(train_X, train_y)
acc_train = accuracy(model, train_X, train_y, 0.10)
print("\nacc train (0.10) = %0.4f " % acc_train)
acc_test = accuracy(model, test_X, test_y, 0.10)
print("acc test (0.10) = %0.4f " % acc_test)
print("========== ")

print("\n========== ")
print("Kernel Ridge Regression RBF gamma=0.005, alpha=0.01")
model = KernelRidge(kernel='rbf', gamma=0.005, alpha=0.01)
model.fit(train_X, train_y)
acc_train = accuracy(model, train_X, train_y, 0.10)
print("\nacc train (0.10) = %0.4f " % acc_train)
acc_test = accuracy(model, test_X, test_y, 0.10)
print("acc test (0.10) = %0.4f " % acc_test)
print("========== ")

print("\n========== ")
print("MLPRegressor max_iter=2000")
model = MLPRegressor(random_state=0, max_iter=2000)
model.fit(train_X, train_y)
acc_train = accuracy(model, train_X, train_y, 0.10)
print("\nacc train (0.10) = %0.4f " % acc_train)
acc_test = accuracy(model, test_X, test_y, 0.10)
print("acc test (0.10) = %0.4f " % acc_test)
print("========== ")

print("\n========== ")
print("Random Forest Regression depth=5")
params = { "n_estimators": 100, "max_depth": 5 }
model = ensemble.RandomForestRegressor(**params)
model.fit(train_X, train_y)
acc_train = accuracy(model, train_X, train_y, 0.10)
print("\nacc train (0.10) = %0.4f " % acc_train)
acc_test = accuracy(model, test_X, test_y, 0.10)
print("acc test (0.10) = %0.4f " % acc_test)
print("========== ")

print("\n========== ")
print("Gradient Boost Regression depth=5, lrn_rate=0.10")
params = {
"n_estimators": 100, "max_depth": 5, "learning_rate": 0.10,
}
model = ensemble.GradientBoostingRegressor(**params)
model.fit(train_X, train_y)
acc_train = accuracy(model, train_X, train_y, 0.10)
print("\nacc train (0.10) = %0.4f " % acc_train)
acc_test = accuracy(model, test_X, test_y, 0.10)
print("acc test (0.10) = %0.4f " % acc_test)
print("========== ")

Decision tree regression is a machine learning technique that incorporates a set of if-then rules in a tree data structure to predict a single numeric value. For example, a decision tree regression model prediction might be, “If employee age is greater than 28.0 and age is less than or equal to 33.5 and years-experience is less than or equal to 7.0 and height is greater than 66.0 then bank account balance is $754.14.”

I have implemented decision tree regression from scratch many times. Decision trees are conceptually simple, but there are a huge number of design alternatives. My latest version is implemented from scratch with C#, using list storage for tree nodes (no pointers/references), a sequential iterative algorithm to build the tree (no recursion, no stack), weighted variance minimization for the node split function, and saves associated row indices in each node for high interpretability. And that just begins to touch on all the design details. Whew!

The demo data looks like:

 0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
. . .

The first five values on each line are the x predictors. The last value on each line is the target y variable to predict. The data is synthetic and was generated by a 5-10-1 neural network with random weights and bias. Therefore, the data has an underlying structure which can be predicted. There are 200 training items and 40 test items.

The output of the demo program is:

Begin decision tree regression (simple version)

Loading synthetic train (200) and test (40) data
Done

First three train X:
 -0.1660  0.4406 -0.9998 -0.3953 -0.7065
  0.0776 -0.1616  0.3704 -0.5911  0.7562
 -0.9452  0.3409 -0.1654  0.1174 -0.7192

First three train y:
  0.4840
  0.1568
  0.8054

Setting maxDepth = 3
Setting minSamples = 2
Setting minLeaf = 18
Using default numSplitCols = -1

Creating and training tree
Done

Tree:
ID 0   | idx   0 | v  -0.2102 | L   1 | R   2 | y   0.3493 | leaf F
ID 1   | idx   4 | v   0.1431 | L   3 | R   4 | y   0.5345 | leaf F
ID 2   | idx   0 | v   0.3915 | L   5 | R   6 | y   0.2382 | leaf F
ID 3   | idx   0 | v  -0.6553 | L   7 | R   8 | y   0.6358 | leaf F
ID 4   | idx  -1 | v   0.0000 | L   9 | R  10 | y   0.4123 | leaf T
ID 5   | idx   4 | v  -0.2987 | L  11 | R  12 | y   0.3032 | leaf F
ID 6   | idx   2 | v   0.3777 | L  13 | R  14 | y   0.1701 | leaf F
ID 7   | idx  -1 | v   0.0000 | L  -1 | R  -1 | y   0.6952 | leaf T
ID 8   | idx  -1 | v   0.0000 | L  -1 | R  -1 | y   0.5598 | leaf T
ID 11  | idx  -1 | v   0.0000 | L  -1 | R  -1 | y   0.4101 | leaf T
ID 12  | idx  -1 | v   0.0000 | L  -1 | R  -1 | y   0.2613 | leaf T
ID 13  | idx  -1 | v   0.0000 | L  -1 | R  -1 | y   0.1882 | leaf T
ID 14  | idx  -1 | v   0.0000 | L  -1 | R  -1 | y   0.1381 | leaf T

Rows associated with node [14]:
1 5 10 11 12 13 15 17 28 30 33 37 39 41 46 52 55 65 71 74 78 82

Evaluating model
Accuracy train (within 0.10) = 0.3750
Accuracy test (within 0.10) = 0.4750

MSE train = 0.0048
MSE test = 0.0054

Predicting for trainX[0] =
  -0.1660   0.4406  -0.9998  -0.3953  -0.7065
Predicted y = 0.4101

IF
column 0  gt   -0.2102 AND
column 0  lte   0.3915 AND
column 4  lte  -0.2987 AND
THEN
predicted = 0.4101

End demo

There is a lot going on in the output. If you compare this output with the visual representation, most of the ideas of decision tree regression will become mostly clear:

The demo decision tree has maxDepth = 3. This creates a tree with 15 nodes ([0] through [14], some of which could be empty). In general, if a tree has maxDepth = n, then a full tree has 2^(n+1) – 1 nodes. These tree nodes could be stored as C# program-defined Node objects and connected via pointers/references. Instead, the architecture used by the demo decision tree is to create a List collection with size/count = 15, where each item in the List is a Node object.

With this design, if the root node is at index [0], then the left child is at [1] and the right child is at [2]. In general, if a node is at index [i], then the left child is at index [2*i+1] and the right child is at index [2*i+2].

The tree splitting part of constructing a decision tree regression system is perhaps best understood by looking at a concrete example. The root node of the demo decision tree is associated with all 200 rows of the training data. The splitting process selects some of the 200 rows to be assigned to the left child node, and the remaining rows to be assigned to the right child node.

For simplicity, suppose that instead of the demo training data with 200 rows and 5 columns of predictors, a tree node is associated with just 5 rows of data, and the data has just 3 columns of predictors:

     X                    y
[0] 0.99  0.22  0.77     3.0
[1] 0.55  0.44  0.00     9.0
[2] 0.88  0.66  0.88     7.0
[3] 0.11  0.33  0.22     1.0
[4] 0.55  0.88  0.33     5.0
     [0]   [1]   [2]

The split algorithm will select some of the five rows to go to the left child and the remaining rows to go to the right child. The idea is to select the two sets of rows in a way so that their associated y values are close together. The demo implementation does this by finding the split that minimizes the weighted variance of the target y values. (There are several other split algorithms).

The splitting algorithm examines every possible x value in every column. For each possible candidate split value (also called the threshold), the weighted variance of the y values of the resulting split is computed. For example, for x = 0.33 in column [1], the rows where the values in column [1] are less than or equal to 0.33 are rows [3] and [0] with associated y values (1.0, 3.0). The other rows are [1], [2], [4] with associated y values (9.0, 7.0, 5.0).

The variance of a set of y values is the average of the sum of the squared differences between the values and their mean (average).

For this proposed split, the variance of y values (1.0, 3.0) is computed as 1.00. The mean of (1.0, 3.0) = (1.0 + 3.0) / 2 = 2.0 and so the variance is ((3.0 – 2.0)^2 + (1.0 – 2.0)^2) / 2 = 1.00.

The mean of y values (9.0, 7.0, 5.0) = (9.0 + 7.0 + 5.0) / 3 = 7.0 and the variance is ((9.0 – 7.0)^2 + (7.0 -7.0)^2 + (5.0 – 7.0)^2) / 3 = 2.67.

Because the proposed split has different numbers of rows (two rows for the left child and three rows for the right child), the weighted variance of the proposed split is used and is ((2 * 1.00) + (3 * 2.67)) / 5 = 2.00.

This process is repeated for every x value in every column. The x value and its column that generate the smallest weighted variance define the split.

The demo does not use recursion to build the tree, which is a common aproach in university computer science data structures classes, but not good for a production environment. Another common approach is to use a stack and push and pop nodes as they are created. But the demo in this blog uses sequential iteration, constructing the tree nodes in order. The details are extrememly tricky and can only be completely understood by carefully dissecting the demo code. Implementation took me many hours of coding and debugging.

Decision trees are rarely used by themselves because they usually overfit the training data, and predict poorly on new, previously unseen data. Multiple decision trees are usually combined into an ensemble technique such as bagging trees (“bootstrap aggregation”), random forest regression, adaptive boosting regression, and gradient boosting regression.

Each node in a decision tree has a left child and a right child. Tree children are neither males nor females. I’m a big fan of early science fiction movies. A surprising number of these movies feature the daughter of a scientist.

Right: In “The Man from Planet X” (1951), an alien spaceship lands in the Scottish moors. It is a scout for a full scale invasion. Professor Elliot is aided by his daughter Enid to defeat the aliens’ plans. My grade = B.

Demo code. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols (my blog editor chokes on symbols).

using System;
using System.IO;
using System.Collections.Generic;

// full List storage with indexes (no pointers)
// iteration-based construction (no stack, no recursion)
// Nodes hold associated row idxs (interpretability)

namespace DecisionTreeRegressionSimple
{
  internal class DecisionTreeRegressionProgram
  {
    static void Main(string[] args)
    {
      Console.WriteLine("\nBegin decision tree" +
        " regression (simple version) ");

      // 1. load data
      Console.WriteLine("\nLoading synthetic train" +
        " (200) and test (40) data");
      string trainFile = "..\\..\\..\\Data\\" +
        "synthetic_train_200.txt";
      int[] colsX = new int[] { 0, 1, 2, 3, 4 };
      int colY = 5;
      double[][] trainX =
        MatLoad(trainFile, colsX, ',', "#");
      double[] trainY =
        MatToVec(MatLoad(trainFile,
        new int[] { colY }, ',', "#"));

      string testFile = "..\\..\\..\\Data\\" +
        "synthetic_test_40.txt";
      double[][] testX =
        MatLoad(testFile, colsX, ',', "#");
      double[] testY =
        MatToVec(MatLoad(testFile,
        new int[] { colY }, ',', "#"));
      Console.WriteLine("Done ");

      Console.WriteLine("\nFirst three train X: ");
      for (int i = 0; i "lt" 3; ++i)
        VecShow(trainX[i], 4, 8);

      Console.WriteLine("\nFirst three train y: ");
      for (int i = 0; i "lt" 3; ++i)
        Console.WriteLine(trainY[i].ToString("F4").
          PadLeft(8));

      // 2. create and train/build tree
      // small tree for blog, article
      int maxDepth = 3; //
      int minSamples = 2;
      int minLeaf = 18;
      int numSplitCols = -1;  // all columns

      Console.WriteLine("\nSetting maxDepth = " +
        maxDepth);
      Console.WriteLine("Setting minSamples = " +
        minSamples);
      Console.WriteLine("Setting minLeaf = " +
        minLeaf);
      Console.WriteLine("Using default numSplitCols = -1 ");

      Console.WriteLine("\nCreating and training tree ");
      DecisionTreeRegressor dtr =
        new DecisionTreeRegressor(maxDepth, minSamples,
        minLeaf, numSplitCols, seed: 0);
      dtr.Train(trainX, trainY);
      Console.WriteLine("Done ");

      Console.WriteLine("\nTree: ");
      dtr.Display(showRows: false);

      Console.WriteLine("\nRows associated w node [14]: ");
      for (int i = 0; i "lt" dtr.tree[14].rows.Count; ++i)
        Console.Write(dtr.tree[12].rows[i] + " ");
      Console.WriteLine("");

      // 3. evaluate model
      Console.WriteLine("\nEvaluating model ");
      double accTrain = dtr.Accuracy(trainX, trainY, 0.10);
      Console.WriteLine("Accuracy train (within 0.10) = " +
        accTrain.ToString("F4"));
      double accTest = dtr.Accuracy(testX, testY, 0.10);
      Console.WriteLine("Accuracy test (within 0.10) = " +
        accTest.ToString("F4"));

      double mseTrain = dtr.MSE(trainX, trainY);
      Console.WriteLine("\nMSE train = " +
        mseTrain.ToString("F4"));
      double mseTest = dtr.MSE(testX, testY);
      Console.WriteLine("MSE test = " +
        mseTest.ToString("F4"));

      // 4. use model
      Console.WriteLine("\nPredicting for trainX[0] = ");
      double[] x = trainX[0];
      VecShow(x, 4, 9);
      double predY = dtr.Predict(x);
      Console.WriteLine("Predicted y = " +
        predY.ToString("F4"));

      dtr.Explain(x);

      Console.WriteLine("\nEnd demo ");
      Console.ReadLine();

    } // Main()

    // ------------------------------------------------------
    // helpers for Main():
    //   MatLoad(), MatToVec(), VecShow().
    // ------------------------------------------------------

    static double[][] MatLoad(string fn, int[] usecols,
      char sep, string comment)
    {
      List"lt"double[]"gt" result =
        new List"lt"double[]"gt"();
      string line = "";
      FileStream ifs = new FileStream(fn, FileMode.Open);
      StreamReader sr = new StreamReader(ifs);
      while ((line = sr.ReadLine()) != null)
      {
        if (line.StartsWith(comment) == true)
          continue;
        string[] tokens = line.Split(sep);
        List"lt"double"gt" lst = new List"lt"double"gt"();
        for (int j = 0; j "lt" usecols.Length; ++j)
          lst.Add(double.Parse(tokens[usecols[j]]));
        double[] row = lst.ToArray();
        result.Add(row);
      }
      sr.Close(); ifs.Close();
      return result.ToArray();
    }

    static double[] MatToVec(double[][] mat)
    {
      int nRows = mat.Length;
      int nCols = mat[0].Length;
      double[] result = new double[nRows * nCols];
      int k = 0;
      for (int i = 0; i "lt" nRows; ++i)
        for (int j = 0; j "lt" nCols; ++j)
          result[k++] = mat[i][j];
      return result;
    }

    static void VecShow(double[] vec, int dec, int wid)
    {
      for (int i = 0; i "lt" vec.Length; ++i)
        Console.Write(vec[i].ToString("F" + dec).
          PadLeft(wid));
      Console.WriteLine("");
    }

  } // class Program

  // ========================================================

  public class DecisionTreeRegressor
  {
    public int maxDepth;
    public int minSamples;  // aka min_samples_split
    public int minLeaf;  // min number of values in a leaf
    public int numSplitCols; // mostly for random forest
    public List"lt"Node"gt" tree = new List"lt"Node"gt"();
    public Random rnd;  // order in which cols are searched

    public double[][] trainX;  // store data by ref
    public double[] trainY;

    // ------------------------------------------------------

    public class Node
    {
      public int id;
      public int colIdx;  // aka featureIdx
      public double thresh;
      public int left;  // index into List
      public int right;
      public double value;
      public bool isLeaf;
      public List"lt"int"gt" rows;  // rows train data

      public Node()
      {
        this.id = -1;
        this.colIdx = -1;
        this.thresh = 0.0;  // aka split value
        this.left = -1;
        this.right = -1;
        this.value = 0.0;  // aka pred y
        this.isLeaf = false;
        this.rows = null;
      }
    } // class Node

    // --------------------------------------------

    public DecisionTreeRegressor(int maxDepth = 2,
      int minSamples = 2, int minLeaf = 1,
      int numSplitCols = -1, int seed = 0)
    {
      // if maxDepth = 0, tree has just a root node
      // if maxDepth = 1, at most 3 nodes (root, l, r)
      // if maxDepth = n, at most 2^(n+1) - 1 nodes
      this.maxDepth = maxDepth;
      this.minSamples = minSamples;
      this.minLeaf = minLeaf;
      this.numSplitCols = numSplitCols;  // for ran. forest

      // create full tree List with null nodes
      int numNodes = (int)Math.Pow(2, (maxDepth + 1)) - 1;
      for (int i = 0; i "lt" numNodes; ++i)
      {
        this.tree.Add(null);  // empty nodes
      }
      this.rnd = new Random(seed);
    }

    // ------------------------------------------------------
    // public: Train(), Predict(), Explain(), Accuracy(),
    //   MSE(), Display().
    // helpers: MakeTree(), BestSplit(), TreeTargetMean(),
    //   TreeTargetVariance().
    // ------------------------------------------------------

    public void Train(double[][] trainX, double[] trainY)
    {
      this.trainX = trainX; // 
      this.trainY = trainY;
      this.MakeTree();
      // at this point the rows List in each node can be
      // nulled out to save space, especially if the
      // decision tree is part of an ensemble
    }

    // ------------------------------------------------------

    public double Predict(double[] x)
    {
      int p = 0;
      Node currNode = this.tree[p];
      while (currNode != null && 
        currNode.isLeaf == false &&
        p "lt" this.tree.Count)
      {
        if (x[currNode.colIdx] "lte" currNode.thresh)
          p = currNode.left;
        else
          p = currNode.right;
        currNode = this.tree[p];
      }
      return this.tree[p].value;
    }

    // ------------------------------------------------------

    public void Explain(double[] x)
    {
      int p = 0;
      Console.WriteLine("\nIF ");
      Node currNode = this.tree[p];
      while (currNode.isLeaf == false)
      {
        Console.Write("column ");
        Console.Write(currNode.colIdx + " ");
        if (x[currNode.colIdx] "lte" currNode.thresh)
        {
          Console.Write(" "lte" ");
          Console.WriteLine(currNode.thresh.
            ToString("F4").PadLeft(8) + " AND ");
          p = currNode.left;
          currNode = this.tree[p];
        }
        else
        {
          Console.Write(" "gt"  ");
          Console.WriteLine(currNode.thresh.
            ToString("F4").PadLeft(8) + " AND ");
          p = currNode.right;
          currNode = this.tree[p];
        }
      }
      Console.WriteLine("THEN \npredicted = " +
        currNode.value.ToString("F4"));
    }

    // ------------------------------------------------------

    public double Accuracy(double[][] dataX, double[] dataY,
      double pctClose)
    {
      int numCorrect = 0; int numWrong = 0;
      for (int i = 0; i "lt" dataX.Length; ++i)
      {
        double actualY = dataY[i];
        double predY = this.Predict(dataX[i]);
        if (Math.Abs(predY - actualY) "lt"
          Math.Abs(pctClose * actualY))
          ++numCorrect;
        else
          ++numWrong;
      }
      return (numCorrect * 1.0) / (numWrong + numCorrect);
    }

    // ------------------------------------------------------

    public double MSE(double[][] dataX,
      double[] dataY)
    {
      // standard machine learning MSE
      int n = dataX.Length;
      double sum = 0.0;
      for (int i = 0; i "lt" n; ++i)
      {
        double actualY = dataY[i];
        double predY = this.Predict(dataX[i]);
        sum += (actualY - predY) * (actualY - predY);
      }
      return sum / n;
    }

    // ------------------------------------------------------

    public void Display(bool showRows)
    {
      for (int i = 0; i "lt" this.tree.Count; ++i)
      {
        Node n = this.tree[i];
        // check for empty nodes
        if (n == null) continue;
        
        string s1 = "ID " +
          n.id.ToString().PadRight(3) + " | ";
        string s2 = "idx " +
          n.colIdx.ToString().PadLeft(3) + " | ";
        string s3 = "v " +
          n.thresh.ToString("F4").PadLeft(8) + " | ";
        string s4 = "L " +
          n.left.ToString().PadLeft(3) + " | ";
        string s5 = "R " +
          n.right.ToString().PadLeft(3) + " | ";
        string s6 = "y " +
          n.value.ToString("F4").PadLeft(8) + " | ";
        string s7 = "leaf " + (n.isLeaf == true ? "T":"F");
          //n.isLeaf.ToString().PadLeft(6);

        string s8;
        if (showRows == true)
        {
          s8 = "\nRows: \n";
          for (int j = 0; j "lt" n.rows.Count; ++j)
          {
            if (j "gt" 0 && j % 20 == 0)
              s8 += "\n";
            s8 += n.rows[j].ToString().PadLeft(4) + " ";
          }
        }
        else
        {
          s8 = "";
        }
        Console.WriteLine(s1 + s2 + s3 + s4 +
          s5 + s6 + s7 + s8);
        if (showRows == true) Console.WriteLine("");
      }
    } // Display()

    // ------------------------------------------------------

    // helpers: MakeTree(), BestSplit(),
    // TreeTargetMean(), TreeTargetVariance()

    private void MakeTree()
    {
      // no recursion, no pointers, List storage, no stack
      if (this.numSplitCols == -1) // use all cols
        this.numSplitCols = this.trainX[0].Length;

      // prepare root node
      List"lt"int"gt" allRows = new List"lt"int"gt"();
      for (int i = 0; i "lt" this.trainX.Length; ++i)
        allRows.Add(i);
      double grandMean = this.TreeTargetMean(allRows);
     
      // wait to supply colIdx and thresh in loop
      Node root = new Node();
      root.id = 0;
      root.left = 1;
      root.right = 2;
      root.value = grandMean;
      root.isLeaf = false; // already set
      root.rows = allRows;
      this.tree[0] = root;

      for (int i = 0; i "lt" this.tree.Count; ++i) 
      {
        Node currNode = this.tree[i];
        // curr node has values for everything
        // except colIdx and thresh

        // curr node too deep to have children OR
        // curr node not enough rows to split then
        // leave both children as null
        if (currNode == null || 
          currNode.rows.Count == 0) { continue; }

        // if parent cannot be split, make parent a leaf
        if (currNode.id "gte" (int)Math.Pow(2,
          (this.maxDepth)) - 1 ||
          currNode.rows.Count "lt" this.minSamples)
        {
          currNode.isLeaf = true;
          continue;
        }

        // parent has enough rows to try to split
        double[] splitInfo = this.BestSplit(currNode.rows);
        int colIdx = (int)splitInfo[0];
        double splitVal = splitInfo[1]; //split value

        if (colIdx == -1) // unable to split, parent is leaf
        {
          currNode.isLeaf = true;
          continue;
        }

        // complete the fields for curr node
        currNode.colIdx = colIdx;
        currNode.thresh = splitVal;
       
        // construct the children, except colIdx and thresh
        Node leftNode = new Node();
        Node rightNode = new Node();

        // construct the children rows using split info
        // all info except colIdx and thresh
        List"lt"int"gt" leftIdxs = new List"lt"int"gt"();
        List"lt"int"gt" rightIdxs = new List"lt"int"gt"();
        for (int k = 0; k "lt" currNode.rows.Count; ++k)
        {
          int r = currNode.rows[k];
          if (this.trainX[r][colIdx] "lte" splitVal)
            leftIdxs.Add(r);
          else
            rightIdxs.Add(r);
        }

        leftNode.id = currNode.id * 2 + 1;
        if (leftNode.id "gt" (int)Math.Pow(2,
          (maxDepth + 1)) - 2) leftNode.id = -1;

        leftNode.left = leftNode.id * 2 + 1;
        if (leftNode.left "gt" (int)Math.Pow(2,
          (maxDepth + 1)) - 2) leftNode.left = -1;

        leftNode.right = leftNode.id * 2 + 2;
        if (leftNode.right "gt" (int)Math.Pow(2,
          (maxDepth + 1)) - 2) leftNode.right = -1;

        leftNode.rows = leftIdxs;
        leftNode.value = 
          this.TreeTargetMean(leftNode.rows);
        this.tree[leftNode.id] = leftNode;

        rightNode.id = currNode.id * 2 + 2;
        if (rightNode.id "gt" (int)Math.Pow(2,
          (maxDepth + 1)) - 2) rightNode.id = -1;

        rightNode.left = rightNode.id * 2 + 1;
        if (rightNode.left "gt" (int)Math.Pow(2,
          (maxDepth + 1)) - 2) rightNode.left = -1;

        rightNode.right = rightNode.id * 2 + 2;
        if (rightNode.right "gt" (int)Math.Pow(2,
          (maxDepth + 1)) - 2) rightNode.right = -1;

        rightNode.rows = rightIdxs;
        rightNode.value = 
          this.TreeTargetMean(rightNode.rows);
        this.tree[rightNode.id] = rightNode;

      } // i
      return;
    }

    // ------------------------------------------------------

    private double[] BestSplit(List"lt"int"gt" rows)
    {
      // implicit params numSplitCols, minLeaf, numsplitCols
      // result[0] = best col idx (as double)
      // result[1] = best split value
      rows.Sort();

      int bestColIdx = -1;  // indicates bad split
      double bestThresh = 0.0;
      double bestVar = double.MaxValue;  // smaller is better

      int nRows = rows.Count;  // or dataY.Length
      int nCols = this.trainX[0].Length;

      if (nRows == 0)
      {
        throw new Exception("empty data in BestSplit()");
      }

      // process cols in scrambled order
      int[] colIndices = new int[nCols];
      for (int k = 0; k "lt" nCols; ++k)
        colIndices[k] = k;
      // shuffle, inline Fisher-Yates
      int n = colIndices.Length;
      for (int i = 0; i "lt" n; ++i)
      {
        int ri = rnd.Next(i, n);  // be careful
        int tmp = colIndices[i];
        colIndices[i] = colIndices[ri];
        colIndices[ri] = tmp;
      }

      // numSplitCols is usually all columns (-1)
      for (int j = 0; j "lt" this.numSplitCols; ++j)
      {
        int colIdx = colIndices[j];
        HashSet"lt"double"gt" examineds = 
          new HashSet"lt"double"gt"();

        for (int i = 0; i "lt" nRows; ++i) // each row
        {
          // if curr thresh been seen, skip it
          double thresh = this.trainX[rows[i]][colIdx];
          if (examineds.Contains(thresh)) continue;
          examineds.Add(thresh);

          // get row idxs where x is lte, gt thresh
          List"lt"int"gt" leftIdxs = new List"lt"int"gt"();
          List"lt"int"gt" rightIdxs = new List"lt"int"gt"();
          for (int k = 0; k "lt" nRows; ++k)
          {
            if (this.trainX[rows[k]][colIdx] "lte" thresh)
              leftIdxs.Add(rows[k]);
            else
              rightIdxs.Add(rows[k]);
          }

          // Check if proposed split has too few values
          if (leftIdxs.Count "lt" this.minLeaf ||
            rightIdxs.Count "lt" this.minLeaf)
            continue;  // to next row

          double leftVar = 
            this.TreeTargetVariance(leftIdxs);
          double rightVar = 
            this.TreeTargetVariance(rightIdxs);
          double weightedVar = (leftIdxs.Count * leftVar +
            rightIdxs.Count * rightVar) / nRows;

          if (weightedVar "lt" bestVar)
          {
            // if this never happens, bestColIdx remains -1
            // which means a bad split. used in MakeTree()
            bestColIdx = colIdx;
            bestThresh = thresh;
            bestVar = weightedVar;
          }

        } // each row
      } // j each col

      double[] result = new double[2];  // out params ugly
      result[0] = 1.0 * bestColIdx;
      result[1] = bestThresh;
      return result;

    } // BestSplit()

    // ------------------------------------------------------

    private double TreeTargetMean(List"lt"int"gt" rows)
    {
      // mean of rows items in trainY: for node prediction
      double sum = 0.0;
      for (int i = 0; i "lt" rows.Count; ++i)
      {
        int r = rows[i];
        sum += this.trainY[r];
      }
      return sum / rows.Count;
    }

    // ------------------------------------------------------

    private double TreeTargetVariance(List"lt"int"gt" rows)
    {
      double mean = this.TreeTargetMean(rows);
      double sum = 0.0;
      for (int i = 0; i "lt" rows.Count; ++i)
      {
        int r = rows[i];
        sum += (this.trainY[r] - mean) *
          (this.trainY[r] - mean);
      }
      return sum / rows.Count;
    }

    // ------------------------------------------------------

  } // class DecisionTreeRegressor

} // ns

Training data:

# synthetic_train_200.txt
#
-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
-0.8299, -0.9219, -0.6603,  0.7563, -0.8033,  0.7955
 0.0663,  0.3838, -0.3690,  0.3730,  0.6693,  0.3206
-0.9634,  0.5003,  0.9777,  0.4963, -0.4391,  0.7377
-0.1042,  0.8172, -0.4128, -0.4244, -0.7399,  0.4801
-0.9613,  0.3577, -0.5767, -0.4689, -0.0169,  0.6861
-0.7065,  0.1786,  0.3995, -0.7953, -0.1719,  0.5569
 0.3888, -0.1716, -0.9001,  0.0718,  0.3276,  0.2500
 0.1731,  0.8068, -0.7251, -0.7214,  0.6148,  0.3297
-0.2046, -0.6693,  0.8550, -0.3045,  0.5016,  0.2129
 0.2473,  0.5019, -0.3022, -0.4601,  0.7918,  0.2613
-0.1438,  0.9297,  0.3269,  0.2434, -0.7705,  0.5171
 0.1568, -0.1837, -0.5259,  0.8068,  0.1474,  0.3307
-0.9943,  0.2343, -0.3467,  0.0541,  0.7719,  0.5581
 0.2467, -0.9684,  0.8589,  0.3818,  0.9946,  0.1092
-0.6553, -0.7257,  0.8652,  0.3936, -0.8680,  0.7018
 0.8460,  0.4230, -0.7515, -0.9602, -0.9476,  0.1996
-0.9434, -0.5076,  0.7201,  0.0777,  0.1056,  0.5664
 0.9392,  0.1221, -0.9627,  0.6013, -0.5341,  0.1533
 0.6142, -0.2243,  0.7271,  0.4942,  0.1125,  0.1661
 0.4260,  0.1194, -0.9749, -0.8561,  0.9346,  0.2230
 0.1362, -0.5934, -0.4953,  0.4877, -0.6091,  0.3810
 0.6937, -0.5203, -0.0125,  0.2399,  0.6580,  0.1460
-0.6864, -0.9628, -0.8600, -0.0273,  0.2127,  0.5387
 0.9772,  0.1595, -0.2397,  0.1019,  0.4907,  0.1611
 0.3385, -0.4702, -0.8673, -0.2598,  0.2594,  0.2270
-0.8669, -0.4794,  0.6095, -0.6131,  0.2789,  0.4700
 0.0493,  0.8496, -0.4734, -0.8681,  0.4701,  0.3516
 0.8639, -0.9721, -0.5313,  0.2336,  0.8980,  0.1412
 0.9004,  0.1133,  0.8312,  0.2831, -0.2200,  0.1782
 0.0991,  0.8524,  0.8375, -0.2102,  0.9265,  0.2150
-0.6521, -0.7473, -0.7298,  0.0113, -0.9570,  0.7422
 0.6190, -0.3105,  0.8802,  0.1640,  0.7577,  0.1056
 0.6895,  0.8108, -0.0802,  0.0927,  0.5972,  0.2214
 0.1982, -0.9689,  0.1870, -0.1326,  0.6147,  0.1310
-0.3695,  0.7858,  0.1557, -0.6320,  0.5759,  0.3773
-0.1596,  0.3581,  0.8372, -0.9992,  0.9535,  0.2071
-0.2468,  0.9476,  0.2094,  0.6577,  0.1494,  0.4132
 0.1737,  0.5000,  0.7166,  0.5102,  0.3961,  0.2611
 0.7290, -0.3546,  0.3416, -0.0983, -0.2358,  0.1332
-0.3652,  0.2438, -0.1395,  0.9476,  0.3556,  0.4170
-0.6029, -0.1466, -0.3133,  0.5953,  0.7600,  0.4334
-0.4596, -0.4953,  0.7098,  0.0554,  0.6043,  0.2775
 0.1450,  0.4663,  0.0380,  0.5418,  0.1377,  0.2931
-0.8636, -0.2442, -0.8407,  0.9656, -0.6368,  0.7429
 0.6237,  0.7499,  0.3768,  0.1390, -0.6781,  0.2185
-0.5499,  0.1850, -0.3755,  0.8326,  0.8193,  0.4399
-0.4858, -0.7782, -0.6141, -0.0008,  0.4572,  0.4197
 0.7033, -0.1683,  0.2334, -0.5327, -0.7961,  0.1776
 0.0317, -0.0457, -0.6947,  0.2436,  0.0880,  0.3345
 0.5031, -0.5559,  0.0387,  0.5706, -0.9553,  0.3107
-0.3513,  0.7458,  0.6894,  0.0769,  0.7332,  0.3170
 0.2205,  0.5992, -0.9309,  0.5405,  0.4635,  0.3532
-0.4806, -0.4859,  0.2646, -0.3094,  0.5932,  0.3202
 0.9809, -0.3995, -0.7140,  0.8026,  0.0831,  0.1600
 0.9495,  0.2732,  0.9878,  0.0921,  0.0529,  0.1289
-0.9476, -0.6792,  0.4913, -0.9392, -0.2669,  0.5966
 0.7247,  0.3854,  0.3819, -0.6227, -0.1162,  0.1550
-0.5922, -0.5045, -0.4757,  0.5003, -0.0860,  0.5863
-0.8861,  0.0170, -0.5761,  0.5972, -0.4053,  0.7301
 0.6877, -0.2380,  0.4997,  0.0223,  0.0819,  0.1404
 0.9189,  0.6079, -0.9354,  0.4188, -0.0700,  0.1907
-0.1428, -0.7820,  0.2676,  0.6059,  0.3936,  0.2790
 0.5324, -0.3151,  0.6917, -0.1425,  0.6480,  0.1071
-0.8432, -0.9633, -0.8666, -0.0828, -0.7733,  0.7784
-0.9444,  0.5097, -0.2103,  0.4939, -0.0952,  0.6787
-0.0520,  0.6063, -0.1952,  0.8094, -0.9259,  0.4836
 0.5477, -0.7487,  0.2370, -0.9793,  0.0773,  0.1241
 0.2450,  0.8116,  0.9799,  0.4222,  0.4636,  0.2355
 0.8186, -0.1983, -0.5003, -0.6531, -0.7611,  0.1511
-0.4714,  0.6382, -0.3788,  0.9648, -0.4667,  0.5950
 0.0673, -0.3711,  0.8215, -0.2669, -0.1328,  0.2677
-0.9381,  0.4338,  0.7820, -0.9454,  0.0441,  0.5518
-0.3480,  0.7190,  0.1170,  0.3805, -0.0943,  0.4724
-0.9813,  0.1535, -0.3771,  0.0345,  0.8328,  0.5438
-0.1471, -0.5052, -0.2574,  0.8637,  0.8737,  0.3042
-0.5454, -0.3712, -0.6505,  0.2142, -0.1728,  0.5783
 0.6327, -0.6297,  0.4038, -0.5193,  0.1484,  0.1153
-0.5424,  0.3282, -0.0055,  0.0380, -0.6506,  0.6613
 0.1414,  0.9935,  0.6337,  0.1887,  0.9520,  0.2540
-0.9351, -0.8128, -0.8693, -0.0965, -0.2491,  0.7353
 0.9507, -0.6640,  0.9456,  0.5349,  0.6485,  0.1059
-0.0462, -0.9737, -0.2940, -0.0159,  0.4602,  0.2606
-0.0627, -0.0852, -0.7247, -0.9782,  0.5166,  0.2977
 0.0478,  0.5098, -0.0723, -0.7504, -0.3750,  0.3335
 0.0090,  0.3477,  0.5403, -0.7393, -0.9542,  0.4415
-0.9748,  0.3449,  0.3736, -0.1015,  0.8296,  0.4358
 0.2887, -0.9895, -0.0311,  0.7186,  0.6608,  0.2057
 0.1570, -0.4518,  0.1211,  0.3435, -0.2951,  0.3244
 0.7117, -0.6099,  0.4946, -0.4208,  0.5476,  0.1096
-0.2929, -0.5726,  0.5346, -0.3827,  0.4665,  0.2465
 0.4889, -0.5572, -0.5718, -0.6021, -0.7150,  0.2163
-0.7782,  0.3491,  0.5996, -0.8389, -0.5366,  0.6516
-0.5847,  0.8347,  0.4226,  0.1078, -0.3910,  0.6134
 0.8469,  0.4121, -0.0439, -0.7476,  0.9521,  0.1571
-0.6803, -0.5948, -0.1376, -0.1916, -0.7065,  0.7156
 0.2878,  0.5086, -0.5785,  0.2019,  0.4979,  0.2980
 0.2764,  0.1943, -0.4090,  0.4632,  0.8906,  0.2960
-0.8877,  0.6705, -0.6155, -0.2098, -0.3998,  0.7107
-0.8398,  0.8093, -0.2597,  0.0614, -0.0118,  0.6502
-0.8476,  0.0158, -0.4769, -0.2859, -0.7839,  0.7715
 0.5751, -0.7868,  0.9714, -0.6457,  0.1448,  0.1175
 0.4802, -0.7001,  0.1022, -0.5668,  0.5184,  0.1090
 0.4458, -0.6469,  0.7239, -0.9604,  0.7205,  0.0779
 0.5175,  0.4339,  0.9747, -0.4438, -0.9924,  0.2879
 0.8678,  0.7158,  0.4577,  0.0334,  0.4139,  0.1678
 0.5406,  0.5012,  0.2264, -0.1963,  0.3946,  0.2088
-0.9938,  0.5498,  0.7928, -0.5214, -0.7585,  0.7687
 0.7661,  0.0863, -0.4266, -0.7233, -0.4197,  0.1466
 0.2277, -0.3517, -0.0853, -0.1118,  0.6563,  0.1767
 0.3499, -0.5570, -0.0655, -0.3705,  0.2537,  0.1632
 0.7547, -0.1046,  0.5689, -0.0861,  0.3125,  0.1257
 0.8186,  0.2110,  0.5335,  0.0094, -0.0039,  0.1391
 0.6858, -0.8644,  0.1465,  0.8855,  0.0357,  0.1845
-0.4967,  0.4015,  0.0805,  0.8977,  0.2487,  0.4663
 0.6760, -0.9841,  0.9787, -0.8446, -0.3557,  0.1509
-0.1203, -0.4885,  0.6054, -0.0443, -0.7313,  0.4854
 0.8557,  0.7919, -0.0169,  0.7134, -0.1628,  0.2002
 0.0115, -0.6209,  0.9300, -0.4116, -0.7931,  0.4052
-0.7114, -0.9718,  0.4319,  0.1290,  0.5892,  0.3661
 0.3915,  0.5557, -0.1870,  0.2955, -0.6404,  0.2954
-0.3564, -0.6548, -0.1827, -0.5172, -0.1862,  0.4622
 0.2392, -0.4959,  0.5857, -0.1341, -0.2850,  0.2470
-0.3394,  0.3947, -0.4627,  0.6166, -0.4094,  0.5325
 0.7107,  0.7768, -0.6312,  0.1707,  0.7964,  0.2757
-0.1078,  0.8437, -0.4420,  0.2177,  0.3649,  0.4028
-0.3139,  0.5595, -0.6505, -0.3161, -0.7108,  0.5546
 0.4335,  0.3986,  0.3770, -0.4932,  0.3847,  0.1810
-0.2562, -0.2894, -0.8847,  0.2633,  0.4146,  0.4036
 0.2272,  0.2966, -0.6601, -0.7011,  0.0284,  0.2778
-0.0743, -0.1421, -0.0054, -0.6770, -0.3151,  0.3597
-0.4762,  0.6891,  0.6007, -0.1467,  0.2140,  0.4266
-0.4061,  0.7193,  0.3432,  0.2669, -0.7505,  0.6147
-0.0588,  0.9731,  0.8966,  0.2902, -0.6966,  0.4955
-0.0627, -0.1439,  0.1985,  0.6999,  0.5022,  0.3077
 0.1587,  0.8494, -0.8705,  0.9827, -0.8940,  0.4263
-0.7850,  0.2473, -0.9040, -0.4308, -0.8779,  0.7199
 0.4070,  0.3369, -0.2428, -0.6236,  0.4940,  0.2215
-0.0242,  0.0513, -0.9430,  0.2885, -0.2987,  0.3947
-0.5416, -0.1322, -0.2351, -0.0604,  0.9590,  0.3683
 0.1055,  0.7783, -0.2901, -0.5090,  0.8220,  0.2984
-0.9129,  0.9015,  0.1128, -0.2473,  0.9901,  0.4776
-0.9378,  0.1424, -0.6391,  0.2619,  0.9618,  0.5368
 0.7498, -0.0963,  0.4169,  0.5549, -0.0103,  0.1614
-0.2612, -0.7156,  0.4538, -0.0460, -0.1022,  0.3717
 0.7720,  0.0552, -0.1818, -0.4622, -0.8560,  0.1685
-0.4177,  0.0070,  0.9319, -0.7812,  0.3461,  0.3052
-0.0001,  0.5542, -0.7128, -0.8336, -0.2016,  0.3803
 0.5356, -0.4194, -0.5662, -0.9666, -0.2027,  0.1776
-0.2378,  0.3187, -0.8582, -0.6948, -0.9668,  0.5474
-0.1947, -0.3579,  0.1158,  0.9869,  0.6690,  0.2992
 0.3992,  0.8365, -0.9205, -0.8593, -0.0520,  0.3154
-0.0209,  0.0793,  0.7905, -0.1067,  0.7541,  0.1864
-0.4928, -0.4524, -0.3433,  0.0951, -0.5597,  0.6261
-0.8118,  0.7404, -0.5263, -0.2280,  0.1431,  0.6349
 0.0516, -0.8480,  0.7483,  0.9023,  0.6250,  0.1959
-0.3212,  0.1093,  0.9488, -0.3766,  0.3376,  0.2735
-0.3481,  0.5490, -0.3484,  0.7797,  0.5034,  0.4379
-0.5785, -0.9170, -0.3563, -0.9258,  0.3877,  0.4121
 0.3407, -0.1391,  0.5356,  0.0720, -0.9203,  0.3458
-0.3287, -0.8954,  0.2102,  0.0241,  0.2349,  0.3247
-0.1353,  0.6954, -0.0919, -0.9692,  0.7461,  0.3338
 0.9036, -0.8982, -0.5299, -0.8733, -0.1567,  0.1187
 0.7277, -0.8368, -0.0538, -0.7489,  0.5458,  0.0830
 0.9049,  0.8878,  0.2279,  0.9470, -0.3103,  0.2194
 0.7957, -0.1308, -0.5284,  0.8817,  0.3684,  0.2172
 0.4647, -0.4931,  0.2010,  0.6292, -0.8918,  0.3371
-0.7390,  0.6849,  0.2367,  0.0626, -0.5034,  0.7039
-0.1567, -0.8711,  0.7940, -0.5932,  0.6525,  0.1710
 0.7635, -0.0265,  0.1969,  0.0545,  0.2496,  0.1445
 0.7675,  0.1354, -0.7698, -0.5460,  0.1920,  0.1728
-0.5211, -0.7372, -0.6763,  0.6897,  0.2044,  0.5217
 0.1913,  0.1980,  0.2314, -0.8816,  0.5006,  0.1998
 0.8964,  0.0694, -0.6149,  0.5059, -0.9854,  0.1825
 0.1767,  0.7104,  0.2093,  0.6452,  0.7590,  0.2832
-0.3580, -0.7541,  0.4426, -0.1193, -0.7465,  0.5657
-0.5996,  0.5766, -0.9758, -0.3933, -0.9572,  0.6800
 0.9950,  0.1641, -0.4132,  0.8579,  0.0142,  0.2003
-0.4717, -0.3894, -0.2567, -0.5111,  0.1691,  0.4266
 0.3917, -0.8561,  0.9422,  0.5061,  0.6123,  0.1212
-0.0366, -0.1087,  0.3449, -0.1025,  0.4086,  0.2475
 0.3633,  0.3943,  0.2372, -0.6980,  0.5216,  0.1925
-0.5325, -0.6466, -0.2178, -0.3589,  0.6310,  0.3568
 0.2271,  0.5200, -0.1447, -0.8011, -0.7699,  0.3128
 0.6415,  0.1993,  0.3777, -0.0178, -0.8237,  0.2181
-0.5298, -0.0768, -0.6028, -0.9490,  0.4588,  0.4356
 0.6870, -0.1431,  0.7294,  0.3141,  0.1621,  0.1632
-0.5985,  0.0591,  0.7889, -0.3900,  0.7419,  0.2945
 0.3661,  0.7984, -0.8486,  0.7572, -0.6183,  0.3449
 0.6995,  0.3342, -0.3113, -0.6972,  0.2707,  0.1712
 0.2565,  0.9126,  0.1798, -0.6043, -0.1413,  0.2893
-0.3265,  0.9839, -0.2395,  0.9854,  0.0376,  0.4770
 0.2690, -0.1722,  0.9818,  0.8599, -0.7015,  0.3954
-0.2102, -0.0768,  0.1219,  0.5607, -0.0256,  0.3949
 0.8216, -0.9555,  0.6422, -0.6231,  0.3715,  0.0801
-0.2896,  0.9484, -0.7545, -0.6249,  0.7789,  0.4370
-0.9985, -0.5448, -0.7092, -0.5931,  0.7926,  0.5402

Test data:

# synthetic_test_40.txt
#
 0.7462,  0.4006, -0.0590,  0.6543, -0.0083,  0.1935
 0.8495, -0.2260, -0.0142, -0.4911,  0.7699,  0.1078
-0.2335, -0.4049,  0.4352, -0.6183, -0.7636,  0.5088
 0.1810, -0.5142,  0.2465,  0.2767, -0.3449,  0.3136
-0.8650,  0.7611, -0.0801,  0.5277, -0.4922,  0.7140
-0.2358, -0.7466, -0.5115, -0.8413, -0.3943,  0.4533
 0.4834,  0.2300,  0.3448, -0.9832,  0.3568,  0.1360
-0.6502, -0.6300,  0.6885,  0.9652,  0.8275,  0.3046
-0.3053,  0.5604,  0.0929,  0.6329, -0.0325,  0.4756
-0.7995,  0.0740, -0.2680,  0.2086,  0.9176,  0.4565
-0.2144, -0.2141,  0.5813,  0.2902, -0.2122,  0.4119
-0.7278, -0.0987, -0.3312, -0.5641,  0.8515,  0.4438
 0.3793,  0.1976,  0.4933,  0.0839,  0.4011,  0.1905
-0.8568,  0.9573, -0.5272,  0.3212, -0.8207,  0.7415
-0.5785,  0.0056, -0.7901, -0.2223,  0.0760,  0.5551
 0.0735, -0.2188,  0.3925,  0.3570,  0.3746,  0.2191
 0.1230, -0.2838,  0.2262,  0.8715,  0.1938,  0.2878
 0.4792, -0.9248,  0.5295,  0.0366, -0.9894,  0.3149
-0.4456,  0.0697,  0.5359, -0.8938,  0.0981,  0.3879
 0.8629, -0.8505, -0.4464,  0.8385,  0.5300,  0.1769
 0.1995,  0.6659,  0.7921,  0.9454,  0.9970,  0.2330
-0.0249, -0.3066, -0.2927, -0.4923,  0.8220,  0.2437
 0.4513, -0.9481, -0.0770, -0.4374, -0.9421,  0.2879
-0.3405,  0.5931, -0.3507, -0.3842,  0.8562,  0.3987
 0.9538,  0.0471,  0.9039,  0.7760,  0.0361,  0.1706
-0.0887,  0.2104,  0.9808,  0.5478, -0.3314,  0.4128
-0.8220, -0.6302,  0.0537, -0.1658,  0.6013,  0.4306
-0.4123, -0.2880,  0.9074, -0.0461, -0.4435,  0.5144
 0.0060,  0.2867, -0.7775,  0.5161,  0.7039,  0.3599
-0.7968, -0.5484,  0.9426, -0.4308,  0.8148,  0.2979
 0.7811,  0.8450, -0.6877,  0.7594,  0.2640,  0.2362
-0.6802, -0.1113, -0.8325, -0.6694, -0.6056,  0.6544
 0.3821,  0.1476,  0.7466, -0.5107,  0.2592,  0.1648
 0.7265,  0.9683, -0.9803, -0.4943, -0.5523,  0.2454
-0.9049, -0.9797, -0.0196, -0.9090, -0.4433,  0.6447
-0.4607,  0.1811, -0.2389,  0.4050, -0.0078,  0.5229
 0.2664, -0.2932, -0.4259, -0.7336,  0.8742,  0.1834
-0.4507,  0.1029, -0.6294, -0.1158, -0.6294,  0.6081
 0.8948, -0.0124,  0.9278,  0.2899, -0.0314,  0.1534
-0.1323, -0.8813, -0.0146, -0.0697,  0.6135,  0.2386

One of the curses of being fascinated by machine learning and computer science is that no problem ever completely leaves your mind. I have implemented k-nearest neighbors regression many times over the years, but I revisit the problem from time to time to incorporate new ideas.

The k-nearest neighbors regression technique is very simple. To predict the y value for an input vector x, you find the k-nearest data items in a set of training data (k is usually set to 3 or 5), then compute the average of the target y values that are associated with the closest data items.

The output of a demo run illustrates the ideas:

Begin k-NN regression using C#

Loading synthetic train (200) and test (40) data

First three train X:
 -0.1660  0.4406 -0.9998 -0.3953 -0.7065
  0.0776 -0.1616  0.3704 -0.5911  0.7562
 -0.9452  0.3409 -0.1654  0.1174 -0.7192

First three train y:
  0.4840
  0.1568
  0.8054

Creating k-NN regression model, k=4
Done

Accuracy train (within 0.10): 0.6100
Accuracy test (within 0.10): 0.5000

MSE train = 0.0017
MSE test = 0.0023

Predicting for trainX[0]
Predicted y = 0.5665

Explaining for train[0]
X = [  0]  -0.17  0.44 -1.00 -0.40 -0.71 | y = 0.4840 | dist = 0.0000 | wt = 0.2500
X = [129]  -0.31  0.56 -0.65 -0.32 -0.71 | y = 0.5546 | dist = 0.4054 | wt = 0.2500
X = [152]  -0.24  0.32 -0.86 -0.69 -0.97 | y = 0.5474 | dist = 0.4444 | wt = 0.2500
X = [179]  -0.60  0.58 -0.98 -0.39 -0.96 | y = 0.6800 | dist = 0.5196 | wt = 0.2500

Predicted y = 0.5665

End demo

I used a set of synthetic data that looks like:

-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345

The first five values on each line are predictors. The last value is the target y to predict. There are 200 training items and 40 test items.

Suppose you set k, the number of nearest neighbors to use, to 4. To predict the y value for the first training item, x = (-0.1660, 0.4406, -0.9998, -0.3953, -0.7065), the system finds the four closest items to x in the training data. It turns out that the closest items, from closest to farthest, are items [0], [129], [152], and [179]. The four associated target y values are (0.4840, 0.5546, 0.5474, 0.6800). Therefore the predicted y is the average of those four y values: (0.4840 + 0.5546 + 0.5474 + 0.6800) / 4 = 0.5665.

Instead of computing a simple average of the closest y values, an alternative design is to weight closer items more heavily. For example you could set four weights to (0.40, 0.30, 0.20, 0.10) then the predicted y value would be computed as (0.4840 * 0.40) + (0.5546 * 0.30) + (0.5474 * 0.20) + (0.6800 * 0.10) = 0.5375.

The problem with weighting the associated target y values to compute the predicted y is that there’s no principled way to set the weight values. In most cases, using equal weights (equivalent to computing the average) works well.

Nearest neighbors regression often works well when the target values tend to clump together, but nearest neighbors regression doesn’t work well when target values are uniformly spread over the target space.

Good fun for me.

I’m a big fan of Sherlock Holmes movies. Actor Basil Rathbone played Sherlock Holmes in 14 films between 1939 and 1946. In the series, three different actors played Holmes’ evil nemesis, Professor Moriarty. All three actors did an excellent job in my opinion.

Right: In “Sherlock Holmes and the Secret Weapon” (1943), actor Lionel Atwell (1885-1946) plays Moriarty, who attempts to steal a new bombsight and give it to Nazi Germany during World War II. Holmes is captured by Moriarty, but is rescued at the last moment by Dr. Watson and Inspector Lestrade (on the right).

Demo program. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols (my blog editor chokes on symbols)

using System;
using System.IO;
using System.Collections.Generic;

namespace KNNRegression
{
  internal class KNNRegressionProgram
  {
    static void Main(string[] args)
    {
      Console.WriteLine("\nBegin k-NN regression" +
        " using C# ");

      // 1. load data
      Console.WriteLine("\nLoading synthetic train" +
        " (200) and test (40) data ");
      string trainFile =
        "..\\..\\..\\Data\\synthetic_train_200.txt";
      int[] colsX = new int[] { 0, 1, 2, 3, 4 };
      double[][] trainX =
        MatLoad(trainFile, colsX, ',', "#");
      double[] trainY =
        MatToVec(MatLoad(trainFile,
        [5], ',', "#"));

      Console.WriteLine("\nFirst three train X: ");
      for (int i = 0; i "lt" 3; ++i)
        VecShow(trainX[i], 4, 8);

      Console.WriteLine("\nFirst three train y: ");
      for (int i = 0; i "lt" 3; ++i)
        Console.WriteLine(trainY[i].ToString("F4").
        PadLeft(8));

      string testFile =
        "..\\..\\..\\Data\\synthetic_test_40.txt";
      double[][] testX =
        MatLoad(testFile, colsX, ',', "#");
      double[] testY =
        MatToVec(MatLoad(testFile,
        [5], ',', "#"));

      // 2. create and train/fit model
      int numNeighbors = 4;
      Console.WriteLine("\nCreating k-NN regression " +
        "model, k=" + numNeighbors);
      KNNR model = new KNNR(numNeighbors);
      model.Train(trainX, trainY); // aka Fit()
      Console.WriteLine("Done ");

      // 3. evaluate model
      double accTrain = model.Accuracy(trainX, trainY, 0.10);
      Console.WriteLine("\nAccuracy train (within 0.10): " +
        accTrain.ToString("F4"));

      double accTest = model.Accuracy(testX, testY, 0.10);
      Console.WriteLine("Accuracy test (within 0.10): " +
        accTest.ToString("F4"));

      double mseTrain = model.MSE(trainX, trainY);
      Console.WriteLine("\nMSE train = " +
        mseTrain.ToString("F4"));
      double mseTest = model.MSE(testX, testY);
      Console.WriteLine("MSE test = " +
        mseTest.ToString("F4"));

      // 4. use model
      Console.WriteLine("\nPredicting for trainX[0] ");
      double[] x = trainX[0];
      double y = model.Predict(x);
      Console.WriteLine("Predicted y = " +
        y.ToString("F4").PadLeft(4));

      Console.WriteLine("\nExplaining for train[0] ");
      model.Explain(x);

      Console.WriteLine("\nEnd demo ");
      Console.ReadLine();
    } // Main

    // helpers for Main(): MatLoad, MatToVec, VecShow

    static double[][] MatLoad(string fn, int[] usecols,
      char sep, string comment)
    {
      List"lt"double[]"gt" result = 
        new List"lt"double[]"gt"();
      string line = "";
      FileStream ifs = new FileStream(fn, FileMode.Open);
      StreamReader sr = new StreamReader(ifs);
      while ((line = sr.ReadLine()) != null)
      {
        if (line.StartsWith(comment) == true)
          continue;
        string[] tokens = line.Split(sep);
        List"lt"double"gt" lst = 
          new List"lt"double"gt"();
        for (int j = 0; j "lt" usecols.Length; ++j)
          lst.Add(double.Parse(tokens[usecols[j]]));
        double[] row = lst.ToArray();
        result.Add(row);
      }
      sr.Close(); ifs.Close();
      return result.ToArray();
    }

    static double[] MatToVec(double[][] mat)
    {
      int nRows = mat.Length;
      int nCols = mat[0].Length;
      double[] result = new double[nRows * nCols];
      int k = 0;
      for (int i = 0; i "lt" nRows; ++i)
        for (int j = 0; j "lt" nCols; ++j)
          result[k++] = mat[i][j];
      return result;
    }

    public static void VecShow(double[] vec,
      int dec, int wid)
    {
      for (int i = 0; i "lt" vec.Length; ++i)
        Console.Write(vec[i].ToString("F" + dec).
          PadLeft(wid));
      Console.WriteLine("");
    }

  } // class Program

  public class KNNR
  {
    public int nNeighbors;
    public double[] weights;
    public double[][] trainX;
    public double[] trainY;

    public KNNR(int nNeighbors)
    {
      this.nNeighbors = nNeighbors;
      this.weights = new double[nNeighbors];
      for (int k = 0; k "lt" nNeighbors; ++k)
        this.weights[k] = 1.0 / nNeighbors;
      this.trainX = new double[0][]; // quasi-null
      this.trainY = new double[0];
    }

    public void Train(double[][] trainX,
      double[] trainY)
    {
      this.trainX = trainX; // store by ref
      this.trainY = trainY;
    }

    public double Predict(double[] x)
    {
      // compute distances from x to all trainX
      int N = trainX.Length;
      double[] dists = new double[N];
      for (int i = 0; i "lt" N; ++i)
        dists[i] = Distance(x, this.trainX[i]);
      // determine sort order
      int[] sortedIdxs = ArgSort(dists);
      double sum = 0.0;
      for (int k = 0; k "lt" this.nNeighbors; ++k)
      {
        int idx = sortedIdxs[k]; // near to far
        sum += this.trainY[idx] * this.weights[k];
      }
      return sum;
    }

    public void Explain(double[] x)
    {
      // 0. set up ordering/indices
      int n = this.trainX.Length;
      int[] indices = new int[n];
      for (int i = 0; i "lt" n; ++i)
        indices[i] = i;

      // 1. compute distances from x to all trainX
      double[] distances = new double[n];
      for (int i = 0; i "lt" n; ++i)
        distances[i] = Distance(x, this.trainX[i]);

      // 2. sort distances, and indices by distances
      Array.Sort(distances, indices);

      // 3. compute predicted y
      double predY = 0.0;
      for (int k = 0; k "lt" this.nNeighbors; ++k)
        predY += this.weights[k] * this.trainY[indices[k]];

      // 4. show info 
      for (int k = 0; k "lt" this.nNeighbors; ++k)
      {
        int i = indices[k];
        Console.Write("X = ");
        Console.Write("[" + i.ToString().PadLeft(3) + "] ");
 
        for (int j = 0; j "lt" this.trainX[i].Length; ++j)
          Console.Write(this.trainX[i][j].
          ToString("F2").PadLeft(6));

        Console.Write(" | y = ");
        Console.Write(this.trainY[i].ToString("F4"));
        Console.Write(" | dist = ");
        Console.Write(distances[k].ToString("F4"));
        Console.Write(" | wt = ");
        Console.Write(this.weights[k].ToString("F4"));
        Console.WriteLine("");
      }

      Console.WriteLine("\nPredicted y = " +
        predY.ToString("F4"));
    } // Explain

    public double Accuracy(double[][] dataX,
      double[] dataY, double pctClose)
    {
      int nCorrect = 0; int nWrong = 0;
      for (int i = 0; i "lt" dataX.Length; ++i)
      {
        double predY = this.Predict(dataX[i]);
        double actualY = dataY[i];
        if (Math.Abs(predY - actualY) "lt" 
          Math.Abs(pctClose * actualY))
          ++nCorrect;
        else
          ++nWrong;
      }
      return (nCorrect * 1.0) / (nCorrect + nWrong);
    }

    public double MSE(double[][] dataX, double[] dataY)
    {
      double sum = 0.0;
      for (int i = 0; i "lt" dataX.Length; ++i)
      {
        double predY = this.Predict(dataX[i]);
        double actualY = dataY[i];
        sum += (predY - actualY) * (predY - actualY);
      }
      return sum / dataX.Length;
    }

    private static double Distance(double[] v1,
      double[] v2)
    {
      // Euclidean distance
      int n = v1.Length;
      double sum = 0.0;
      for (int i = 0; i "lt" n; ++i)
        sum += (v1[i] - v2[i]) * (v1[i] - v2[i]);
      return Math.Sqrt(sum);
    }

    private static int[] ArgSort(double[] values)
    {
      int n = values.Length;
      double[] copy = new double[n];
      int[] indices = new int[n];
      for (int i = 0; i "lt" n; ++i)
      {
        copy[i] = values[i];
        indices[i] = i;
      }
      Array.Sort(copy, indices);
      return indices;
    }
  } // class KNNR

} // ns

Training data:

# synthetic_train_200.txt
#
-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
-0.8299, -0.9219, -0.6603,  0.7563, -0.8033,  0.7955
 0.0663,  0.3838, -0.3690,  0.3730,  0.6693,  0.3206
-0.9634,  0.5003,  0.9777,  0.4963, -0.4391,  0.7377
-0.1042,  0.8172, -0.4128, -0.4244, -0.7399,  0.4801
-0.9613,  0.3577, -0.5767, -0.4689, -0.0169,  0.6861
-0.7065,  0.1786,  0.3995, -0.7953, -0.1719,  0.5569
 0.3888, -0.1716, -0.9001,  0.0718,  0.3276,  0.2500
 0.1731,  0.8068, -0.7251, -0.7214,  0.6148,  0.3297
-0.2046, -0.6693,  0.8550, -0.3045,  0.5016,  0.2129
 0.2473,  0.5019, -0.3022, -0.4601,  0.7918,  0.2613
-0.1438,  0.9297,  0.3269,  0.2434, -0.7705,  0.5171
 0.1568, -0.1837, -0.5259,  0.8068,  0.1474,  0.3307
-0.9943,  0.2343, -0.3467,  0.0541,  0.7719,  0.5581
 0.2467, -0.9684,  0.8589,  0.3818,  0.9946,  0.1092
-0.6553, -0.7257,  0.8652,  0.3936, -0.8680,  0.7018
 0.8460,  0.4230, -0.7515, -0.9602, -0.9476,  0.1996
-0.9434, -0.5076,  0.7201,  0.0777,  0.1056,  0.5664
 0.9392,  0.1221, -0.9627,  0.6013, -0.5341,  0.1533
 0.6142, -0.2243,  0.7271,  0.4942,  0.1125,  0.1661
 0.4260,  0.1194, -0.9749, -0.8561,  0.9346,  0.2230
 0.1362, -0.5934, -0.4953,  0.4877, -0.6091,  0.3810
 0.6937, -0.5203, -0.0125,  0.2399,  0.6580,  0.1460
-0.6864, -0.9628, -0.8600, -0.0273,  0.2127,  0.5387
 0.9772,  0.1595, -0.2397,  0.1019,  0.4907,  0.1611
 0.3385, -0.4702, -0.8673, -0.2598,  0.2594,  0.2270
-0.8669, -0.4794,  0.6095, -0.6131,  0.2789,  0.4700
 0.0493,  0.8496, -0.4734, -0.8681,  0.4701,  0.3516
 0.8639, -0.9721, -0.5313,  0.2336,  0.8980,  0.1412
 0.9004,  0.1133,  0.8312,  0.2831, -0.2200,  0.1782
 0.0991,  0.8524,  0.8375, -0.2102,  0.9265,  0.2150
-0.6521, -0.7473, -0.7298,  0.0113, -0.9570,  0.7422
 0.6190, -0.3105,  0.8802,  0.1640,  0.7577,  0.1056
 0.6895,  0.8108, -0.0802,  0.0927,  0.5972,  0.2214
 0.1982, -0.9689,  0.1870, -0.1326,  0.6147,  0.1310
-0.3695,  0.7858,  0.1557, -0.6320,  0.5759,  0.3773
-0.1596,  0.3581,  0.8372, -0.9992,  0.9535,  0.2071
-0.2468,  0.9476,  0.2094,  0.6577,  0.1494,  0.4132
 0.1737,  0.5000,  0.7166,  0.5102,  0.3961,  0.2611
 0.7290, -0.3546,  0.3416, -0.0983, -0.2358,  0.1332
-0.3652,  0.2438, -0.1395,  0.9476,  0.3556,  0.4170
-0.6029, -0.1466, -0.3133,  0.5953,  0.7600,  0.4334
-0.4596, -0.4953,  0.7098,  0.0554,  0.6043,  0.2775
 0.1450,  0.4663,  0.0380,  0.5418,  0.1377,  0.2931
-0.8636, -0.2442, -0.8407,  0.9656, -0.6368,  0.7429
 0.6237,  0.7499,  0.3768,  0.1390, -0.6781,  0.2185
-0.5499,  0.1850, -0.3755,  0.8326,  0.8193,  0.4399
-0.4858, -0.7782, -0.6141, -0.0008,  0.4572,  0.4197
 0.7033, -0.1683,  0.2334, -0.5327, -0.7961,  0.1776
 0.0317, -0.0457, -0.6947,  0.2436,  0.0880,  0.3345
 0.5031, -0.5559,  0.0387,  0.5706, -0.9553,  0.3107
-0.3513,  0.7458,  0.6894,  0.0769,  0.7332,  0.3170
 0.2205,  0.5992, -0.9309,  0.5405,  0.4635,  0.3532
-0.4806, -0.4859,  0.2646, -0.3094,  0.5932,  0.3202
 0.9809, -0.3995, -0.7140,  0.8026,  0.0831,  0.1600
 0.9495,  0.2732,  0.9878,  0.0921,  0.0529,  0.1289
-0.9476, -0.6792,  0.4913, -0.9392, -0.2669,  0.5966
 0.7247,  0.3854,  0.3819, -0.6227, -0.1162,  0.1550
-0.5922, -0.5045, -0.4757,  0.5003, -0.0860,  0.5863
-0.8861,  0.0170, -0.5761,  0.5972, -0.4053,  0.7301
 0.6877, -0.2380,  0.4997,  0.0223,  0.0819,  0.1404
 0.9189,  0.6079, -0.9354,  0.4188, -0.0700,  0.1907
-0.1428, -0.7820,  0.2676,  0.6059,  0.3936,  0.2790
 0.5324, -0.3151,  0.6917, -0.1425,  0.6480,  0.1071
-0.8432, -0.9633, -0.8666, -0.0828, -0.7733,  0.7784
-0.9444,  0.5097, -0.2103,  0.4939, -0.0952,  0.6787
-0.0520,  0.6063, -0.1952,  0.8094, -0.9259,  0.4836
 0.5477, -0.7487,  0.2370, -0.9793,  0.0773,  0.1241
 0.2450,  0.8116,  0.9799,  0.4222,  0.4636,  0.2355
 0.8186, -0.1983, -0.5003, -0.6531, -0.7611,  0.1511
-0.4714,  0.6382, -0.3788,  0.9648, -0.4667,  0.5950
 0.0673, -0.3711,  0.8215, -0.2669, -0.1328,  0.2677
-0.9381,  0.4338,  0.7820, -0.9454,  0.0441,  0.5518
-0.3480,  0.7190,  0.1170,  0.3805, -0.0943,  0.4724
-0.9813,  0.1535, -0.3771,  0.0345,  0.8328,  0.5438
-0.1471, -0.5052, -0.2574,  0.8637,  0.8737,  0.3042
-0.5454, -0.3712, -0.6505,  0.2142, -0.1728,  0.5783
 0.6327, -0.6297,  0.4038, -0.5193,  0.1484,  0.1153
-0.5424,  0.3282, -0.0055,  0.0380, -0.6506,  0.6613
 0.1414,  0.9935,  0.6337,  0.1887,  0.9520,  0.2540
-0.9351, -0.8128, -0.8693, -0.0965, -0.2491,  0.7353
 0.9507, -0.6640,  0.9456,  0.5349,  0.6485,  0.1059
-0.0462, -0.9737, -0.2940, -0.0159,  0.4602,  0.2606
-0.0627, -0.0852, -0.7247, -0.9782,  0.5166,  0.2977
 0.0478,  0.5098, -0.0723, -0.7504, -0.3750,  0.3335
 0.0090,  0.3477,  0.5403, -0.7393, -0.9542,  0.4415
-0.9748,  0.3449,  0.3736, -0.1015,  0.8296,  0.4358
 0.2887, -0.9895, -0.0311,  0.7186,  0.6608,  0.2057
 0.1570, -0.4518,  0.1211,  0.3435, -0.2951,  0.3244
 0.7117, -0.6099,  0.4946, -0.4208,  0.5476,  0.1096
-0.2929, -0.5726,  0.5346, -0.3827,  0.4665,  0.2465
 0.4889, -0.5572, -0.5718, -0.6021, -0.7150,  0.2163
-0.7782,  0.3491,  0.5996, -0.8389, -0.5366,  0.6516
-0.5847,  0.8347,  0.4226,  0.1078, -0.3910,  0.6134
 0.8469,  0.4121, -0.0439, -0.7476,  0.9521,  0.1571
-0.6803, -0.5948, -0.1376, -0.1916, -0.7065,  0.7156
 0.2878,  0.5086, -0.5785,  0.2019,  0.4979,  0.2980
 0.2764,  0.1943, -0.4090,  0.4632,  0.8906,  0.2960
-0.8877,  0.6705, -0.6155, -0.2098, -0.3998,  0.7107
-0.8398,  0.8093, -0.2597,  0.0614, -0.0118,  0.6502
-0.8476,  0.0158, -0.4769, -0.2859, -0.7839,  0.7715
 0.5751, -0.7868,  0.9714, -0.6457,  0.1448,  0.1175
 0.4802, -0.7001,  0.1022, -0.5668,  0.5184,  0.1090
 0.4458, -0.6469,  0.7239, -0.9604,  0.7205,  0.0779
 0.5175,  0.4339,  0.9747, -0.4438, -0.9924,  0.2879
 0.8678,  0.7158,  0.4577,  0.0334,  0.4139,  0.1678
 0.5406,  0.5012,  0.2264, -0.1963,  0.3946,  0.2088
-0.9938,  0.5498,  0.7928, -0.5214, -0.7585,  0.7687
 0.7661,  0.0863, -0.4266, -0.7233, -0.4197,  0.1466
 0.2277, -0.3517, -0.0853, -0.1118,  0.6563,  0.1767
 0.3499, -0.5570, -0.0655, -0.3705,  0.2537,  0.1632
 0.7547, -0.1046,  0.5689, -0.0861,  0.3125,  0.1257
 0.8186,  0.2110,  0.5335,  0.0094, -0.0039,  0.1391
 0.6858, -0.8644,  0.1465,  0.8855,  0.0357,  0.1845
-0.4967,  0.4015,  0.0805,  0.8977,  0.2487,  0.4663
 0.6760, -0.9841,  0.9787, -0.8446, -0.3557,  0.1509
-0.1203, -0.4885,  0.6054, -0.0443, -0.7313,  0.4854
 0.8557,  0.7919, -0.0169,  0.7134, -0.1628,  0.2002
 0.0115, -0.6209,  0.9300, -0.4116, -0.7931,  0.4052
-0.7114, -0.9718,  0.4319,  0.1290,  0.5892,  0.3661
 0.3915,  0.5557, -0.1870,  0.2955, -0.6404,  0.2954
-0.3564, -0.6548, -0.1827, -0.5172, -0.1862,  0.4622
 0.2392, -0.4959,  0.5857, -0.1341, -0.2850,  0.2470
-0.3394,  0.3947, -0.4627,  0.6166, -0.4094,  0.5325
 0.7107,  0.7768, -0.6312,  0.1707,  0.7964,  0.2757
-0.1078,  0.8437, -0.4420,  0.2177,  0.3649,  0.4028
-0.3139,  0.5595, -0.6505, -0.3161, -0.7108,  0.5546
 0.4335,  0.3986,  0.3770, -0.4932,  0.3847,  0.1810
-0.2562, -0.2894, -0.8847,  0.2633,  0.4146,  0.4036
 0.2272,  0.2966, -0.6601, -0.7011,  0.0284,  0.2778
-0.0743, -0.1421, -0.0054, -0.6770, -0.3151,  0.3597
-0.4762,  0.6891,  0.6007, -0.1467,  0.2140,  0.4266
-0.4061,  0.7193,  0.3432,  0.2669, -0.7505,  0.6147
-0.0588,  0.9731,  0.8966,  0.2902, -0.6966,  0.4955
-0.0627, -0.1439,  0.1985,  0.6999,  0.5022,  0.3077
 0.1587,  0.8494, -0.8705,  0.9827, -0.8940,  0.4263
-0.7850,  0.2473, -0.9040, -0.4308, -0.8779,  0.7199
 0.4070,  0.3369, -0.2428, -0.6236,  0.4940,  0.2215
-0.0242,  0.0513, -0.9430,  0.2885, -0.2987,  0.3947
-0.5416, -0.1322, -0.2351, -0.0604,  0.9590,  0.3683
 0.1055,  0.7783, -0.2901, -0.5090,  0.8220,  0.2984
-0.9129,  0.9015,  0.1128, -0.2473,  0.9901,  0.4776
-0.9378,  0.1424, -0.6391,  0.2619,  0.9618,  0.5368
 0.7498, -0.0963,  0.4169,  0.5549, -0.0103,  0.1614
-0.2612, -0.7156,  0.4538, -0.0460, -0.1022,  0.3717
 0.7720,  0.0552, -0.1818, -0.4622, -0.8560,  0.1685
-0.4177,  0.0070,  0.9319, -0.7812,  0.3461,  0.3052
-0.0001,  0.5542, -0.7128, -0.8336, -0.2016,  0.3803
 0.5356, -0.4194, -0.5662, -0.9666, -0.2027,  0.1776
-0.2378,  0.3187, -0.8582, -0.6948, -0.9668,  0.5474
-0.1947, -0.3579,  0.1158,  0.9869,  0.6690,  0.2992
 0.3992,  0.8365, -0.9205, -0.8593, -0.0520,  0.3154
-0.0209,  0.0793,  0.7905, -0.1067,  0.7541,  0.1864
-0.4928, -0.4524, -0.3433,  0.0951, -0.5597,  0.6261
-0.8118,  0.7404, -0.5263, -0.2280,  0.1431,  0.6349
 0.0516, -0.8480,  0.7483,  0.9023,  0.6250,  0.1959
-0.3212,  0.1093,  0.9488, -0.3766,  0.3376,  0.2735
-0.3481,  0.5490, -0.3484,  0.7797,  0.5034,  0.4379
-0.5785, -0.9170, -0.3563, -0.9258,  0.3877,  0.4121
 0.3407, -0.1391,  0.5356,  0.0720, -0.9203,  0.3458
-0.3287, -0.8954,  0.2102,  0.0241,  0.2349,  0.3247
-0.1353,  0.6954, -0.0919, -0.9692,  0.7461,  0.3338
 0.9036, -0.8982, -0.5299, -0.8733, -0.1567,  0.1187
 0.7277, -0.8368, -0.0538, -0.7489,  0.5458,  0.0830
 0.9049,  0.8878,  0.2279,  0.9470, -0.3103,  0.2194
 0.7957, -0.1308, -0.5284,  0.8817,  0.3684,  0.2172
 0.4647, -0.4931,  0.2010,  0.6292, -0.8918,  0.3371
-0.7390,  0.6849,  0.2367,  0.0626, -0.5034,  0.7039
-0.1567, -0.8711,  0.7940, -0.5932,  0.6525,  0.1710
 0.7635, -0.0265,  0.1969,  0.0545,  0.2496,  0.1445
 0.7675,  0.1354, -0.7698, -0.5460,  0.1920,  0.1728
-0.5211, -0.7372, -0.6763,  0.6897,  0.2044,  0.5217
 0.1913,  0.1980,  0.2314, -0.8816,  0.5006,  0.1998
 0.8964,  0.0694, -0.6149,  0.5059, -0.9854,  0.1825
 0.1767,  0.7104,  0.2093,  0.6452,  0.7590,  0.2832
-0.3580, -0.7541,  0.4426, -0.1193, -0.7465,  0.5657
-0.5996,  0.5766, -0.9758, -0.3933, -0.9572,  0.6800
 0.9950,  0.1641, -0.4132,  0.8579,  0.0142,  0.2003
-0.4717, -0.3894, -0.2567, -0.5111,  0.1691,  0.4266
 0.3917, -0.8561,  0.9422,  0.5061,  0.6123,  0.1212
-0.0366, -0.1087,  0.3449, -0.1025,  0.4086,  0.2475
 0.3633,  0.3943,  0.2372, -0.6980,  0.5216,  0.1925
-0.5325, -0.6466, -0.2178, -0.3589,  0.6310,  0.3568
 0.2271,  0.5200, -0.1447, -0.8011, -0.7699,  0.3128
 0.6415,  0.1993,  0.3777, -0.0178, -0.8237,  0.2181
-0.5298, -0.0768, -0.6028, -0.9490,  0.4588,  0.4356
 0.6870, -0.1431,  0.7294,  0.3141,  0.1621,  0.1632
-0.5985,  0.0591,  0.7889, -0.3900,  0.7419,  0.2945
 0.3661,  0.7984, -0.8486,  0.7572, -0.6183,  0.3449
 0.6995,  0.3342, -0.3113, -0.6972,  0.2707,  0.1712
 0.2565,  0.9126,  0.1798, -0.6043, -0.1413,  0.2893
-0.3265,  0.9839, -0.2395,  0.9854,  0.0376,  0.4770
 0.2690, -0.1722,  0.9818,  0.8599, -0.7015,  0.3954
-0.2102, -0.0768,  0.1219,  0.5607, -0.0256,  0.3949
 0.8216, -0.9555,  0.6422, -0.6231,  0.3715,  0.0801
-0.2896,  0.9484, -0.7545, -0.6249,  0.7789,  0.4370
-0.9985, -0.5448, -0.7092, -0.5931,  0.7926,  0.5402

Test data:

# synthetic_test_40.txt
#
 0.7462,  0.4006, -0.0590,  0.6543, -0.0083,  0.1935
 0.8495, -0.2260, -0.0142, -0.4911,  0.7699,  0.1078
-0.2335, -0.4049,  0.4352, -0.6183, -0.7636,  0.5088
 0.1810, -0.5142,  0.2465,  0.2767, -0.3449,  0.3136
-0.8650,  0.7611, -0.0801,  0.5277, -0.4922,  0.7140
-0.2358, -0.7466, -0.5115, -0.8413, -0.3943,  0.4533
 0.4834,  0.2300,  0.3448, -0.9832,  0.3568,  0.1360
-0.6502, -0.6300,  0.6885,  0.9652,  0.8275,  0.3046
-0.3053,  0.5604,  0.0929,  0.6329, -0.0325,  0.4756
-0.7995,  0.0740, -0.2680,  0.2086,  0.9176,  0.4565
-0.2144, -0.2141,  0.5813,  0.2902, -0.2122,  0.4119
-0.7278, -0.0987, -0.3312, -0.5641,  0.8515,  0.4438
 0.3793,  0.1976,  0.4933,  0.0839,  0.4011,  0.1905
-0.8568,  0.9573, -0.5272,  0.3212, -0.8207,  0.7415
-0.5785,  0.0056, -0.7901, -0.2223,  0.0760,  0.5551
 0.0735, -0.2188,  0.3925,  0.3570,  0.3746,  0.2191
 0.1230, -0.2838,  0.2262,  0.8715,  0.1938,  0.2878
 0.4792, -0.9248,  0.5295,  0.0366, -0.9894,  0.3149
-0.4456,  0.0697,  0.5359, -0.8938,  0.0981,  0.3879
 0.8629, -0.8505, -0.4464,  0.8385,  0.5300,  0.1769
 0.1995,  0.6659,  0.7921,  0.9454,  0.9970,  0.2330
-0.0249, -0.3066, -0.2927, -0.4923,  0.8220,  0.2437
 0.4513, -0.9481, -0.0770, -0.4374, -0.9421,  0.2879
-0.3405,  0.5931, -0.3507, -0.3842,  0.8562,  0.3987
 0.9538,  0.0471,  0.9039,  0.7760,  0.0361,  0.1706
-0.0887,  0.2104,  0.9808,  0.5478, -0.3314,  0.4128
-0.8220, -0.6302,  0.0537, -0.1658,  0.6013,  0.4306
-0.4123, -0.2880,  0.9074, -0.0461, -0.4435,  0.5144
 0.0060,  0.2867, -0.7775,  0.5161,  0.7039,  0.3599
-0.7968, -0.5484,  0.9426, -0.4308,  0.8148,  0.2979
 0.7811,  0.8450, -0.6877,  0.7594,  0.2640,  0.2362
-0.6802, -0.1113, -0.8325, -0.6694, -0.6056,  0.6544
 0.3821,  0.1476,  0.7466, -0.5107,  0.2592,  0.1648
 0.7265,  0.9683, -0.9803, -0.4943, -0.5523,  0.2454
-0.9049, -0.9797, -0.0196, -0.9090, -0.4433,  0.6447
-0.4607,  0.1811, -0.2389,  0.4050, -0.0078,  0.5229
 0.2664, -0.2932, -0.4259, -0.7336,  0.8742,  0.1834
-0.4507,  0.1029, -0.6294, -0.1158, -0.6294,  0.6081
 0.8948, -0.0124,  0.9278,  0.2899, -0.0314,  0.1534
-0.1323, -0.8813, -0.0146, -0.0697,  0.6135,  0.2386

I’ve been working on a book tentatively titled “Machine Learning Regression Using Classical Techniques With C#”. I intend to cover five classical (non-tree-based) regression techniques: linear regression, nearest neighbors regression, quadratic regression, kernel ridge regression, and neural network regression.

Note: I’m explicitly ignoring support vector regression which, in my opinion, is a solution-in-search-of-a-problem intellectual abomination. But that’s another story.

For my introductory chapter, I want to present a canonical example of regression. I scanned my brain for the simplest possible regression technique to use as the example. I have decades of experience with machine learning and I figured there was some obscure, super-simple regression technique, but a technique that illustrates all, or at least most, of the key ideas of machine learning regression using the C# language.

I was surprised when I couldn’t find anything simpler than standard linear regression or nearest neighbors regression. So, I put together what I consider a canonical demo using linear regression.

The demo uses training data that has known correct input predictors (sometimes called features) and target y values. The 12-item raw training data is:

age income   debt      balance
-------------------------------
24,  medium, $2325.00, $1972.00
33,  high,   $3140.00, $2324.00
44,  low,    $1425.00, $2284.00
38,  medium, $3000.00, $2606.00
41,  high,   $2833.00, $3193.00
57,  high,   $3875.00, $2709.00
25,  low,    $1974.00, $2109.00
29,  high,   $3900.00, $3209.00
57,  medium, $3642.00, $2799.00
64,  low,    $1525.00, $2708.00
26,  high,   $3625.00, $2406.00
38,  high,   $3400.00, $2634.00

The goal is to predict bank account balance from age, income and debt.

The numeric predictors (age, debt) are normalized so they’re all between 0.0 and 1.0. The categorical predictor (income) is encoded to equal-interval numeric values. The 12-item dataset is randomly split into an 8-item training set and a 4-item test set:

age   income  debt  balance
---------------------------
0.24, 0.50, 0.2325, 0.1972
0.57, 0.75, 0.3875, 0.2709
0.38, 0.50, 0.3000, 0.2606
0.29, 0.75, 0.3900, 0.3209
0.44, 0.25, 0.1425, 0.2284
0.64, 0.25, 0.1525, 0.2708
0.26, 0.75, 0.3625, 0.2406
0.38, 0.75, 0.3400, 0.2634

age   income  debt  balance
---------------------------
0.33, 0.75, 0.3140, 0.2324
0.41, 0.75, 0.2833, 0.3193
0.25, 0.25, 0.1974, 0.2109
0.57, 0.50, 0.3642, 0.2799

The output of the demo regression program is:

Begin predict bank balance from age, income, debt

Setting up train and test data
Done

First two train X:
   0.2400   0.5000   0.2325
   0.5700   0.7500   0.3875

First two train y:
  0.1972
  0.2709

Creating linear regression model
Done

Setting lrnRate = 0.01, maxEpochs = 100
Starting training using SGD
epoch =     0  MSE =   0.0011
epoch =   200  MSE =   0.0007
epoch =   400  MSE =   0.0007
epoch =   600  MSE =   0.0007
epoch =   800  MSE =   0.0007
Done

Weights/coefficients:
0.0999  -0.0708  0.3717
Bias/constant: 0.1487

Evaluating model
Accuracy train (within 0.10) = 0.8750
Accuracy test (within 0.10) = 0.7500

Predicting for 35  medium  $19,000
predicted y = $2188.64

End demo

The demo output illustrates nine key ideas for machine learning regression using C#. They are: 1.) vectors and matrices, 2.) training and test data, 3.) data normalization and encoding, 4.) different regression techniques, 5.) different training algorithms, 6.) model and training hyperparameters, 7.) model interpretability, 8.) model evaluation, and 9.) using a trained regression model to make a prediction.

A good thought experiment.

I find great beauty in mathematics and machine learning. I’m not so good at recognizing beauty in Art. But here are three photographs by Valeria Lokinskaya that combine math and art that I think are beautiful. (I’m not sure if the Asian characters in the middle photo are numbers are not, but they look nice to me).

Demo program. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols.

using System;
using System.IO;

namespace IntroductionToRegression
{
  internal class IntroductionToRegressionProgram
  {
    static void Main(string[] args)
    {
      Console.WriteLine("\nBegin predict bank balance" +
        " from age, income, debt ");

      Console.WriteLine("\nSetting up train and test data ");
      double[][] trainX = new double[8][];
      trainX[0] = new double[] { 0.24, 0.50, 0.2325 };
      trainX[1] = new double[] { 0.57, 0.75, 0.3875 };
      trainX[2] = new double[] { 0.38, 0.50, 0.3000 };
      trainX[3] = new double[] { 0.29, 0.75, 0.3900 };
      trainX[4] = new double[] { 0.44, 0.25, 0.1425 };
      trainX[5] = new double[] { 0.64, 0.25, 0.1525 };
      trainX[6] = new double[] { 0.26, 0.75, 0.3625 };
      trainX[7] = new double[] { 0.38, 0.75, 0.3400 };

      double[] trainY = new double[] { 0.1972, 0.2709,
        0.2606, 0.3209, 0.2284, 0.2708, 0.2406, 0.2634 };

      double[][] testX = new double[4][];
      testX[0] = new double[] { 0.33, 0.75, 0.3140 };
      testX[1] = new double[] { 0.41, 0.75, 0.2833 };
      testX[2] = new double[] { 0.25, 0.25, 0.1974 };
      testX[3] = new double[] { 0.57, 0.50, 0.3642 };

      double[] testY = new double[] { 0.2324, 0.3193,
        0.2109, 0.2799 };
      Console.WriteLine("Done ");

      Console.WriteLine("\nFirst two train X: ");
      for (int i = 0; i "lt" 2; ++i)
      {
        for (int j = 0; j "lt" trainX[0].Length; ++j)
          Console.Write(trainX[i][j].ToString("F4").
            PadLeft(9));
        Console.WriteLine("");
      }

      Console.WriteLine("\nFirst two train y: ");
      for (int i = 0; i "lt" 2; ++i)
        Console.WriteLine(trainY[i].ToString("F4").
          PadLeft(8));

      Console.WriteLine("\nCreating linear regression model ");
      LinearRegressor model = new LinearRegressor(seed: 0);
      Console.WriteLine("Done ");

      Console.WriteLine("\nSetting lrnRate = 0.01," +
        " maxEpochs = 100 ");
      Console.WriteLine("Starting training using SGD ");
      model.TrainSGD(trainX, trainY, 0.20, 1000);
      Console.WriteLine("Done ");

      Console.WriteLine("\nWeights/coefficients: ");
      for (int i = 0; i "lt" model.weights.Length; ++i)
        Console.Write(model.weights[i].ToString("F4") + "  ");
      Console.WriteLine("\nBias/constant: " +
        model.bias.ToString("F4"));

      Console.WriteLine("\nEvaluating model ");
      double accTrain = model.Accuracy(trainX, trainY, 0.15);
      Console.WriteLine("Accuracy train (within 0.15) = " +
        accTrain.ToString("F4"));
      double accTest = model.Accuracy(testX, testY, 0.15);
      Console.WriteLine("Accuracy test (within 0.15) = " +
        accTest.ToString("F4"));

      Console.WriteLine("\nPredicting for 35" +
        "  medium  $19,000 ");
      double[] x = new double[] { 0.35, 0.50, 0.19000 };
      double predY = model.Predict(x);
      Console.WriteLine("predicted y = $" + 
        (predY * 10000).ToString("F2"));

      Console.WriteLine("\nEnd demo ");
      Console.ReadLine();

    } // Main

    public class LinearRegressor
    {
      public double[] weights;
      public double bias;
      private Random rnd;

      public LinearRegressor(int seed = 0)
      {
        this.weights = new double[0]; // quasi-null
        this.bias = 0.0;
        this.rnd = new Random(seed);
      }

      // ------------------------------------------------------

      public void TrainSGD(double[][] trainX, double[] trainY,
        double lrnRate, int maxEpochs)
      {
        int n = trainX.Length;
        int dim = trainX[0].Length;
        this.weights = new double[dim];
        double lo = -0.01; double hi = 0.01;
        for (int i = 0; i "lt" dim; ++i)
          this.weights[i] = (hi - lo) *
            this.rnd.NextDouble() + lo;
        this.bias = (hi - lo) *
            this.rnd.NextDouble() + lo;

        int[] indices = new int[n];
        for (int i = 0; i "lt" n; ++i)
          indices[i] = i;

        for (int epoch = 0; epoch "lt" maxEpochs; ++epoch)
        {
          // shuffle indices
          for (int i = 0; i "lt" n; ++i)
          {
            int ri = rnd.Next(i, n);
            int tmp = indices[i];
            indices[i] = indices[ri];
            indices[ri] = tmp;
          }
          for (int i = 0; i "lt" n; ++i)
          {
            int ii = indices[i];
            double[] x = trainX[ii];
            double actualY = trainY[ii];
            double predY = this.Predict(x);

            for (int j = 0; j "lt" dim; ++j)
              this.weights[j] -= lrnRate *
                (predY - actualY) * x[j];
            this.bias -= lrnRate * (predY - actualY);
          }
          if (epoch % (int)(maxEpochs / 5) == 0)
          {
            double mse = this.MSE(trainX, trainY);
            string s1 = "epoch = " + epoch.ToString().
              PadLeft(5);
            string s2 = "  MSE = " + mse.ToString("F4").
              PadLeft(8);
            Console.WriteLine(s1 + s2);
          }
        }
      }

      // ------------------------------------------------------

      public double Predict(double[] x)
      {
        double result = 0.0;
        for (int j = 0; j "lt" x.Length; ++j)
          result += x[j] * this.weights[j];
        result += this.bias;
        return result;
      }

      // ------------------------------------------------------

      public double Accuracy(double[][] dataX, double[] dataY,
        double pctClose)
      {
        int numCorrect = 0; int numWrong = 0;
        for (int i = 0; i "lt" dataX.Length; ++i)
        {
          double actualY = dataY[i];
          double predY = this.Predict(dataX[i]);
          if (Math.Abs(predY - actualY) "lt" 
            Math.Abs(pctClose * actualY))
          {
            ++numCorrect;
          }
          else
          {
            ++numWrong;
          }
        }
        return (numCorrect * 1.0) / (numWrong + numCorrect);
      }

      // ------------------------------------------------------

      public double MSE(double[][] dataX, double[] dataY)
      {
        int n = dataX.Length;
        double sum = 0.0;
        for (int i = 0; i "lt" n; ++i)
        {
          double actualY = dataY[i];
          double predY = this.Predict(dataX[i]);
          sum += (actualY - predY) * (actualY - predY);
        }
        return sum / n;
      }

      // ------------------------------------------------------

    } // class LinearRegressor
  } // Program
} // ns

Bottom Line: Auto-stopping for stochastic gradient descent training is great in theory, but, in my opinion, having to set the stopping tolerance constant (which depends on how the training data is scaled) and the consecutive-no-improve constant (which is arbitrary), outweigh the benefits of auto-stopping in many (but not all) scenarios.

The goal of a machine learning regression problem is to predict a single numeric value. Many regression techniques create a mathematical prediction equation that has constants called weights. Finding the values of the model weights is called training the model. One common form of training is called stochastic gradient descent (SGD). SGD is iterative where you must specify how mant times to iterate (max epochs) and how much to adjust the weights in each update (learn rate).

If you add auto-stopping to SGD training, the code checks the mean square error (MSE) of the current model after each epoch (one complete pass through the training data). If the current model has not significantly improved (measured using MSE with a small constant called a stop tolerance) for n consecutive epochs, then the training stops early, the idea being that the model is not improving.

Note: The term “early stopping” is sometimes synonymous with “auto-stopping”, but early stopping usually applies to a set of validation data instead of training data.

This sounds great in theory. For some scenarios, SGD auto-stop is a good idea. But in most of the scenarios I work with, the difficulties of specifying the stop tolerance and the number of consecutive no-improves required to exit, outweigh the benefit of an auto-stop.

First, consider determining if there is no improvement in MSE. The standard way to do this is to keep track of the best MSE found. Then the condition (currMSE “gt” bestMSE) indicates extreme no improvement, in fact, the current model is worse. But it’s common to relax this consition a bit to make it a bit easier to have no improvement: (currMSE “gt” bestMSE – a_little_bit). This is surprisingly subtle.

If the a_little_bit is called the stopping tolerance, the problem becomes specifying it.

The smallest possible tolerance is 0, in which case it is realtively easier to improve, which means it is relatively difficult to auto-stop, which leads to a more accurate prediction model, but one that is prone to overfitting.

If the tolerance is relatively large, like 0.01, then it is more diffiuclt to improve, easier to auto-stop, which leads to a model that predicts more poorly on training data but less prone to overfitting.

Unfortunately, MSE depends on how the training data is scaled, which means that specifying a good stopping tolerance dependes on the data. Suppose a predictor variable age has values (33, 51, 27, . . .) and some model MSE is 0.4567. But if the age values are scaled by dividing by 100 to (0.33, 0.51, 0.27 . . .) then the MSE on the same model will be divided by 100: 0.0045.

A second problem is specifying how many consecutive times there is no improvement that are required to indicate no progress and therefore trigger an auto-stop. The default value in the scikit SGDRegressor module is 5, but that is purely arbitrary.

I coded up a demo of auto-stopping for a linear regression model implemented using C#, to illustrate the ideas. The output is:

Begin C# linear regression using SGD
 training with auto-stopping

Loading synthetic train (200) and
 test (40) data
Done

First three train X:
 -0.1660  0.4406 -0.9998 -0.3953 -0.7065
  0.0776 -0.1616  0.3704 -0.5911  0.7562
 -0.9452  0.3409 -0.1654  0.1174 -0.7192

First three train y:
  0.4840
  0.1568
  0.8054

Setting lrnRate = 0.0010
Setting maxEpochs = 500
Setting stopTol = 1.000000e-004

Creating and training Linear Regression model
epoch =     0  MSE =   0.1132
epoch =    25  MSE =   0.0035
epoch =    50  MSE =   0.0026
epoch =    75  MSE =   0.0026
epoch =   100  MSE =   0.0026
Auto stop at epoch 118
Done

Weights/coefficients:
-0.2655  0.0333  -0.0455  0.0358  -0.1146
Bias/constant: 0.3620

Evaluating model

Acc train (within 0.10) = 0.4600
Acc test (within 0.10) = 0.6250

MSE train = 0.0026
MSE test = 0.0020

Predicting for x =
  -0.1660   0.4406  -0.9998  -0.3953  -0.7065

Predicted y = 0.5330

End demo

I used synthetic data. I hard-coded the consecutive-no-improves to 5.

An interesting way to spend some time on a Saturday afternoon.

The concept of auto-stopping for stochastic gradient descent is an interesting topic. Videos of automobile traffic stops can be very interesting too. A minority percentage of the U.S. population think that laws and civilized behavior don’t apply to them. Here are three examples where a simple traffic stop plus failure to follow requests for ID and insurance, turned into felony arrests.

Demo program. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols. (My blog editor chokes on symbols).

using System;
using System.IO;
using System.Collections.Generic;

// SGD training with auto-stopping

namespace LinearRegressionStochasticStopping
{
  internal class LinearRegressionProgram
  {
    static void Main(string[] args)
    {
      Console.WriteLine("\nBegin C# linear regression " +
        "using SGD training with auto-stopping ");

      // 1. load data
      Console.WriteLine("\nLoading synthetic train" +
        " (200) and test (40) data");
      string trainFile =
        "..\\..\\..\\Data\\synthetic_train_200.txt";
      int[] colsX = new int[] { 0, 1, 2, 3, 4 };
      double[][] trainX =
        MatLoad(trainFile, colsX, ',', "#");
      double[] trainY =
        MatToVec(MatLoad(trainFile,
        new int[] { 5 }, ',', "#"));

      string testFile =
        "..\\..\\..\\Data\\synthetic_test_40.txt";
      double[][] testX =
        MatLoad(testFile, colsX, ',', "#");
      double[] testY =
        MatToVec(MatLoad(testFile,
        new int[] { 5 }, ',', "#"));
      Console.WriteLine("Done ");

      Console.WriteLine("\nFirst three train X: ");
      for (int i = 0; i "lt" 3; ++i)
        VecShow(trainX[i], 4, 8);

      Console.WriteLine("\nFirst three train y: ");
      for (int i = 0; i "lt" 3; ++i)
        Console.WriteLine(trainY[i].ToString("F4").
          PadLeft(8));

      // 2. create and train model
      double lrnRate = 0.001;
      int maxEpochs = 500;
      double stopTol = 1.0e-4;
      // very small stopTol (like 1.0e-8) = 
      // easy to improve = hard to stop = better acc.
      // large stopTol (like 0.0) =
      // hard to improve = easy to stop = worse acc.
      
      Console.WriteLine("\nSetting lrnRate = " +
        lrnRate.ToString("F4"));
      Console.WriteLine("Setting maxEpochs = " +
        maxEpochs);
      Console.WriteLine("Setting stopTol = " +
        stopTol.ToString("e"));

      Console.WriteLine("\nCreating and training" +
        " Linear Regression model ");
      LinearRegressor model =
        new LinearRegressor();
      model.TrainSGD(trainX, trainY,
        lrnRate, maxEpochs, stopTol);
      Console.WriteLine("Done ");

      // 2b.show model parameters
      Console.WriteLine("\nWeights/coefficients: ");
      for (int i = 0; i "lt" model.weights.Length; ++i)
        Console.Write(model.weights[i].
          ToString("F4") + "  ");
      Console.WriteLine("\nBias/constant: " +
        model.bias.ToString("F4"));

      // 3. evaluate model
      Console.WriteLine("\nEvaluating model ");

      double accTrain = model.Accuracy(trainX, trainY,
        0.10);
      Console.WriteLine("\nAcc train (within 0.10) = " +
        accTrain.ToString("F4"));
      double accTest = model.Accuracy(testX, testY,
        0.10);
      Console.WriteLine("Acc test (within 0.10) = " +
        accTest.ToString("F4"));

      double mseTrain = model.MSE(trainX, trainY);
      Console.WriteLine("\nMSE train = " +
        mseTrain.ToString("F4"));
      double mseTest = model.MSE(testX, testY);
      Console.WriteLine("MSE test = " +
        mseTest.ToString("F4"));

      // 4. use model
      double[] x = trainX[0];
      Console.WriteLine("\nPredicting for x = ");
      VecShow(x, 4, 9);
      double predY = model.Predict(x);
      Console.WriteLine("\nPredicted y = " +
        predY.ToString("F4"));

      Console.WriteLine("\nEnd demo ");
      Console.ReadLine();
    } // Main

    // ------------------------------------------------------
    // helpers for Main()
    // ------------------------------------------------------

    static double[][] MatLoad(string fn, int[] usecols,
      char sep, string comment)
    {
      List"lt"double[]"gt" result = 
        new List"lt"double[]"gt"();
      string line = "";
      FileStream ifs = new FileStream(fn, FileMode.Open);
      StreamReader sr = new StreamReader(ifs);
      while ((line = sr.ReadLine()) != null)
      {
        if (line.StartsWith(comment) == true)
          continue;
        string[] tokens = line.Split(sep);
        List"lt"double"gt" lst = new List"lt"double"gt"();
        for (int j = 0; j "lt" usecols.Length; ++j)
          lst.Add(double.Parse(tokens[usecols[j]]));
        double[] row = lst.ToArray();
        result.Add(row);
      }
      sr.Close(); ifs.Close();
      return result.ToArray();
    }

    static double[] MatToVec(double[][] M)
    {
      int nRows = M.Length;
      int nCols = M[0].Length;
      double[] result = new double[nRows * nCols];
      int k = 0;
      for (int i = 0; i "lt" nRows; ++i)
        for (int j = 0; j "lt" nCols; ++j)
          result[k++] = M[i][j];
      return result;
    }

    static void VecShow(double[] vec, int dec, int wid)
    {
      for (int i = 0; i "lt" vec.Length; ++i)
        Console.Write(vec[i].ToString("F" + dec).
          PadLeft(wid));
      Console.WriteLine("");
    }
  } // class Program

  public class LinearRegressor
  {
    public double[] weights;
    public double bias;
    private Random rnd;

    public LinearRegressor(int seed = 0)
    {
      this.weights = new double[0]; // keep compiler happy
      this.bias = 0;
      this.rnd = new Random(seed);
    }

    public void TrainSGD(double[][] trainX,
      double[] trainY, double lrnRate, int maxEpochs, 
      double stopTol)
    {
      // auto stop if MSE does not improve over best
      // found MSE for 5 consecutive epochs,
      // where improve has stopTol

      int n = trainX.Length;  int dim = trainX[0].Length;
      this.weights = new double[dim];

      // initialize weights and bias
      double low = -0.01; double hi = 0.01;
      for (int i = 0; i "lt" dim; ++i)
        this.weights[i] = (hi - low) *
          this.rnd.NextDouble() + low;
      this.bias = (hi - low) *
          this.rnd.NextDouble() + low;

      int[] indices = new int[n];  // of train data
      for (int i = 0; i "lt" n; ++i)
        indices[i] = i;
      
      // small tol: easier to improve = harder to stop
      // bigger tol: harder to improve = easier to stop
      double bestMSE = double.MaxValue;
      int consecNoImprove = 0;

      for (int epoch = 0; epoch "lt" maxEpochs; ++epoch)
      {
        // shuffle order pf training data indices
        for (int i = 0; i "lt" indices.Length; ++i)
        {
          int ri = this.rnd.Next(i, indices.Length);
          int tmp = indices[i];
          indices[i] = indices[ri];
          indices[ri] = tmp;
        }

        for (int i = 0; i "lt" n; ++i) // each train item
        {
          int ii = indices[i];
          double[] x = trainX[ii];
          double predY = this.Predict(x);
          double actualY = trainY[ii];
          for (int j = 0; j "lt" dim; ++j) // each weight
            this.weights[j] -= lrnRate *
              (predY - actualY) * x[j];
          this.bias -= lrnRate * (predY - actualY);
        }

        //Console.WriteLine("epoch " + epoch);

        double currMSE = this.MSE(trainX, trainY);

        //Console.WriteLine("curr MSE = " +
        //  currMSE.ToString("F8"));
        if (currMSE "lt" bestMSE)
        {
          bestMSE = currMSE;
          //Console.WriteLine("New best MSE " + 
          //  bestMSE.ToString("F8") + " at epoch " + epoch);
          consecNoImprove = 0;
        }

        // either reset consecutive counter or increment it
        // currMSE "gt" bestMSE is extreme no improvement
        // but make it a bit easier for no improve:
        // currMSE "gt" bestMSE - a_little_bit
        // curr_loss "gt" best_loss - tol means no signif improve
        
        if (currMSE "gt" bestMSE - stopTol) // no sig improve
        {
          //Console.WriteLine("no significant improvement" +
          //  " at epoch " + epoch);
          ++consecNoImprove;
        }
        else // improved
        {
          //Console.WriteLine("some improvement at" +
          //  " epoch " + epoch);
          consecNoImprove = 0;
        }

        if (consecNoImprove == 5)
        {
          Console.WriteLine("Auto stop at epoch " + epoch);
          break;
        }

        if (epoch % (int)(maxEpochs / 20) == 0) // progress
        {
          double mse = this.MSE(trainX, trainY);
          string s = "";
          s += "epoch = " + epoch.ToString().PadLeft(5);
          s += "  MSE = " + mse.ToString("F4").PadLeft(8);
          Console.WriteLine(s);
        }

      } // each epoch



    } // Train

    // ------------------------------------------------------

    public double Predict(double[] x)
    {
      double result = 0.0;
      for (int j = 0; j "lt" x.Length; ++j)
        result += x[j] * this.weights[j];
      result += this.bias;
      return result;
    }

    // ------------------------------------------------------

    public double Accuracy(double[][] dataX, double[] dataY,
      double pctClose)
    {
      int numCorrect = 0; int numWrong = 0;
      for (int i = 0; i "lt" dataX.Length; ++i)
      {
        double actualY = dataY[i];
        double predY = this.Predict(dataX[i]);
        if (Math.Abs(predY - actualY) "lt"
          (pctClose * actualY))
          ++numCorrect;
        else
          ++numWrong;
      }
      return (numCorrect * 1.0) / (numWrong + numCorrect);
    }

    // ------------------------------------------------------

    public double MSE(double[][] dataX, double[] dataY)
    {
      int n = dataX.Length;
      double sum = 0.0;
      for (int i = 0; i "lt" n; ++i)
      {
        double actualY = dataY[i];
        double predY = this.Predict(dataX[i]);
        sum += (actualY - predY) * (actualY - predY);
      }
      return sum / n;
    }

  } // class LinearRegressor

} // ns

Training data:

# synthetic_train_200.txt
#
-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
-0.8299, -0.9219, -0.6603,  0.7563, -0.8033,  0.7955
 0.0663,  0.3838, -0.3690,  0.3730,  0.6693,  0.3206
-0.9634,  0.5003,  0.9777,  0.4963, -0.4391,  0.7377
-0.1042,  0.8172, -0.4128, -0.4244, -0.7399,  0.4801
-0.9613,  0.3577, -0.5767, -0.4689, -0.0169,  0.6861
-0.7065,  0.1786,  0.3995, -0.7953, -0.1719,  0.5569
 0.3888, -0.1716, -0.9001,  0.0718,  0.3276,  0.2500
 0.1731,  0.8068, -0.7251, -0.7214,  0.6148,  0.3297
-0.2046, -0.6693,  0.8550, -0.3045,  0.5016,  0.2129
 0.2473,  0.5019, -0.3022, -0.4601,  0.7918,  0.2613
-0.1438,  0.9297,  0.3269,  0.2434, -0.7705,  0.5171
 0.1568, -0.1837, -0.5259,  0.8068,  0.1474,  0.3307
-0.9943,  0.2343, -0.3467,  0.0541,  0.7719,  0.5581
 0.2467, -0.9684,  0.8589,  0.3818,  0.9946,  0.1092
-0.6553, -0.7257,  0.8652,  0.3936, -0.8680,  0.7018
 0.8460,  0.4230, -0.7515, -0.9602, -0.9476,  0.1996
-0.9434, -0.5076,  0.7201,  0.0777,  0.1056,  0.5664
 0.9392,  0.1221, -0.9627,  0.6013, -0.5341,  0.1533
 0.6142, -0.2243,  0.7271,  0.4942,  0.1125,  0.1661
 0.4260,  0.1194, -0.9749, -0.8561,  0.9346,  0.2230
 0.1362, -0.5934, -0.4953,  0.4877, -0.6091,  0.3810
 0.6937, -0.5203, -0.0125,  0.2399,  0.6580,  0.1460
-0.6864, -0.9628, -0.8600, -0.0273,  0.2127,  0.5387
 0.9772,  0.1595, -0.2397,  0.1019,  0.4907,  0.1611
 0.3385, -0.4702, -0.8673, -0.2598,  0.2594,  0.2270
-0.8669, -0.4794,  0.6095, -0.6131,  0.2789,  0.4700
 0.0493,  0.8496, -0.4734, -0.8681,  0.4701,  0.3516
 0.8639, -0.9721, -0.5313,  0.2336,  0.8980,  0.1412
 0.9004,  0.1133,  0.8312,  0.2831, -0.2200,  0.1782
 0.0991,  0.8524,  0.8375, -0.2102,  0.9265,  0.2150
-0.6521, -0.7473, -0.7298,  0.0113, -0.9570,  0.7422
 0.6190, -0.3105,  0.8802,  0.1640,  0.7577,  0.1056
 0.6895,  0.8108, -0.0802,  0.0927,  0.5972,  0.2214
 0.1982, -0.9689,  0.1870, -0.1326,  0.6147,  0.1310
-0.3695,  0.7858,  0.1557, -0.6320,  0.5759,  0.3773
-0.1596,  0.3581,  0.8372, -0.9992,  0.9535,  0.2071
-0.2468,  0.9476,  0.2094,  0.6577,  0.1494,  0.4132
 0.1737,  0.5000,  0.7166,  0.5102,  0.3961,  0.2611
 0.7290, -0.3546,  0.3416, -0.0983, -0.2358,  0.1332
-0.3652,  0.2438, -0.1395,  0.9476,  0.3556,  0.4170
-0.6029, -0.1466, -0.3133,  0.5953,  0.7600,  0.4334
-0.4596, -0.4953,  0.7098,  0.0554,  0.6043,  0.2775
 0.1450,  0.4663,  0.0380,  0.5418,  0.1377,  0.2931
-0.8636, -0.2442, -0.8407,  0.9656, -0.6368,  0.7429
 0.6237,  0.7499,  0.3768,  0.1390, -0.6781,  0.2185
-0.5499,  0.1850, -0.3755,  0.8326,  0.8193,  0.4399
-0.4858, -0.7782, -0.6141, -0.0008,  0.4572,  0.4197
 0.7033, -0.1683,  0.2334, -0.5327, -0.7961,  0.1776
 0.0317, -0.0457, -0.6947,  0.2436,  0.0880,  0.3345
 0.5031, -0.5559,  0.0387,  0.5706, -0.9553,  0.3107
-0.3513,  0.7458,  0.6894,  0.0769,  0.7332,  0.3170
 0.2205,  0.5992, -0.9309,  0.5405,  0.4635,  0.3532
-0.4806, -0.4859,  0.2646, -0.3094,  0.5932,  0.3202
 0.9809, -0.3995, -0.7140,  0.8026,  0.0831,  0.1600
 0.9495,  0.2732,  0.9878,  0.0921,  0.0529,  0.1289
-0.9476, -0.6792,  0.4913, -0.9392, -0.2669,  0.5966
 0.7247,  0.3854,  0.3819, -0.6227, -0.1162,  0.1550
-0.5922, -0.5045, -0.4757,  0.5003, -0.0860,  0.5863
-0.8861,  0.0170, -0.5761,  0.5972, -0.4053,  0.7301
 0.6877, -0.2380,  0.4997,  0.0223,  0.0819,  0.1404
 0.9189,  0.6079, -0.9354,  0.4188, -0.0700,  0.1907
-0.1428, -0.7820,  0.2676,  0.6059,  0.3936,  0.2790
 0.5324, -0.3151,  0.6917, -0.1425,  0.6480,  0.1071
-0.8432, -0.9633, -0.8666, -0.0828, -0.7733,  0.7784
-0.9444,  0.5097, -0.2103,  0.4939, -0.0952,  0.6787
-0.0520,  0.6063, -0.1952,  0.8094, -0.9259,  0.4836
 0.5477, -0.7487,  0.2370, -0.9793,  0.0773,  0.1241
 0.2450,  0.8116,  0.9799,  0.4222,  0.4636,  0.2355
 0.8186, -0.1983, -0.5003, -0.6531, -0.7611,  0.1511
-0.4714,  0.6382, -0.3788,  0.9648, -0.4667,  0.5950
 0.0673, -0.3711,  0.8215, -0.2669, -0.1328,  0.2677
-0.9381,  0.4338,  0.7820, -0.9454,  0.0441,  0.5518
-0.3480,  0.7190,  0.1170,  0.3805, -0.0943,  0.4724
-0.9813,  0.1535, -0.3771,  0.0345,  0.8328,  0.5438
-0.1471, -0.5052, -0.2574,  0.8637,  0.8737,  0.3042
-0.5454, -0.3712, -0.6505,  0.2142, -0.1728,  0.5783
 0.6327, -0.6297,  0.4038, -0.5193,  0.1484,  0.1153
-0.5424,  0.3282, -0.0055,  0.0380, -0.6506,  0.6613
 0.1414,  0.9935,  0.6337,  0.1887,  0.9520,  0.2540
-0.9351, -0.8128, -0.8693, -0.0965, -0.2491,  0.7353
 0.9507, -0.6640,  0.9456,  0.5349,  0.6485,  0.1059
-0.0462, -0.9737, -0.2940, -0.0159,  0.4602,  0.2606
-0.0627, -0.0852, -0.7247, -0.9782,  0.5166,  0.2977
 0.0478,  0.5098, -0.0723, -0.7504, -0.3750,  0.3335
 0.0090,  0.3477,  0.5403, -0.7393, -0.9542,  0.4415
-0.9748,  0.3449,  0.3736, -0.1015,  0.8296,  0.4358
 0.2887, -0.9895, -0.0311,  0.7186,  0.6608,  0.2057
 0.1570, -0.4518,  0.1211,  0.3435, -0.2951,  0.3244
 0.7117, -0.6099,  0.4946, -0.4208,  0.5476,  0.1096
-0.2929, -0.5726,  0.5346, -0.3827,  0.4665,  0.2465
 0.4889, -0.5572, -0.5718, -0.6021, -0.7150,  0.2163
-0.7782,  0.3491,  0.5996, -0.8389, -0.5366,  0.6516
-0.5847,  0.8347,  0.4226,  0.1078, -0.3910,  0.6134
 0.8469,  0.4121, -0.0439, -0.7476,  0.9521,  0.1571
-0.6803, -0.5948, -0.1376, -0.1916, -0.7065,  0.7156
 0.2878,  0.5086, -0.5785,  0.2019,  0.4979,  0.2980
 0.2764,  0.1943, -0.4090,  0.4632,  0.8906,  0.2960
-0.8877,  0.6705, -0.6155, -0.2098, -0.3998,  0.7107
-0.8398,  0.8093, -0.2597,  0.0614, -0.0118,  0.6502
-0.8476,  0.0158, -0.4769, -0.2859, -0.7839,  0.7715
 0.5751, -0.7868,  0.9714, -0.6457,  0.1448,  0.1175
 0.4802, -0.7001,  0.1022, -0.5668,  0.5184,  0.1090
 0.4458, -0.6469,  0.7239, -0.9604,  0.7205,  0.0779
 0.5175,  0.4339,  0.9747, -0.4438, -0.9924,  0.2879
 0.8678,  0.7158,  0.4577,  0.0334,  0.4139,  0.1678
 0.5406,  0.5012,  0.2264, -0.1963,  0.3946,  0.2088
-0.9938,  0.5498,  0.7928, -0.5214, -0.7585,  0.7687
 0.7661,  0.0863, -0.4266, -0.7233, -0.4197,  0.1466
 0.2277, -0.3517, -0.0853, -0.1118,  0.6563,  0.1767
 0.3499, -0.5570, -0.0655, -0.3705,  0.2537,  0.1632
 0.7547, -0.1046,  0.5689, -0.0861,  0.3125,  0.1257
 0.8186,  0.2110,  0.5335,  0.0094, -0.0039,  0.1391
 0.6858, -0.8644,  0.1465,  0.8855,  0.0357,  0.1845
-0.4967,  0.4015,  0.0805,  0.8977,  0.2487,  0.4663
 0.6760, -0.9841,  0.9787, -0.8446, -0.3557,  0.1509
-0.1203, -0.4885,  0.6054, -0.0443, -0.7313,  0.4854
 0.8557,  0.7919, -0.0169,  0.7134, -0.1628,  0.2002
 0.0115, -0.6209,  0.9300, -0.4116, -0.7931,  0.4052
-0.7114, -0.9718,  0.4319,  0.1290,  0.5892,  0.3661
 0.3915,  0.5557, -0.1870,  0.2955, -0.6404,  0.2954
-0.3564, -0.6548, -0.1827, -0.5172, -0.1862,  0.4622
 0.2392, -0.4959,  0.5857, -0.1341, -0.2850,  0.2470
-0.3394,  0.3947, -0.4627,  0.6166, -0.4094,  0.5325
 0.7107,  0.7768, -0.6312,  0.1707,  0.7964,  0.2757
-0.1078,  0.8437, -0.4420,  0.2177,  0.3649,  0.4028
-0.3139,  0.5595, -0.6505, -0.3161, -0.7108,  0.5546
 0.4335,  0.3986,  0.3770, -0.4932,  0.3847,  0.1810
-0.2562, -0.2894, -0.8847,  0.2633,  0.4146,  0.4036
 0.2272,  0.2966, -0.6601, -0.7011,  0.0284,  0.2778
-0.0743, -0.1421, -0.0054, -0.6770, -0.3151,  0.3597
-0.4762,  0.6891,  0.6007, -0.1467,  0.2140,  0.4266
-0.4061,  0.7193,  0.3432,  0.2669, -0.7505,  0.6147
-0.0588,  0.9731,  0.8966,  0.2902, -0.6966,  0.4955
-0.0627, -0.1439,  0.1985,  0.6999,  0.5022,  0.3077
 0.1587,  0.8494, -0.8705,  0.9827, -0.8940,  0.4263
-0.7850,  0.2473, -0.9040, -0.4308, -0.8779,  0.7199
 0.4070,  0.3369, -0.2428, -0.6236,  0.4940,  0.2215
-0.0242,  0.0513, -0.9430,  0.2885, -0.2987,  0.3947
-0.5416, -0.1322, -0.2351, -0.0604,  0.9590,  0.3683
 0.1055,  0.7783, -0.2901, -0.5090,  0.8220,  0.2984
-0.9129,  0.9015,  0.1128, -0.2473,  0.9901,  0.4776
-0.9378,  0.1424, -0.6391,  0.2619,  0.9618,  0.5368
 0.7498, -0.0963,  0.4169,  0.5549, -0.0103,  0.1614
-0.2612, -0.7156,  0.4538, -0.0460, -0.1022,  0.3717
 0.7720,  0.0552, -0.1818, -0.4622, -0.8560,  0.1685
-0.4177,  0.0070,  0.9319, -0.7812,  0.3461,  0.3052
-0.0001,  0.5542, -0.7128, -0.8336, -0.2016,  0.3803
 0.5356, -0.4194, -0.5662, -0.9666, -0.2027,  0.1776
-0.2378,  0.3187, -0.8582, -0.6948, -0.9668,  0.5474
-0.1947, -0.3579,  0.1158,  0.9869,  0.6690,  0.2992
 0.3992,  0.8365, -0.9205, -0.8593, -0.0520,  0.3154
-0.0209,  0.0793,  0.7905, -0.1067,  0.7541,  0.1864
-0.4928, -0.4524, -0.3433,  0.0951, -0.5597,  0.6261
-0.8118,  0.7404, -0.5263, -0.2280,  0.1431,  0.6349
 0.0516, -0.8480,  0.7483,  0.9023,  0.6250,  0.1959
-0.3212,  0.1093,  0.9488, -0.3766,  0.3376,  0.2735
-0.3481,  0.5490, -0.3484,  0.7797,  0.5034,  0.4379
-0.5785, -0.9170, -0.3563, -0.9258,  0.3877,  0.4121
 0.3407, -0.1391,  0.5356,  0.0720, -0.9203,  0.3458
-0.3287, -0.8954,  0.2102,  0.0241,  0.2349,  0.3247
-0.1353,  0.6954, -0.0919, -0.9692,  0.7461,  0.3338
 0.9036, -0.8982, -0.5299, -0.8733, -0.1567,  0.1187
 0.7277, -0.8368, -0.0538, -0.7489,  0.5458,  0.0830
 0.9049,  0.8878,  0.2279,  0.9470, -0.3103,  0.2194
 0.7957, -0.1308, -0.5284,  0.8817,  0.3684,  0.2172
 0.4647, -0.4931,  0.2010,  0.6292, -0.8918,  0.3371
-0.7390,  0.6849,  0.2367,  0.0626, -0.5034,  0.7039
-0.1567, -0.8711,  0.7940, -0.5932,  0.6525,  0.1710
 0.7635, -0.0265,  0.1969,  0.0545,  0.2496,  0.1445
 0.7675,  0.1354, -0.7698, -0.5460,  0.1920,  0.1728
-0.5211, -0.7372, -0.6763,  0.6897,  0.2044,  0.5217
 0.1913,  0.1980,  0.2314, -0.8816,  0.5006,  0.1998
 0.8964,  0.0694, -0.6149,  0.5059, -0.9854,  0.1825
 0.1767,  0.7104,  0.2093,  0.6452,  0.7590,  0.2832
-0.3580, -0.7541,  0.4426, -0.1193, -0.7465,  0.5657
-0.5996,  0.5766, -0.9758, -0.3933, -0.9572,  0.6800
 0.9950,  0.1641, -0.4132,  0.8579,  0.0142,  0.2003
-0.4717, -0.3894, -0.2567, -0.5111,  0.1691,  0.4266
 0.3917, -0.8561,  0.9422,  0.5061,  0.6123,  0.1212
-0.0366, -0.1087,  0.3449, -0.1025,  0.4086,  0.2475
 0.3633,  0.3943,  0.2372, -0.6980,  0.5216,  0.1925
-0.5325, -0.6466, -0.2178, -0.3589,  0.6310,  0.3568
 0.2271,  0.5200, -0.1447, -0.8011, -0.7699,  0.3128
 0.6415,  0.1993,  0.3777, -0.0178, -0.8237,  0.2181
-0.5298, -0.0768, -0.6028, -0.9490,  0.4588,  0.4356
 0.6870, -0.1431,  0.7294,  0.3141,  0.1621,  0.1632
-0.5985,  0.0591,  0.7889, -0.3900,  0.7419,  0.2945
 0.3661,  0.7984, -0.8486,  0.7572, -0.6183,  0.3449
 0.6995,  0.3342, -0.3113, -0.6972,  0.2707,  0.1712
 0.2565,  0.9126,  0.1798, -0.6043, -0.1413,  0.2893
-0.3265,  0.9839, -0.2395,  0.9854,  0.0376,  0.4770
 0.2690, -0.1722,  0.9818,  0.8599, -0.7015,  0.3954
-0.2102, -0.0768,  0.1219,  0.5607, -0.0256,  0.3949
 0.8216, -0.9555,  0.6422, -0.6231,  0.3715,  0.0801
-0.2896,  0.9484, -0.7545, -0.6249,  0.7789,  0.4370
-0.9985, -0.5448, -0.7092, -0.5931,  0.7926,  0.5402

Test data:

# synthetic_test_40.txt
#
 0.7462,  0.4006, -0.0590,  0.6543, -0.0083,  0.1935
 0.8495, -0.2260, -0.0142, -0.4911,  0.7699,  0.1078
-0.2335, -0.4049,  0.4352, -0.6183, -0.7636,  0.5088
 0.1810, -0.5142,  0.2465,  0.2767, -0.3449,  0.3136
-0.8650,  0.7611, -0.0801,  0.5277, -0.4922,  0.7140
-0.2358, -0.7466, -0.5115, -0.8413, -0.3943,  0.4533
 0.4834,  0.2300,  0.3448, -0.9832,  0.3568,  0.1360
-0.6502, -0.6300,  0.6885,  0.9652,  0.8275,  0.3046
-0.3053,  0.5604,  0.0929,  0.6329, -0.0325,  0.4756
-0.7995,  0.0740, -0.2680,  0.2086,  0.9176,  0.4565
-0.2144, -0.2141,  0.5813,  0.2902, -0.2122,  0.4119
-0.7278, -0.0987, -0.3312, -0.5641,  0.8515,  0.4438
 0.3793,  0.1976,  0.4933,  0.0839,  0.4011,  0.1905
-0.8568,  0.9573, -0.5272,  0.3212, -0.8207,  0.7415
-0.5785,  0.0056, -0.7901, -0.2223,  0.0760,  0.5551
 0.0735, -0.2188,  0.3925,  0.3570,  0.3746,  0.2191
 0.1230, -0.2838,  0.2262,  0.8715,  0.1938,  0.2878
 0.4792, -0.9248,  0.5295,  0.0366, -0.9894,  0.3149
-0.4456,  0.0697,  0.5359, -0.8938,  0.0981,  0.3879
 0.8629, -0.8505, -0.4464,  0.8385,  0.5300,  0.1769
 0.1995,  0.6659,  0.7921,  0.9454,  0.9970,  0.2330
-0.0249, -0.3066, -0.2927, -0.4923,  0.8220,  0.2437
 0.4513, -0.9481, -0.0770, -0.4374, -0.9421,  0.2879
-0.3405,  0.5931, -0.3507, -0.3842,  0.8562,  0.3987
 0.9538,  0.0471,  0.9039,  0.7760,  0.0361,  0.1706
-0.0887,  0.2104,  0.9808,  0.5478, -0.3314,  0.4128
-0.8220, -0.6302,  0.0537, -0.1658,  0.6013,  0.4306
-0.4123, -0.2880,  0.9074, -0.0461, -0.4435,  0.5144
 0.0060,  0.2867, -0.7775,  0.5161,  0.7039,  0.3599
-0.7968, -0.5484,  0.9426, -0.4308,  0.8148,  0.2979
 0.7811,  0.8450, -0.6877,  0.7594,  0.2640,  0.2362
-0.6802, -0.1113, -0.8325, -0.6694, -0.6056,  0.6544
 0.3821,  0.1476,  0.7466, -0.5107,  0.2592,  0.1648
 0.7265,  0.9683, -0.9803, -0.4943, -0.5523,  0.2454
-0.9049, -0.9797, -0.0196, -0.9090, -0.4433,  0.6447
-0.4607,  0.1811, -0.2389,  0.4050, -0.0078,  0.5229
 0.2664, -0.2932, -0.4259, -0.7336,  0.8742,  0.1834
-0.4507,  0.1029, -0.6294, -0.1158, -0.6294,  0.6081
 0.8948, -0.0124,  0.9278,  0.2899, -0.0314,  0.1534
-0.1323, -0.8813, -0.0146, -0.0697,  0.6135,  0.2386

I had been working on a decision tree regression system, implemented from scratch, using the C# language. In order to validate from from-scratch implementation, I figured I’d run some data through my from-scratch implementation and the scikit DecisionTreeRegressor module to make sure I got the same results.

For my investigation, I decided to use the well-known diabetes dataset. The goal of the dataset is to predict a diabetes score from 10 predictor variables. The data looks like:

59, 2, 32.1, 101.00, 157, 93.2, 38, 4.00, 4.8598, 87, 151
48, 1, 21.6, 87.00, 183, 103.2, 70, 3.00, 3.8918, 69, 75
. . .

The 10 predictor variables are age, sex, body mass index, blood pressure, serum-cholesterol, low-density lipoproteins, high-density lipoproteins, total-cholesterol, triglycerides, blood-sugar. The target diabetes score is in last column.

There are 442 items in the diabetes dataset. I split the data into a 342-item training set and a 100-item test set. I did not normalize the data.

The output of the scikit demo is:

Decision tree regression using scikit

Loading diabetes train (340), test (100) data
Done

First three X predictors:
[ 59.0000   2.0000  32.1000 . . .   87.0000]
[ 48.0000   1.0000  21.6000  . . .  69.0000]
[ 72.0000   2.0000  30.5000  . . .  85.0000]

First three y targets:
151.0000
75.0000
141.0000

Setting max_depth = 12

Accuracy on train (within 0.10) = 0.8304
Accuracy on test (within 0.10) = 0.1200

MSE on train = 190.5995
MSE on test = 6936.7650

Predicting for:
[[ 59.0000   2.0000  32.1000 . . .  87.0000]]
Predicted y = 151.0000

Done

I set the DecisionTreeRegressor max-depth parameter to 12, and left all other parameters with default values, in particular min_samples_split=2 and min_samples_leaf=1.

The results show the key weakness of decision tree regression: the model almost always overfits the training data and therefore gives poor prediction accuracy on new, previously unseen data. This is why decision tree regression is rarely used by itself — instead many decision trees are used together in common ensemble techniques: bagging tree (bootstrap aggregation), random forest tree, adaptive boosting, gradient boosting.

The output of my from-scratch C# implementation was almost the same:

Begin decision tree regression (no stack construction)

Loading diabetes train (342) and test (100) data
Done

First three train X:
 59.0000  2.0000 32.1000 . . . 87.0000
 48.0000  1.0000 21.6000 . . . 69.0000
 72.0000  2.0000 30.5000 . . . 85.0000

First three train y:
151.0000
 75.0000
141.0000

Setting maxDepth = 12
Setting minSamples = 2
Setting minLeaf = 1
Using default numSplitCols = -1

Creating and training tree
Done

Evaluating model
Accuracy train (within 0.10) = 0.8304
Accuracy test (within 0.10) = 0.1300

MSE train = 190.5995
MSE test = 6936.0606

Predicting for trainX[0] =
  59.0000   2.0000  32.1000 . . . 87.0000
Predicted y = 151.0000

IF
column 8  >    4.8203 AND
column 3  <= 112.0000 AND
column 9  <=  98.0000 AND
column 2  <=  32.1000 AND
column 3  >   83.0000 AND
column 5  >   84.2000 AND
column 2  >   30.0000 AND
column 0  >   40.0000 AND
column 8  <=   5.2470 AND
column 6  <=  38.0000 AND
THEN
predicted = 151.0000

End demo

The two systems (scikit version and C# version) don't give identical results because they use slightly different algorithms for splitting subtrees when constructing the decision tree.

Good fun.

Each node in a decision tree has a left child and a right child. Tree children are neither sons not daughters.

Right: In "Monster from Green Hell" (1957), the U.S. space program is sending animals and insects into space. One rocket that is carrying wasps malfunctions and crash lands in Africa. Wasps + cosmic radiation = giant wasps. Scientist Dr. Lorentz and his daughter Lorna (actress Barbara Turner) are in the area and help U.S. scientist Dr. Quent Brady to try and deal with the wasps. Luckily, a well-timed volcanic eruption solves the wasp problem.

Demo code. Replace the "lt" with the Boolean less-than operator.

# decision_tree_scikit.py

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.tree import DecisionTreeRegressor

np.set_printoptions(precision=4, suppress=True,
  floatmode='fixed', linewidth=120)

# -----------------------------------------------------------

def accuracy(model, data_X, data_y, pct_close):
  n = len(data_X)
  n_correct = 0; n_wrong = 0
  for i in range(n):
    x = data_X[i].reshape(1,-1)
    y = data_y[i]
    y_pred = model.predict(x)

    if np.abs(y - y_pred) "lt" np.abs(y * pct_close):
      n_correct += 1
    else: 
      n_wrong += 1
  # print("Correct = " + str(n_correct))
  # print("Wrong   = " + str(n_wrong))
  return n_correct / (n_correct + n_wrong)

# -----------------------------------------------------------

def MSE(model, data_X, data_y):
  n = len(data_X)
  sum = 0.0
  for i in range(n):
    x = data_X[i].reshape(1,-1)
    y = data_y[i]
    y_pred = model.predict(x)[0]
    # print(y_pred); input()
    sum += (y - y_pred) * (y - y_pred)

  return sum / n

# -----------------------------------------------------------

# class sklearn.tree.DecisionTreeRegressor(*,
# criterion='squared_error', splitter='best',
# max_depth=None, min_samples_split=2,
# min_samples_leaf=1, min_weight_fraction_leaf=0.0,
# max_features=None, random_state=None,
# max_leaf_nodes=None, min_impurity_decrease=0.0,
# ccp_alpha=0.0, monotonic_cst=None

print("\nDecision tree regression using scikit ")

print("\nLoading diabetes train (342), test (100) data ")
train_file = ".\\Data\\diabetes_train_342.txt"
train_X = np.loadtxt(train_file, comments="#",
  usecols=[0,1,2,3,4,5,6,7,8,9],
  delimiter=",",  dtype=np.float64)
train_y = np.loadtxt(train_file, comments="#", usecols=10,
  delimiter=",",  dtype=np.float64)

test_file = ".\\Data\\diabetes_test_100.txt"
test_X = np.loadtxt(test_file, comments="#",
  usecols=[0,1,2,3,4,5,6,7,8,9],
  delimiter=",",  dtype=np.float64)
test_y = np.loadtxt(test_file, comments="#", usecols=10,
  delimiter=",",  dtype=np.float64)
print("Done ")

print("\nFirst three X predictors: ")
for i in range(3):
  print(train_X[i])
print("\nFirst three y targets: ")
for i in range(3):
  print("%0.4f" % train_y[i])

maxd = 12
print("\nSetting max_depth = " + str(maxd))
model = DecisionTreeRegressor(max_depth=maxd,
  random_state=0)
model.fit(train_X, train_y)

acc_train = accuracy(model, train_X, train_y, 0.10)
print("\nAccuracy on train (within 0.10) = \
%0.4f " % acc_train)
acc_test = accuracy(model, test_X, test_y, 0.10)
print("Accuracy on test (within 0.10) = \
%0.4f " % acc_test)

mse_train = MSE(model, train_X, train_y)
print("\nMSE on train = %0.4f " % mse_train)
mse_test = MSE(model, test_X, test_y)
print("MSE on test = %0.4f " % mse_test)

x = train_X[0].reshape(1,-1)
print("\nPredicting for: ")
print(x)
y_pred = model.predict(x)[0]
print("Predicted y = %0.4f " % y_pred)

print("\nDone ")

Training data:

# diabetes_train_342.txt
# 10 predictors, target y
# 342 items
# www4.stat.ncsu.edu/~boos/var.select/diabetes.tab.txt
#
59, 2, 32.1, 101.00, 157, 93.2, 38, 4.00, 4.8598, 87, 151
48, 1, 21.6, 87.00, 183, 103.2, 70, 3.00, 3.8918, 69, 75
72, 2, 30.5, 93.00, 156, 93.6, 41, 4.00, 4.6728, 85, 141
24, 1, 25.3, 84.00, 198, 131.4, 40, 5.00, 4.8903, 89, 206
50, 1, 23.0, 101.00, 192, 125.4, 52, 4.00, 4.2905, 80, 135
23, 1, 22.6, 89.00, 139, 64.8, 61, 2.00, 4.1897, 68, 97
36, 2, 22.0, 90.00, 160, 99.6, 50, 3.00, 3.9512, 82, 138
66, 2, 26.2, 114.00, 255, 185.0, 56, 4.55, 4.2485, 92, 63
60, 2, 32.1, 83.00, 179, 119.4, 42, 4.00, 4.4773, 94, 110
29, 1, 30.0, 85.00, 180, 93.4, 43, 4.00, 5.3845, 88, 310
22, 1, 18.6, 97.00, 114, 57.6, 46, 2.00, 3.9512, 83, 101
56, 2, 28.0, 85.00, 184, 144.8, 32, 6.00, 3.5835, 77, 69
53, 1, 23.7, 92.00, 186, 109.2, 62, 3.00, 4.3041, 81, 179
50, 2, 26.2, 97.00, 186, 105.4, 49, 4.00, 5.0626, 88, 185
61, 1, 24.0, 91.00, 202, 115.4, 72, 3.00, 4.2905, 73, 118
34, 2, 24.7, 118.00, 254, 184.2, 39, 7.00, 5.0370, 81, 171
47, 1, 30.3, 109.00, 207, 100.2, 70, 3.00, 5.2149, 98, 166
68, 2, 27.5, 111.00, 214, 147.0, 39, 5.00, 4.9416, 91, 144
38, 1, 25.4, 84.00, 162, 103.0, 42, 4.00, 4.4427, 87, 97
41, 1, 24.7, 83.00, 187, 108.2, 60, 3.00, 4.5433, 78, 168
35, 1, 21.1, 82.00, 156, 87.8, 50, 3.00, 4.5109, 95, 68
25, 2, 24.3, 95.00, 162, 98.6, 54, 3.00, 3.8501, 87, 49
25, 1, 26.0, 92.00, 187, 120.4, 56, 3.00, 3.9703, 88, 68
61, 2, 32.0, 103.67, 210, 85.2, 35, 6.00, 6.1070, 124, 245
31, 1, 29.7, 88.00, 167, 103.4, 48, 4.00, 4.3567, 78, 184
30, 2, 25.2, 83.00, 178, 118.4, 34, 5.00, 4.8520, 83, 202
19, 1, 19.2, 87.00, 124, 54.0, 57, 2.00, 4.1744, 90, 137
42, 1, 31.9, 83.00, 158, 87.6, 53, 3.00, 4.4659, 101, 85
63, 1, 24.4, 73.00, 160, 91.4, 48, 3.00, 4.6347, 78, 131
67, 2, 25.8, 113.00, 158, 54.2, 64, 2.00, 5.2933, 104, 283
32, 1, 30.5, 89.00, 182, 110.6, 56, 3.00, 4.3438, 89, 129
42, 1, 20.3, 71.00, 161, 81.2, 66, 2.00, 4.2341, 81, 59
58, 2, 38.0, 103.00, 150, 107.2, 22, 7.00, 4.6444, 98, 341
57, 1, 21.7, 94.00, 157, 58.0, 82, 2.00, 4.4427, 92, 87
53, 1, 20.5, 78.00, 147, 84.2, 52, 3.00, 3.9890, 75, 65
62, 2, 23.5, 80.33, 225, 112.8, 86, 2.62, 4.8752, 96, 102
52, 1, 28.5, 110.00, 195, 97.2, 60, 3.00, 5.2417, 85, 265
46, 1, 27.4, 78.00, 171, 88.0, 58, 3.00, 4.8283, 90, 276
48, 2, 33.0, 123.00, 253, 163.6, 44, 6.00, 5.4250, 97, 252
48, 2, 27.7, 73.00, 191, 119.4, 46, 4.00, 4.8520, 92, 90
50, 2, 25.6, 101.00, 229, 162.2, 43, 5.00, 4.7791, 114, 100
21, 1, 20.1, 63.00, 135, 69.0, 54, 3.00, 4.0943, 89, 55
32, 2, 25.4, 90.33, 153, 100.4, 34, 4.50, 4.5326, 83, 61
54, 1, 24.2, 74.00, 204, 109.0, 82, 2.00, 4.1744, 109, 92
61, 2, 32.7, 97.00, 177, 118.4, 29, 6.00, 4.9972, 87, 259
56, 2, 23.1, 104.00, 181, 116.4, 47, 4.00, 4.4773, 79, 53
33, 1, 25.3, 85.00, 155, 85.0, 51, 3.00, 4.5539, 70, 190
27, 1, 19.6, 78.00, 128, 68.0, 43, 3.00, 4.4427, 71, 142
67, 2, 22.5, 98.00, 191, 119.2, 61, 3.00, 3.9890, 86, 75
37, 2, 27.7, 93.00, 180, 119.4, 30, 6.00, 5.0304, 88, 142
58, 1, 25.7, 99.00, 157, 91.6, 49, 3.00, 4.4067, 93, 155
65, 2, 27.9, 103.00, 159, 96.8, 42, 4.00, 4.6151, 86, 225
34, 1, 25.5, 93.00, 218, 144.0, 57, 4.00, 4.4427, 88, 59
46, 1, 24.9, 115.00, 198, 129.6, 54, 4.00, 4.2767, 103, 104
35, 1, 28.7, 97.00, 204, 126.8, 64, 3.00, 4.1897, 93, 182
37, 1, 21.8, 84.00, 184, 101.0, 73, 3.00, 3.9120, 93, 128
37, 1, 30.2, 87.00, 166, 96.0, 40, 4.15, 5.0106, 87, 52
41, 1, 20.5, 80.00, 124, 48.8, 64, 2.00, 4.0254, 75, 37
60, 1, 20.4, 105.00, 198, 78.4, 99, 2.00, 4.6347, 79, 170
66, 2, 24.0, 98.00, 236, 146.4, 58, 4.00, 5.0626, 96, 170
29, 1, 26.0, 83.00, 141, 65.2, 64, 2.00, 4.0775, 83, 61
37, 2, 26.8, 79.00, 157, 98.0, 28, 6.00, 5.0434, 96, 144
41, 2, 25.7, 83.00, 181, 106.6, 66, 3.00, 3.7377, 85, 52
39, 1, 22.9, 77.00, 204, 143.2, 46, 4.00, 4.3041, 74, 128
67, 2, 24.0, 83.00, 143, 77.2, 49, 3.00, 4.4308, 94, 71
36, 2, 24.1, 112.00, 193, 125.0, 35, 6.00, 5.1059, 95, 163
46, 2, 24.7, 85.00, 174, 123.2, 30, 6.00, 4.6444, 96, 150
60, 2, 25.0, 89.67, 185, 120.8, 46, 4.02, 4.5109, 92, 97
59, 2, 23.6, 83.00, 165, 100.0, 47, 4.00, 4.4998, 92, 160
53, 1, 22.1, 93.00, 134, 76.2, 46, 3.00, 4.0775, 96, 178
48, 1, 19.9, 91.00, 189, 109.6, 69, 3.00, 3.9512, 101, 48
48, 1, 29.5, 131.00, 207, 132.2, 47, 4.00, 4.9345, 106, 270
66, 2, 26.0, 91.00, 264, 146.6, 65, 4.00, 5.5683, 87, 202
52, 2, 24.5, 94.00, 217, 149.4, 48, 5.00, 4.5850, 89, 111
52, 2, 26.6, 111.00, 209, 126.4, 61, 3.00, 4.6821, 109, 85
46, 2, 23.5, 87.00, 181, 114.8, 44, 4.00, 4.7095, 98, 42
40, 2, 29.0, 115.00, 97, 47.2, 35, 2.77, 4.3041, 95, 170
22, 1, 23.0, 73.00, 161, 97.8, 54, 3.00, 3.8286, 91, 200
50, 1, 21.0, 88.00, 140, 71.8, 35, 4.00, 5.1120, 71, 252
20, 1, 22.9, 87.00, 191, 128.2, 53, 4.00, 3.8918, 85, 113
68, 1, 27.5, 107.00, 241, 149.6, 64, 4.00, 4.9200, 90, 143
52, 2, 24.3, 86.00, 197, 133.6, 44, 5.00, 4.5747, 91, 51
44, 1, 23.1, 87.00, 213, 126.4, 77, 3.00, 3.8712, 72, 52
38, 1, 27.3, 81.00, 146, 81.6, 47, 3.00, 4.4659, 81, 210
49, 1, 22.7, 65.33, 168, 96.2, 62, 2.71, 3.8918, 60, 65
61, 1, 33.0, 95.00, 182, 114.8, 54, 3.00, 4.1897, 74, 141
29, 2, 19.4, 83.00, 152, 105.8, 39, 4.00, 3.5835, 83, 55
61, 1, 25.8, 98.00, 235, 125.8, 76, 3.00, 5.1120, 82, 134
34, 2, 22.6, 75.00, 166, 91.8, 60, 3.00, 4.2627, 108, 42
36, 1, 21.9, 89.00, 189, 105.2, 68, 3.00, 4.3694, 96, 111
52, 1, 24.0, 83.00, 167, 86.6, 71, 2.00, 3.8501, 94, 98
61, 1, 31.2, 79.00, 235, 156.8, 47, 5.00, 5.0499, 96, 164
43, 1, 26.8, 123.00, 193, 102.2, 67, 3.00, 4.7791, 94, 48
35, 1, 20.4, 65.00, 187, 105.6, 67, 2.79, 4.2767, 78, 96
27, 1, 24.8, 91.00, 189, 106.8, 69, 3.00, 4.1897, 69, 90
29, 1, 21.0, 71.00, 156, 97.0, 38, 4.00, 4.6540, 90, 162
64, 2, 27.3, 109.00, 186, 107.6, 38, 5.00, 5.3083, 99, 150
41, 1, 34.6, 87.33, 205, 142.6, 41, 5.00, 4.6728, 110, 279
49, 2, 25.9, 91.00, 178, 106.6, 52, 3.00, 4.5747, 75, 92
48, 1, 20.4, 98.00, 209, 139.4, 46, 5.00, 4.7707, 78, 83
53, 1, 28.0, 88.00, 233, 143.8, 58, 4.00, 5.0499, 91, 128
53, 2, 22.2, 113.00, 197, 115.2, 67, 3.00, 4.3041, 100, 102
23, 1, 29.0, 90.00, 216, 131.4, 65, 3.00, 4.5850, 91, 302
65, 2, 30.2, 98.00, 219, 160.6, 40, 5.00, 4.5218, 84, 198
41, 1, 32.4, 94.00, 171, 104.4, 56, 3.00, 3.9703, 76, 95
55, 2, 23.4, 83.00, 166, 101.6, 46, 4.00, 4.5218, 96, 53
22, 1, 19.3, 82.00, 156, 93.2, 52, 3.00, 3.9890, 71, 134
56, 1, 31.0, 78.67, 187, 141.4, 34, 5.50, 4.0604, 90, 144
54, 2, 30.6, 103.33, 144, 79.8, 30, 4.80, 5.1417, 101, 232
59, 2, 25.5, 95.33, 190, 139.4, 35, 5.43, 4.3567, 117, 81
60, 2, 23.4, 88.00, 153, 89.8, 58, 3.00, 3.2581, 95, 104
54, 1, 26.8, 87.00, 206, 122.0, 68, 3.00, 4.3820, 80, 59
25, 1, 28.3, 87.00, 193, 128.0, 49, 4.00, 4.3820, 92, 246
54, 2, 27.7, 113.00, 200, 128.4, 37, 5.00, 5.1533, 113, 297
55, 1, 36.6, 113.00, 199, 94.4, 43, 4.63, 5.7301, 97, 258
40, 2, 26.5, 93.00, 236, 147.0, 37, 7.00, 5.5607, 92, 229
62, 2, 31.8, 115.00, 199, 128.6, 44, 5.00, 4.8828, 98, 275
65, 1, 24.4, 120.00, 222, 135.6, 37, 6.00, 5.5094, 124, 281
33, 2, 25.4, 102.00, 206, 141.0, 39, 5.00, 4.8675, 105, 179
53, 1, 22.0, 94.00, 175, 88.0, 59, 3.00, 4.9416, 98, 200
35, 1, 26.8, 98.00, 162, 103.6, 45, 4.00, 4.2047, 86, 200
66, 1, 28.0, 101.00, 195, 129.2, 40, 5.00, 4.8598, 94, 173
62, 2, 33.9, 101.00, 221, 156.4, 35, 6.00, 4.9972, 103, 180
50, 2, 29.6, 94.33, 300, 242.4, 33, 9.09, 4.8122, 109, 84
47, 1, 28.6, 97.00, 164, 90.6, 56, 3.00, 4.4659, 88, 121
47, 2, 25.6, 94.00, 165, 74.8, 40, 4.00, 5.5255, 93, 161
24, 1, 20.7, 87.00, 149, 80.6, 61, 2.00, 3.6109, 78, 99
58, 2, 26.2, 91.00, 217, 124.2, 71, 3.00, 4.6913, 68, 109
34, 1, 20.6, 87.00, 185, 112.2, 58, 3.00, 4.3041, 74, 115
51, 1, 27.9, 96.00, 196, 122.2, 42, 5.00, 5.0689, 120, 268
31, 2, 35.3, 125.00, 187, 112.4, 48, 4.00, 4.8903, 109, 274
22, 1, 19.9, 75.00, 175, 108.6, 54, 3.00, 4.1271, 72, 158
53, 2, 24.4, 92.00, 214, 146.0, 50, 4.00, 4.4998, 97, 107
37, 2, 21.4, 83.00, 128, 69.6, 49, 3.00, 3.8501, 84, 83
28, 1, 30.4, 85.00, 198, 115.6, 67, 3.00, 4.3438, 80, 103
47, 1, 31.6, 84.00, 154, 88.0, 30, 5.10, 5.1985, 105, 272
23, 1, 18.8, 78.00, 145, 72.0, 63, 2.00, 3.9120, 86, 85
50, 1, 31.0, 123.00, 178, 105.0, 48, 4.00, 4.8283, 88, 280
58, 2, 36.7, 117.00, 166, 93.8, 44, 4.00, 4.9488, 109, 336
55, 1, 32.1, 110.00, 164, 84.2, 42, 4.00, 5.2417, 90, 281
60, 2, 27.7, 107.00, 167, 114.6, 38, 4.00, 4.2767, 95, 118
41, 1, 30.8, 81.00, 214, 152.0, 28, 7.60, 5.1358, 123, 317
60, 2, 27.5, 106.00, 229, 143.8, 51, 4.00, 5.1417, 91, 235
40, 1, 26.9, 92.00, 203, 119.8, 70, 3.00, 4.1897, 81, 60
57, 2, 30.7, 90.00, 204, 147.8, 34, 6.00, 4.7095, 93, 174
37, 1, 38.3, 113.00, 165, 94.6, 53, 3.00, 4.4659, 79, 259
40, 2, 31.9, 95.00, 198, 135.6, 38, 5.00, 4.8040, 93, 178
33, 1, 35.0, 89.00, 200, 130.4, 42, 4.76, 4.9273, 101, 128
32, 2, 27.8, 89.00, 216, 146.2, 55, 4.00, 4.3041, 91, 96
35, 2, 25.9, 81.00, 174, 102.4, 31, 6.00, 5.3132, 82, 126
55, 1, 32.9, 102.00, 164, 106.2, 41, 4.00, 4.4308, 89, 288
49, 1, 26.0, 93.00, 183, 100.2, 64, 3.00, 4.5433, 88, 88
39, 2, 26.3, 115.00, 218, 158.2, 32, 7.00, 4.9345, 109, 292
60, 2, 22.3, 113.00, 186, 125.8, 46, 4.00, 4.2627, 94, 71
67, 2, 28.3, 93.00, 204, 132.2, 49, 4.00, 4.7362, 92, 197
41, 2, 32.0, 109.00, 251, 170.6, 49, 5.00, 5.0562, 103, 186
44, 1, 25.4, 95.00, 162, 92.6, 53, 3.00, 4.4067, 83, 25
48, 2, 23.3, 89.33, 212, 142.8, 46, 4.61, 4.7536, 98, 84
45, 1, 20.3, 74.33, 190, 126.2, 49, 3.88, 4.3041, 79, 96
47, 1, 30.4, 120.00, 199, 120.0, 46, 4.00, 5.1059, 87, 195
46, 1, 20.6, 73.00, 172, 107.0, 51, 3.00, 4.2485, 80, 53
36, 2, 32.3, 115.00, 286, 199.4, 39, 7.00, 5.4723, 112, 217
34, 1, 29.2, 73.00, 172, 108.2, 49, 4.00, 4.3041, 91, 172
53, 2, 33.1, 117.00, 183, 119.0, 48, 4.00, 4.3820, 106, 131
61, 1, 24.6, 101.00, 209, 106.8, 77, 3.00, 4.8363, 88, 214
37, 1, 20.2, 81.00, 162, 87.8, 63, 3.00, 4.0254, 88, 59
33, 2, 20.8, 84.00, 125, 70.2, 46, 3.00, 3.7842, 66, 70
68, 1, 32.8, 105.67, 205, 116.4, 40, 5.13, 5.4931, 117, 220
49, 2, 31.9, 94.00, 234, 155.8, 34, 7.00, 5.3982, 122, 268
48, 1, 23.9, 109.00, 232, 105.2, 37, 6.00, 6.1070, 96, 152
55, 2, 24.5, 84.00, 179, 105.8, 66, 3.00, 3.5835, 87, 47
43, 1, 22.1, 66.00, 134, 77.2, 45, 3.00, 4.0775, 80, 74
60, 2, 33.0, 97.00, 217, 125.6, 45, 5.00, 5.4467, 112, 295
31, 2, 19.0, 93.00, 137, 73.0, 47, 3.00, 4.4427, 78, 101
53, 2, 27.3, 82.00, 119, 55.0, 39, 3.00, 4.8283, 93, 151
67, 1, 22.8, 87.00, 166, 98.6, 52, 3.00, 4.3438, 92, 127
61, 2, 28.2, 106.00, 204, 132.0, 52, 4.00, 4.6052, 96, 237
62, 1, 28.9, 87.33, 206, 127.2, 33, 6.24, 5.4337, 99, 225
60, 1, 25.6, 87.00, 207, 125.8, 69, 3.00, 4.1109, 84, 81
42, 1, 24.9, 91.00, 204, 141.8, 38, 5.00, 4.7958, 89, 151
38, 2, 26.8, 105.00, 181, 119.2, 37, 5.00, 4.8203, 91, 107
62, 1, 22.4, 79.00, 222, 147.4, 59, 4.00, 4.3567, 76, 64
61, 2, 26.9, 111.00, 236, 172.4, 39, 6.00, 4.8122, 89, 138
61, 2, 23.1, 113.00, 186, 114.4, 47, 4.00, 4.8122, 105, 185
53, 1, 28.6, 88.00, 171, 98.8, 41, 4.00, 5.0499, 99, 265
28, 2, 24.7, 97.00, 175, 99.6, 32, 5.00, 5.3799, 87, 101
26, 2, 30.3, 89.00, 218, 152.2, 31, 7.00, 5.1591, 82, 137
30, 1, 21.3, 87.00, 134, 63.0, 63, 2.00, 3.6889, 66, 143
50, 1, 26.1, 109.00, 243, 160.6, 62, 4.00, 4.6250, 89, 141
48, 1, 20.2, 95.00, 187, 117.4, 53, 4.00, 4.4188, 85, 79
51, 1, 25.2, 103.00, 176, 112.2, 37, 5.00, 4.8978, 90, 292
47, 2, 22.5, 82.00, 131, 66.8, 41, 3.00, 4.7536, 89, 178
64, 2, 23.5, 97.00, 203, 129.0, 59, 3.00, 4.3175, 77, 91
51, 2, 25.9, 76.00, 240, 169.0, 39, 6.00, 5.0752, 96, 116
30, 1, 20.9, 104.00, 152, 83.8, 47, 3.00, 4.6634, 97, 86
56, 2, 28.7, 99.00, 208, 146.4, 39, 5.00, 4.7274, 97, 122
42, 1, 22.1, 85.00, 213, 138.6, 60, 4.00, 4.2767, 94, 72
62, 2, 26.7, 115.00, 183, 124.0, 35, 5.00, 4.7875, 100, 129
34, 1, 31.4, 87.00, 149, 93.8, 46, 3.00, 3.8286, 77, 142
60, 1, 22.2, 104.67, 221, 105.4, 60, 3.68, 5.6276, 93, 90
64, 1, 21.0, 92.33, 227, 146.8, 65, 3.49, 4.3307, 102, 158
39, 2, 21.2, 90.00, 182, 110.4, 60, 3.00, 4.0604, 98, 39
71, 2, 26.5, 105.00, 281, 173.6, 55, 5.00, 5.5683, 84, 196
48, 2, 29.2, 110.00, 218, 151.6, 39, 6.00, 4.9200, 98, 222
79, 2, 27.0, 103.00, 169, 110.8, 37, 5.00, 4.6634, 110, 277
40, 1, 30.7, 99.00, 177, 85.4, 50, 4.00, 5.3375, 85, 99
49, 2, 28.8, 92.00, 207, 140.0, 44, 5.00, 4.7449, 92, 196
51, 1, 30.6, 103.00, 198, 106.6, 57, 3.00, 5.1475, 100, 202
57, 1, 30.1, 117.00, 202, 139.6, 42, 5.00, 4.6250, 120, 155
59, 2, 24.7, 114.00, 152, 104.8, 29, 5.00, 4.5109, 88, 77
51, 1, 27.7, 99.00, 229, 145.6, 69, 3.00, 4.2767, 77, 191
74, 1, 29.8, 101.00, 171, 104.8, 50, 3.00, 4.3944, 86, 70
67, 1, 26.7, 105.00, 225, 135.4, 69, 3.00, 4.6347, 96, 73
49, 1, 19.8, 88.00, 188, 114.8, 57, 3.00, 4.3944, 93, 49
57, 1, 23.3, 88.00, 155, 63.6, 78, 2.00, 4.2047, 78, 65
56, 2, 35.1, 123.00, 164, 95.0, 38, 4.00, 5.0434, 117, 263
52, 2, 29.7, 109.00, 228, 162.8, 31, 8.00, 5.1417, 103, 248
69, 1, 29.3, 124.00, 223, 139.0, 54, 4.00, 5.0106, 102, 296
37, 1, 20.3, 83.00, 185, 124.6, 38, 5.00, 4.7185, 88, 214
24, 1, 22.5, 89.00, 141, 68.0, 52, 3.00, 4.6540, 84, 185
55, 2, 22.7, 93.00, 154, 94.2, 53, 3.00, 3.5264, 75, 78
36, 1, 22.8, 87.00, 178, 116.0, 41, 4.00, 4.6540, 82, 93
42, 2, 24.0, 107.00, 150, 85.0, 44, 3.00, 4.6540, 96, 252
21, 1, 24.2, 76.00, 147, 77.0, 53, 3.00, 4.4427, 79, 150
41, 1, 20.2, 62.00, 153, 89.0, 50, 3.00, 4.2485, 89, 77
57, 2, 29.4, 109.00, 160, 87.6, 31, 5.00, 5.3327, 92, 208
20, 2, 22.1, 87.00, 171, 99.6, 58, 3.00, 4.2047, 78, 77
67, 2, 23.6, 111.33, 189, 105.4, 70, 2.70, 4.2195, 93, 108
34, 1, 25.2, 77.00, 189, 120.6, 53, 4.00, 4.3438, 79, 160
41, 2, 24.9, 86.00, 192, 115.0, 61, 3.00, 4.3820, 94, 53
38, 2, 33.0, 78.00, 301, 215.0, 50, 6.02, 5.1930, 108, 220
51, 1, 23.5, 101.00, 195, 121.0, 51, 4.00, 4.7449, 94, 154
52, 2, 26.4, 91.33, 218, 152.0, 39, 5.59, 4.9053, 99, 259
67, 1, 29.8, 80.00, 172, 93.4, 63, 3.00, 4.3567, 82, 90
61, 1, 30.0, 108.00, 194, 100.0, 52, 3.73, 5.3471, 105, 246
67, 2, 25.0, 111.67, 146, 93.4, 33, 4.42, 4.5850, 103, 124
56, 1, 27.0, 105.00, 247, 160.6, 54, 5.00, 5.0876, 94, 67
64, 1, 20.0, 74.67, 189, 114.8, 62, 3.05, 4.1109, 91, 72
58, 2, 25.5, 112.00, 163, 110.6, 29, 6.00, 4.7622, 86, 257
55, 1, 28.2, 91.00, 250, 140.2, 67, 4.00, 5.3660, 103, 262
62, 2, 33.3, 114.00, 182, 114.0, 38, 5.00, 5.0106, 96, 275
57, 2, 25.6, 96.00, 200, 133.0, 52, 3.85, 4.3175, 105, 177
20, 2, 24.2, 88.00, 126, 72.2, 45, 3.00, 3.7842, 74, 71
53, 2, 22.1, 98.00, 165, 105.2, 47, 4.00, 4.1589, 81, 47
32, 2, 31.4, 89.00, 153, 84.2, 56, 3.00, 4.1589, 90, 187
41, 1, 23.1, 86.00, 148, 78.0, 58, 3.00, 4.0943, 60, 125
60, 1, 23.4, 76.67, 247, 148.0, 65, 3.80, 5.1358, 77, 78
26, 1, 18.8, 83.00, 191, 103.6, 69, 3.00, 4.5218, 69, 51
37, 1, 30.8, 112.00, 282, 197.2, 43, 7.00, 5.3423, 101, 258
45, 1, 32.0, 110.00, 224, 134.2, 45, 5.00, 5.4116, 93, 215
67, 1, 31.6, 116.00, 179, 90.4, 41, 4.00, 5.4723, 100, 303
34, 2, 35.5, 120.00, 233, 146.6, 34, 7.00, 5.5683, 101, 243
50, 1, 31.9, 78.33, 207, 149.2, 38, 5.45, 4.5951, 84, 91
71, 1, 29.5, 97.00, 227, 151.6, 45, 5.00, 5.0239, 108, 150
57, 2, 31.6, 117.00, 225, 107.6, 40, 6.00, 5.9584, 113, 310
49, 1, 20.3, 93.00, 184, 103.0, 61, 3.00, 4.6052, 93, 153
35, 1, 41.3, 81.00, 168, 102.8, 37, 5.00, 4.9488, 94, 346
41, 2, 21.2, 102.00, 184, 100.4, 64, 3.00, 4.5850, 79, 63
70, 2, 24.1, 82.33, 194, 149.2, 31, 6.26, 4.2341, 105, 89
52, 1, 23.0, 107.00, 179, 123.7, 42.5, 4.21, 4.1589, 93, 50
60, 1, 25.6, 78.00, 195, 95.4, 91, 2.00, 3.7612, 87, 39
62, 1, 22.5, 125.00, 215, 99.0, 98, 2.00, 4.4998, 95, 103
44, 2, 38.2, 123.00, 201, 126.6, 44, 5.00, 5.0239, 92, 308
28, 2, 19.2, 81.00, 155, 94.6, 51, 3.00, 3.8501, 87, 116
58, 2, 29.0, 85.00, 156, 109.2, 36, 4.00, 3.9890, 86, 145
39, 2, 24.0, 89.67, 190, 113.6, 52, 3.65, 4.8040, 101, 74
34, 2, 20.6, 98.00, 183, 92.0, 83, 2.00, 3.6889, 92, 45
65, 1, 26.3, 70.00, 244, 166.2, 51, 5.00, 4.8978, 98, 115
66, 2, 34.6, 115.00, 204, 139.4, 36, 6.00, 4.9628, 109, 264
51, 1, 23.4, 87.00, 220, 108.8, 93, 2.00, 4.5109, 82, 87
50, 2, 29.2, 119.00, 162, 85.2, 54, 3.00, 4.7362, 95, 202
59, 2, 27.2, 107.00, 158, 102.0, 39, 4.00, 4.4427, 93, 127
52, 1, 27.0, 78.33, 134, 73.0, 44, 3.05, 4.4427, 69, 182
69, 2, 24.5, 108.00, 243, 136.4, 40, 6.00, 5.8081, 100, 241
53, 1, 24.1, 105.00, 184, 113.4, 46, 4.00, 4.8122, 95, 66
47, 2, 25.3, 98.00, 173, 105.6, 44, 4.00, 4.7622, 108, 94
52, 1, 28.8, 113.00, 280, 174.0, 67, 4.00, 5.2730, 86, 283
39, 1, 20.9, 95.00, 150, 65.6, 68, 2.00, 4.4067, 95, 64
67, 2, 23.0, 70.00, 184, 128.0, 35, 5.00, 4.6540, 99, 102
59, 2, 24.1, 96.00, 170, 98.6, 54, 3.00, 4.4659, 85, 200
51, 2, 28.1, 106.00, 202, 122.2, 55, 4.00, 4.8203, 87, 265
23, 2, 18.0, 78.00, 171, 96.0, 48, 4.00, 4.9053, 92, 94
68, 1, 25.9, 93.00, 253, 181.2, 53, 5.00, 4.5433, 98, 230
44, 1, 21.5, 85.00, 157, 92.2, 55, 3.00, 3.8918, 84, 181
60, 2, 24.3, 103.00, 141, 86.6, 33, 4.00, 4.6728, 78, 156
52, 1, 24.5, 90.00, 198, 129.0, 29, 7.00, 5.2983, 86, 233
38, 1, 21.3, 72.00, 165, 60.2, 88, 2.00, 4.4308, 90, 60
61, 1, 25.8, 90.00, 280, 195.4, 55, 5.00, 4.9972, 90, 219
68, 2, 24.8, 101.00, 221, 151.4, 60, 4.00, 3.8712, 87, 80
28, 2, 31.5, 83.00, 228, 149.4, 38, 6.00, 5.3132, 83, 68
65, 2, 33.5, 102.00, 190, 126.2, 35, 5.00, 4.9698, 102, 332
69, 1, 28.1, 113.00, 234, 142.8, 52, 4.00, 5.2781, 77, 248
51, 1, 24.3, 85.33, 153, 71.6, 71, 2.15, 3.9512, 82, 84
29, 1, 35.0, 98.33, 204, 142.6, 50, 4.08, 4.0431, 91, 200
55, 2, 23.5, 93.00, 177, 126.8, 41, 4.00, 3.8286, 83, 55
34, 2, 30.0, 83.00, 185, 107.2, 53, 3.00, 4.8203, 92, 85
67, 1, 20.7, 83.00, 170, 99.8, 59, 3.00, 4.0254, 77, 89
49, 1, 25.6, 76.00, 161, 99.8, 51, 3.00, 3.9318, 78, 31
55, 2, 22.9, 81.00, 123, 67.2, 41, 3.00, 4.3041, 88, 129
59, 2, 25.1, 90.00, 163, 101.4, 46, 4.00, 4.3567, 91, 83
53, 1, 33.2, 82.67, 186, 106.8, 46, 4.04, 5.1120, 102, 275
48, 2, 24.1, 110.00, 209, 134.6, 58, 4.00, 4.4067, 100, 65
52, 1, 29.5, 104.33, 211, 132.8, 49, 4.31, 4.9836, 98, 198
69, 1, 29.6, 122.00, 231, 128.4, 56, 4.00, 5.4510, 86, 236
60, 2, 22.8, 110.00, 245, 189.8, 39, 6.00, 4.3944, 88, 253
46, 2, 22.7, 83.00, 183, 125.8, 32, 6.00, 4.8363, 75, 124
51, 2, 26.2, 101.00, 161, 99.6, 48, 3.00, 4.2047, 88, 44
67, 2, 23.5, 96.00, 207, 138.2, 42, 5.00, 4.8978, 111, 172
49, 1, 22.1, 85.00, 136, 63.4, 62, 2.19, 3.9703, 72, 114
46, 2, 26.5, 94.00, 247, 160.2, 59, 4.00, 4.9345, 111, 142
47, 1, 32.4, 105.00, 188, 125.0, 46, 4.09, 4.4427, 99, 109
75, 1, 30.1, 78.00, 222, 154.2, 44, 5.05, 4.7791, 97, 180
28, 1, 24.2, 93.00, 174, 106.4, 54, 3.00, 4.2195, 84, 144
65, 2, 31.3, 110.00, 213, 128.0, 47, 5.00, 5.2470, 91, 163
42, 1, 30.1, 91.00, 182, 114.8, 49, 4.00, 4.5109, 82, 147
51, 1, 24.5, 79.00, 212, 128.6, 65, 3.00, 4.5218, 91, 97
53, 2, 27.7, 95.00, 190, 101.8, 41, 5.00, 5.4638, 101, 220
54, 1, 23.2, 110.67, 238, 162.8, 48, 4.96, 4.9127, 108, 190
73, 1, 27.0, 102.00, 211, 121.0, 67, 3.00, 4.7449, 99, 109
54, 1, 26.8, 108.00, 176, 80.6, 67, 3.00, 4.9558, 106, 191
42, 1, 29.2, 93.00, 249, 174.2, 45, 6.00, 5.0039, 92, 122
75, 1, 31.2, 117.67, 229, 138.8, 29, 7.90, 5.7236, 106, 230
55, 2, 32.1, 112.67, 207, 92.4, 25, 8.28, 6.1048, 111, 242
68, 2, 25.7, 109.00, 233, 112.6, 35, 7.00, 6.0568, 105, 248
57, 1, 26.9, 98.00, 246, 165.2, 38, 7.00, 5.3660, 96, 249
48, 1, 31.4, 75.33, 242, 151.6, 38, 6.37, 5.5683, 103, 192
61, 2, 25.6, 85.00, 184, 116.2, 39, 5.00, 4.9698, 98, 131
69, 1, 37.0, 103.00, 207, 131.4, 55, 4.00, 4.6347, 90, 237
38, 1, 32.6, 77.00, 168, 100.6, 47, 4.00, 4.6250, 96, 78
45, 2, 21.2, 94.00, 169, 96.8, 55, 3.00, 4.4543, 102, 135
51, 2, 29.2, 107.00, 187, 139.0, 32, 6.00, 4.3820, 95, 244
71, 2, 24.0, 84.00, 138, 85.8, 39, 4.00, 4.1897, 90, 199
57, 1, 36.1, 117.00, 181, 108.2, 34, 5.00, 5.2679, 100, 270
56, 2, 25.8, 103.00, 177, 114.4, 34, 5.00, 4.9628, 99, 164
32, 2, 22.0, 88.00, 137, 78.6, 48, 3.00, 3.9512, 78, 72
50, 1, 21.9, 91.00, 190, 111.2, 67, 3.00, 4.0775, 77, 96
43, 1, 34.3, 84.00, 256, 172.6, 33, 8.00, 5.5294, 104, 306
54, 2, 25.2, 115.00, 181, 120.0, 39, 5.00, 4.7005, 92, 91
31, 1, 23.3, 85.00, 190, 130.8, 43, 4.00, 4.3944, 77, 214
56, 1, 25.7, 80.00, 244, 151.6, 59, 4.00, 5.1180, 95, 95
44, 1, 25.1, 133.00, 182, 113.0, 55, 3.00, 4.2485, 84, 216
57, 2, 31.9, 111.00, 173, 116.2, 41, 4.00, 4.3694, 87, 263

Test data:

# diabetes_test_100.txt
#
64, 2, 28.4, 111.00, 184, 127.0, 41, 4.00, 4.3820, 97, 178
43, 1, 28.1, 121.00, 192, 121.0, 60, 3.00, 4.0073, 93, 113
19, 1, 25.3, 83.00, 225, 156.6, 46, 5.00, 4.7185, 84, 200
71, 2, 26.1, 85.00, 220, 152.4, 47, 5.00, 4.6347, 91, 139
50, 2, 28.0, 104.00, 282, 196.8, 44, 6.00, 5.3279, 95, 139
59, 2, 23.6, 73.00, 180, 107.4, 51, 4.00, 4.6821, 84, 88
57, 1, 24.5, 93.00, 186, 96.6, 71, 3.00, 4.5218, 91, 148
49, 2, 21.0, 82.00, 119, 85.4, 23, 5.00, 3.9703, 74, 88
41, 2, 32.0, 126.00, 198, 104.2, 49, 4.00, 5.4116, 124, 243
25, 2, 22.6, 85.00, 130, 71.0, 48, 3.00, 4.0073, 81, 71
52, 2, 19.7, 81.00, 152, 53.4, 82, 2.00, 4.4188, 82, 77
34, 1, 21.2, 84.00, 254, 113.4, 52, 5.00, 6.0936, 92, 109
42, 2, 30.6, 101.00, 269, 172.2, 50, 5.00, 5.4553, 106, 272
28, 2, 25.5, 99.00, 162, 101.6, 46, 4.00, 4.2767, 94, 60
47, 2, 23.3, 90.00, 195, 125.8, 54, 4.00, 4.3307, 73, 54
32, 2, 31.0, 100.00, 177, 96.2, 45, 4.00, 5.1874, 77, 221
43, 1, 18.5, 87.00, 163, 93.6, 61, 2.67, 3.7377, 80, 90
59, 2, 26.9, 104.00, 194, 126.6, 43, 5.00, 4.8040, 106, 311
53, 1, 28.3, 101.00, 179, 107.0, 48, 4.00, 4.7875, 101, 281
60, 1, 25.7, 103.00, 158, 84.6, 64, 2.00, 3.8501, 97, 182
54, 2, 36.1, 115.00, 163, 98.4, 43, 4.00, 4.6821, 101, 321
35, 2, 24.1, 94.67, 155, 97.4, 32, 4.84, 4.8520, 94, 58
49, 2, 25.8, 89.00, 182, 118.6, 39, 5.00, 4.8040, 115, 262
58, 1, 22.8, 91.00, 196, 118.8, 48, 4.00, 4.9836, 115, 206
36, 2, 39.1, 90.00, 219, 135.8, 38, 6.00, 5.4205, 103, 233
46, 2, 42.2, 99.00, 211, 137.0, 44, 5.00, 5.0106, 99, 242
44, 2, 26.6, 99.00, 205, 109.0, 43, 5.00, 5.5797, 111, 123
46, 1, 29.9, 83.00, 171, 113.0, 38, 4.50, 4.5850, 98, 167
54, 1, 21.0, 78.00, 188, 107.4, 70, 3.00, 3.9703, 73, 63
63, 2, 25.5, 109.00, 226, 103.2, 46, 5.00, 5.9506, 87, 197
41, 2, 24.2, 90.00, 199, 123.6, 57, 4.00, 4.5218, 86, 71
28, 1, 25.4, 93.00, 141, 79.0, 49, 3.00, 4.1744, 91, 168
19, 1, 23.2, 75.00, 143, 70.4, 52, 3.00, 4.6347, 72, 140
61, 2, 26.1, 126.00, 215, 129.8, 57, 4.00, 4.9488, 96, 217
48, 1, 32.7, 93.00, 276, 198.6, 43, 6.42, 5.1475, 91, 121
54, 2, 27.3, 100.00, 200, 144.0, 33, 6.00, 4.7449, 76, 235
53, 2, 26.6, 93.00, 185, 122.4, 36, 5.00, 4.8903, 82, 245
48, 1, 22.8, 101.00, 110, 41.6, 56, 2.00, 4.1271, 97, 40
53, 1, 28.8, 111.67, 145, 87.2, 46, 3.15, 4.0775, 85, 52
29, 2, 18.1, 73.00, 158, 99.0, 41, 4.00, 4.4998, 78, 104
62, 1, 32.0, 88.00, 172, 69.0, 38, 4.00, 5.7838, 100, 132
50, 2, 23.7, 92.00, 166, 97.0, 52, 3.00, 4.4427, 93, 88
58, 2, 23.6, 96.00, 257, 171.0, 59, 4.00, 4.9053, 82, 69
55, 2, 24.6, 109.00, 143, 76.4, 51, 3.00, 4.3567, 88, 219
54, 1, 22.6, 90.00, 183, 104.2, 64, 3.00, 4.3041, 92, 72
36, 1, 27.8, 73.00, 153, 104.4, 42, 4.00, 3.4965, 73, 201
63, 2, 24.1, 111.00, 184, 112.2, 44, 4.00, 4.9345, 82, 110
47, 2, 26.5, 70.00, 181, 104.8, 63, 3.00, 4.1897, 70, 51
51, 2, 32.8, 112.00, 202, 100.6, 37, 5.00, 5.7746, 109, 277
42, 1, 19.9, 76.00, 146, 83.2, 55, 3.00, 3.6636, 79, 63
37, 2, 23.6, 94.00, 205, 138.8, 53, 4.00, 4.1897, 107, 118
28, 1, 22.1, 82.00, 168, 100.6, 54, 3.00, 4.2047, 86, 69
58, 1, 28.1, 111.00, 198, 80.6, 31, 6.00, 6.0684, 93, 273
32, 1, 26.5, 86.00, 184, 101.6, 53, 4.00, 4.9904, 78, 258
25, 2, 23.5, 88.00, 143, 80.8, 55, 3.00, 3.5835, 83, 43
63, 1, 26.0, 85.67, 155, 78.2, 46, 3.37, 5.0370, 97, 198
52, 1, 27.8, 85.00, 219, 136.0, 49, 4.00, 5.1358, 75, 242
# diabetes_test_100.txt
# 10 predictors, target y
# 100 items
#
65, 2, 28.5, 109.00, 201, 123.0, 46, 4.00, 5.0752, 96, 232
42, 1, 30.6, 121.00, 176, 92.8, 69, 3.00, 4.2627, 89, 175
53, 1, 22.2, 78.00, 164, 81.0, 70, 2.00, 4.1744, 101, 93
79, 2, 23.3, 88.00, 186, 128.4, 33, 6.00, 4.8122, 102, 168
43, 1, 35.4, 93.00, 185, 100.2, 44, 4.00, 5.3181, 101, 275
44, 1, 31.4, 115.00, 165, 97.6, 52, 3.00, 4.3438, 89, 293
62, 2, 37.8, 119.00, 113, 51.0, 31, 4.00, 5.0434, 84, 281
33, 1, 18.9, 70.00, 162, 91.8, 59, 3.00, 4.0254, 58, 72
56, 1, 35.0, 79.33, 195, 140.8, 42, 4.64, 4.1109, 96, 140
66, 1, 21.7, 126.00, 212, 127.8, 45, 4.71, 5.2781, 101, 189
34, 2, 25.3, 111.00, 230, 162.0, 39, 6.00, 4.9767, 90, 181
46, 2, 23.8, 97.00, 224, 139.2, 42, 5.00, 5.3660, 81, 209
50, 1, 31.8, 82.00, 136, 69.2, 55, 2.00, 4.0775, 85, 136
69, 1, 34.3, 113.00, 200, 123.8, 54, 4.00, 4.7095, 112, 261
34, 1, 26.3, 87.00, 197, 120.0, 63, 3.00, 4.2485, 96, 113
71, 2, 27.0, 93.33, 269, 190.2, 41, 6.56, 5.2417, 93, 131
47, 1, 27.2, 80.00, 208, 145.6, 38, 6.00, 4.8040, 92, 174
41, 1, 33.8, 123.33, 187, 127.0, 45, 4.16, 4.3175, 100, 257
34, 1, 33.0, 73.00, 178, 114.6, 51, 3.49, 4.1271, 92, 55
51, 1, 24.1, 87.00, 261, 175.6, 69, 4.00, 4.4067, 93, 84
43, 1, 21.3, 79.00, 141, 78.8, 53, 3.00, 3.8286, 90, 42
55, 1, 23.0, 94.67, 190, 137.6, 38, 5.00, 4.2767, 106, 146
59, 2, 27.9, 101.00, 218, 144.2, 38, 6.00, 5.1874, 95, 212
27, 2, 33.6, 110.00, 246, 156.6, 57, 4.00, 5.0876, 89, 233
51, 2, 22.7, 103.00, 217, 162.4, 30, 7.00, 4.8122, 80, 91
49, 2, 27.4, 89.00, 177, 113.0, 37, 5.00, 4.9053, 97, 111
27, 1, 22.6, 71.00, 116, 43.4, 56, 2.00, 4.4188, 79, 152
57, 2, 23.2, 107.33, 231, 159.4, 41, 5.63, 5.0304, 112, 120
39, 2, 26.9, 93.00, 136, 75.4, 48, 3.00, 4.1431, 99, 67
62, 2, 34.6, 120.00, 215, 129.2, 43, 5.00, 5.3660, 123, 310
37, 1, 23.3, 88.00, 223, 142.0, 65, 3.40, 4.3567, 82, 94
46, 1, 21.1, 80.00, 205, 144.4, 42, 5.00, 4.5326, 87, 183
68, 2, 23.5, 101.00, 162, 85.4, 59, 3.00, 4.4773, 91, 66
51, 1, 31.5, 93.00, 231, 144.0, 49, 4.70, 5.2523, 117, 173
41, 1, 20.8, 86.00, 223, 128.2, 83, 3.00, 4.0775, 89, 72
53, 1, 26.5, 97.00, 193, 122.4, 58, 3.00, 4.1431, 99, 49
45, 1, 24.2, 83.00, 177, 118.4, 45, 4.00, 4.2195, 82, 64
33, 1, 19.5, 80.00, 171, 85.4, 75, 2.00, 3.9703, 80, 48
60, 2, 28.2, 112.00, 185, 113.8, 42, 4.00, 4.9836, 93, 178
47, 2, 24.9, 75.00, 225, 166.0, 42, 5.00, 4.4427, 102, 104
60, 2, 24.9, 99.67, 162, 106.6, 43, 3.77, 4.1271, 95, 132
36, 1, 30.0, 95.00, 201, 125.2, 42, 4.79, 5.1299, 85, 220
36, 1, 19.6, 71.00, 250, 133.2, 97, 3.00, 4.5951, 92, 57