Neural Network Regression with Arctan Hidden Node Activation Using C#

Neural network regressors have two activation functions. One activation function is used when computing the values of nodes in the hidden layer(s), and one function is used when computing the value of the output node.

When training a neural network using back-propagaion and stochastic gradient descent (SGD), you need to compute the calculus derivatives of the activation functions.

The output node usually uses the identity activation function. It’s derivative is always the constant 1.0. For regression, hidden node activation is usually tanh. But many other functions are possible. In the early 2010s researchers, including me, investigated alternaives to tanh hidden node activation.

The reason you don’t ever hear about these alternatives to tanh is that they don’t work better than tanh, and they are more complicated to implement.

One Sunday evening, I decided to visit one of my investigations from 2015. See the article at https://visualstudiomagazine.com/articles/2015/06/01/alternative-activation-functions.aspx.

For my investigation, I started with a C# implementation of neural network regression with exactly two hidden layers that used tanh and identity activations. I replaced the tanh activation with arctan activation. As expected, there were no significant differences in the predctive power of the two approaches.

Output of the standard tanh indentity version:

Creating 5-(10-10)-1 tanh() identity() neural
 network regressor

Setting lrnRate = 0.0600
Setting maxEpochs = 30000
Setting decay = 0.00000100

Starting training
epoch:      0  MSE = 0.0354  acc = 0.0600
. . .
epoch:  24000  MSE = 0.0001  acc = 0.8000

Evaluating model

Accuracy (5%) on train data = 0.8100
Accuracy (5%) on test data  = 0.8000

MSE on train data = 0.0001
MSE on test data = 0.0002

And the output of my arctan identity version:

Creating 5-(10-10)-1 arctan() identity()
 neural network regressor

Setting lrnRate = 0.0600
Setting maxEpochs = 30000
Setting decay = 0.00000100

Starting training
epoch:      0  MSE = 0.0354  acc = 0.0600
. . .
epoch:  24000  MSE = 0.0001  acc = 0.7900
Done

Evaluating model

Accuracy (5%) on train data = 0.8050
Accuracy (5%) on test data  = 0.7750

MSE on train data = 0.0001
MSE on test data = 0.0002

Implementing arctan hidden node activation is trickier than it might seem. It would take several pages to explain fully, but breifly:

If some ordinary math function y = 3x^2 + 7x, then the caculus derivative is y’ = 6x + 7. For almost any function of x, the derivative will have an x term.

But if y = tanh(x), due to some lucky math identites, the calculus derivative is y’ = (1 – y) * (1 + y). Notice there is no x component, so the derivative is drop-dead easy to compute.

But if y = arctan(x), the calculus derivative is y’ = 1 / (1 + x^2). This is somewhat more difficult to compute because the x value is not immediately available — it must be saved during the forward pass of training.

A good trip down memory lane for me.



I’m a big fan of science fiction movies from the 1950s. Many of these movies showcased the latest developments in technology, especially jet aircraft.

Top Left: The North American B-45 Tornado was the first operational U.S. jet bomber. It was in service from 1948 to 1956, when it was replaced with the much superior B-47 Stratojet, the B-52 Stratofortress, and the B-58 Hustler,

Top Right: In “War of the Worlds” (1953), Martians invade Earth and quickly cut all global communication. Earth’s communications between world leaders are maintained using jet aircraft, including the B-45, as couriers.

Bottom Left: In “Target Earth” (1954), aliens from Venus invade Earth using 8-foot tall robots with death rays. In Chicago, a small group of people miss evacuation. The military plans to drop nuclear bombs on the almost-empty Chicago to stop the invaders, but the bombers, including a B-45 escorted by a P-80 fighter, are all destroyed by the robots.

Bottom Right: In “The Magnetic Monster” (1953), a rogue scientist creates a new isotope that absorbs any kind of energy doubles in size and mass every 11 hours. Earth is doomed. Scientists use a B-45 to travel from the U.S. to a Canadian experimental power generator, which is used to destroy the menace.


Demo program. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols. (My blog editor chokes on symbols).

using System;
using System.IO;
using System.Collections.Generic;

namespace NeuralNetworkBistratumArctanRegression
{
  internal class NeuralNetworkProgram
  {
    static void Main(string[] args)
    {
      Console.WriteLine("\nNeural network bistratum" +
        " arctan() identity() regression using C# ");

      // 1. load data
      Console.WriteLine("\nLoading synthetic train (200)" +
        " and test (40) data ");
      string trainFile =
        "..\\..\\..\\Data\\synthetic_train_200.txt";
      int[] colsX = new int[] { 0, 1, 2, 3, 4 };
      int colY = 5;
      double[][] trainX = MatLoad(trainFile, colsX, ',', "#");
      double[] trainY = MatToVec(MatLoad(trainFile,
        new int[] { colY }, ',', "#"));

      string testFile =
        "..\\..\\..\\Data\\synthetic_test_40.txt";
      double[][] testX = MatLoad(testFile, colsX, ',', "#");
      double[] testY = MatToVec(MatLoad(testFile,
        new int[] { colY }, ',', "#"));
      Console.WriteLine("Done ");

      Console.WriteLine("\nFirst three train X: ");
      for (int i = 0; i "lt" 3; ++i)
        VecShow(trainX[i], 4, 8);

      Console.WriteLine("\nFirst three train y: ");
      for (int i = 0; i "lt" 3; ++i)
        Console.WriteLine(trainY[i].ToString("F4").
          PadLeft(8));

      // 2. create NN
      Console.WriteLine("\nCreating 5-(10-10)-1 arctan()" +
        " identity() neural network regressor ");
      NeuralNetworkBistratumRegressor nn =
         new NeuralNetworkBistratumRegressor(5, 10, 10, 1);

      // 3. train NN
      double lrnRate = 0.06;
      int maxEpochs = 30_000;
      double decay = 0.000001; // .81  .80  .0001 .0002

      Console.WriteLine("\nSetting lrnRate = " +
        lrnRate.ToString("F4"));
      Console.WriteLine("Setting maxEpochs = " +
        maxEpochs);
      Console.WriteLine("Setting decay = " +
        decay.ToString("F8"));
      Console.WriteLine("\nStarting training ");
      nn.Train(trainX, trainY, lrnRate, maxEpochs, decay);
      Console.WriteLine("Done ");

      // TODO: programmatically analyze weights and
      // biases for extreme values

      // 4. evaluate trained model
      Console.WriteLine("\nEvaluating model ");
      double trainAcc = nn.Accuracy(trainX, trainY, 0.05);
      double testAcc = nn.Accuracy(testX, testY, 0.05);
      Console.WriteLine("\nAccuracy (5%) on train data = " +
        trainAcc.ToString("F4"));
      Console.WriteLine("Accuracy (5%) on test data  = " +
        testAcc.ToString("F4"));

      double trainMSE = nn.MSE(trainX, trainY);
      double testMSE = nn.MSE(testX, testY);
      Console.WriteLine("\nMSE on train data = " +
        trainMSE.ToString("F4"));
      Console.WriteLine("MSE on test data = " +
        testMSE.ToString("F4"));

      // 5. use model
      Console.WriteLine("\nPredicting y for train[0] ");
      double[] x = trainX[0];
      double predY = nn.Predict(x);
      Console.WriteLine("Predicted y = " +
        predY.ToString("F4"));

      Console.WriteLine("\nEnd demo ");
      Console.ReadLine();

    } // Main

    // ------------------------------------------------------
    // helpers for Main(): MatLoad(), MatToVec(), VecShow()
    // ------------------------------------------------------

    static double[][] MatLoad(string fn, int[] usecols,
      char sep, string comment)
    {
      List"lt"double[]"gt" result =
        new List"lt"double[]"gt"();
      string line = "";
      FileStream ifs = new FileStream(fn, FileMode.Open);
      StreamReader sr = new StreamReader(ifs);
      while ((line = sr.ReadLine()) != null)
      {
        if (line.StartsWith(comment) == true)
          continue;
        string[] tokens = line.Split(sep);
        List"lt"double"gt" lst = new List"lt"double"gt"();
        for (int j = 0; j "lt" usecols.Length; ++j)
          lst.Add(double.Parse(tokens[usecols[j]]));
        double[] row = lst.ToArray();
        result.Add(row);
      }
      sr.Close(); ifs.Close();
      return result.ToArray();
    }

    static double[] MatToVec(double[][] M)
    {
      int nRows = M.Length;
      int nCols = M[0].Length;
      double[] result = new double[nRows * nCols];
      int k = 0;
      for (int i = 0; i "lt" nRows; ++i)
        for (int j = 0; j "lt" nCols; ++j)
          result[k++] = M[i][j];
      return result;
    }

    static void VecShow(double[] vec, int dec, int wid)
    {
      for (int i = 0; i "lt" vec.Length; ++i)
        Console.Write(vec[i].ToString("F" + dec).
          PadLeft(wid));
      Console.WriteLine("");
    }

    // ------------------------------------------------------

  } // class Program

  public class NeuralNetworkBistratumRegressor
  {
    // two hidden layers
    public int numInput;
    public int numHiddenA;
    public int numHiddenB;
    public int numOutput;

    public double[] iNodes;  // input nodes
    public double[] aNodes;
    public double[] bNodes;
    public double[] oNodes;  // output nodes

    // pre-activation sums for hidden layers
    // needed to calculate the arctan derivative
    public double[] aSums;
    public double[] bSums;

    public double[][] iaWeights; // input to hidden A
    public double[][] abWeights; // hidden A to hidden B
    public double[][] boWeights; // hidden B to output

    public double[] aBiases;
    public double[] bBiases;
    public double[] oBiases;

    private Random rnd; // wt init and train shuffle

    // ------------------------------------------------------

    public NeuralNetworkBistratumRegressor(int numInput,
      int numHiddenA, int numHiddenB, int numOutput,
      int seed = 0)
    {
      this.numInput = numInput;
      this.numHiddenA = numHiddenA;
      this.numHiddenB = numHiddenB;
      this.numOutput = numOutput; // 1 for regression

      this.iNodes = new double[numInput];

      this.iaWeights = MatMake(numInput, numHiddenA);
      this.aBiases = new double[numHiddenA];
      this.aNodes = new double[numHiddenA];
      this.aSums = new double[numHiddenA];

      this.abWeights = MatMake(numHiddenA, numHiddenB);
      this.bBiases = new double[numHiddenB];
      this.bNodes = new double[numHiddenB];
      this.bSums = new double[numHiddenB];

      this.boWeights = MatMake(numHiddenB, numOutput);
      this.oBiases = new double[numOutput];
      this.oNodes = new double[numOutput];

      this.rnd = new Random(seed);
    }

    // ------------------------------------------------------

    private static double[][] MatMake(int nRows, int nCols)
    {
      double[][] result = new double[nRows][];

      for (int i = 0; i "lt" nRows; ++i)
        result[i] = new double[nCols];

      return result;
    }

    // ------------------------------------------------------

    public double Predict(double[] x)
    {
      // copy input into iNodes
      for (int i = 0; i "lt" this.numInput; ++i)
        this.iNodes[i] = x[i];

      // ----------------------------------------------------
      // compute hidden layer A node values
      // ----------------------------------------------------

      for (int j = 0; j "lt" numHiddenA; ++j)
      {
        double sum = 0.0;

        for (int i = 0; i "lt" this.numInput; ++i)
          sum += this.iNodes[i] * this.iaWeights[i][j];

        sum += this.aBiases[j];

        // Save pre-activation value.
        // Needed later when calculating the derivative.
        this.aSums[j] = sum;

        // Arctan activation
        this.aNodes[j] = Arctan(sum);
      }

      // ----------------------------------------------------
      // compute hidden layer B node values
      // ----------------------------------------------------

      for (int j = 0; j "lt" this.numHiddenB; ++j)
      {
        double sum = 0.0;

        for (int i = 0; i "lt" this.numHiddenA; ++i)
          sum += this.aNodes[i] * this.abWeights[i][j];

        sum += this.bBiases[j];

        // Save pre-activation value.
        // Needed later when calculating the derivative.
        this.bSums[j] = sum;

        // Arctan activation
        this.bNodes[j] = Arctan(sum);
      }

      // ----------------------------------------------------
      // compute output layer node
      // ----------------------------------------------------

      for (int j = 0; j "lt" this.numOutput; ++j)
      {
        double sum = 0.0;

        for (int i = 0; i "lt" this.numHiddenB; ++i)
          sum += this.bNodes[i] * this.boWeights[i][j];

        sum += this.oBiases[j];

        // Identity activation
        this.oNodes[j] = Identity(sum);
      }

      return this.oNodes[0];  // a single value
    }

    // ------------------------------------------------------

    private static double Arctan(double x)
    {
      if (x "lt" -10.0)
        return -Math.PI / 2;

      else if (x "gt" 10.0)
        return Math.PI / 2;

      else
        return Math.Atan(x);
    }

    // ------------------------------------------------------

    private static double ArctanDerivative(double x)
    {
      // Arctan() is clipped outside [-10, +10].
      // Therefore the derivative of the implemented
      // function is zero in those regions.

      if (x "lt" -10.0 || x "gt" 10.0)
        return 0.0;

      return 1.0 / (1.0 + x * x);
    }

    // ------------------------------------------------------

    private static double Identity(double x)
    {
      return x;
    }

    // ------------------------------------------------------

    public void Train(double[][] trainX, double[] trainY,
      double lrnRate, int maxEpochs, double decay)
    {
      // 'online' aka batch-1 SGD training (not batch)

      // init weights, leave biases at 0
      double lo = -0.01;
      double hi = 0.01;

      for (int i = 0; i "lt" this.numInput; ++i)
        for (int j = 0; j "lt" this.numHiddenA; ++j)
          this.iaWeights[i][j] =
            (hi - lo) * this.rnd.NextDouble() + lo;

      for (int i = 0; i "lt" this.numHiddenA; ++i)
        for (int j = 0; j "lt" this.numHiddenB; ++j)
          this.abWeights[i][j] =
            (hi - lo) * this.rnd.NextDouble() + lo;

      for (int i = 0; i "lt" this.numHiddenB; ++i)
        for (int j = 0; j "lt" this.numOutput; ++j)
          this.boWeights[i][j] =
            (hi - lo) * this.rnd.NextDouble() + lo;

      // ----------------------------------------------------
      // gradients
      // ----------------------------------------------------

      double[][] boGrads =
        MatMake(numHiddenB, numOutput);

      double[][] abGrads =
        MatMake(numHiddenA, numHiddenB);

      double[][] iaGrads =
        MatMake(numInput, numHiddenA);

      double[] oBiasGrads =
        new double[numOutput];

      double[] bBiasGrads =
        new double[numHiddenB];

      double[] aBiasGrads =
        new double[numHiddenA];

      // ----------------------------------------------------
      // signals
      // ----------------------------------------------------

      // each output and hidden node has a 'signal'
      // which is gradient without associated input
      // (lower case delta in Wikipedia)

      double[] oSignals =
        new double[numOutput];

      double[] bSignals =
        new double[numHiddenB];

      double[] aSignals =
        new double[numHiddenA];

      // ----------------------------------------------------
      // indices used to shuffle training data
      // ----------------------------------------------------

      int[] indices = new int[trainX.Length];

      for (int i = 0; i "lt" indices.Length; ++i)
        indices[i] = i;

      int freq = (int)(maxEpochs / 5);

      // ----------------------------------------------------
      // training loop
      // ----------------------------------------------------

      for (int epoch = 0; epoch "lt" maxEpochs; ++epoch)
      {
        this.Shuffle(indices);

        for (int ii = 0; ii "lt" trainX.Length; ++ii)
        {
          int idx = indices[ii];

          double[] x = trainX[idx];
          double actualY = trainY[idx];

          // Forward pass
          double predY = this.Predict(x);

          // ------------------------------------------------
          // output node signals
          // ------------------------------------------------

          for (int k = 0; k "lt" this.numOutput; ++k)
          {
            double error = predY - actualY;

            // derivative of identity activation
            double derivative = 1.0;

            oSignals[k] = error * derivative;
          }

          // ------------------------------------------------
          // B node signals
          // ------------------------------------------------

          for (int j = 0; j "lt" numHiddenB; ++j)
          {
            // derivative of the implemented arctan
            // activation
            double derivative =
              ArctanDerivative(this.bSums[j]);

            double sum = 0.0;

            for (int k = 0; k "lt" numOutput; ++k)
              sum += oSignals[k] * this.boWeights[j][k];

            bSignals[j] = derivative * sum;
          }

          // ------------------------------------------------
          // A node signals
          // ------------------------------------------------

          for (int j = 0; j "lt" numHiddenA; ++j)
          {
            // derivative of the implemented arctan
            // activation
            double derivative =
              ArctanDerivative(this.aSums[j]);

            double sum = 0.0;

            for (int k = 0; k "lt" numHiddenB; ++k)
              sum += bSignals[k] * this.abWeights[j][k];

            aSignals[j] = derivative * sum;
          }

          // ------------------------------------------------
          // calculate gradients
          // ------------------------------------------------

          for (int i = 0; i "lt" numInput; ++i)
            for (int j = 0; j "lt" numHiddenA; ++j)
              iaGrads[i][j] =
                this.iNodes[i] * aSignals[j];

          for (int i = 0; i "lt" numHiddenA; ++i)
            for (int j = 0; j "lt" numHiddenB; ++j)
              abGrads[i][j] =
                this.aNodes[i] * bSignals[j];

          for (int i = 0; i "lt" numHiddenB; ++i)
            for (int j = 0; j "lt" numOutput; ++j)
              boGrads[i][j] =
                this.bNodes[i] * oSignals[j];

          // ------------------------------------------------
          // bias gradients
          // ------------------------------------------------

          for (int j = 0; j "lt" numHiddenA; ++j)
            aBiasGrads[j] = aSignals[j];

          for (int j = 0; j "lt" numHiddenB; ++j)
            bBiasGrads[j] = bSignals[j];

          for (int j = 0; j "lt" numOutput; ++j)
            oBiasGrads[j] = oSignals[j];

          // ------------------------------------------------
          // weight decay
          // ------------------------------------------------

          for (int i = 0; i "lt" numInput; ++i)
            for (int j = 0; j "lt" numHiddenA; ++j)
              this.iaWeights[i][j] *= (1 - decay);

          for (int i = 0; i "lt" numHiddenA; ++i)
            for (int j = 0; j "lt" numHiddenB; ++j)
              this.abWeights[i][j] *= (1 - decay);

          for (int i = 0; i "lt" numHiddenB; ++i)
            for (int j = 0; j "lt" numOutput; ++j)
              this.boWeights[i][j] *= (1 - decay);

          // ------------------------------------------------
          // update weights
          // ------------------------------------------------

          for (int i = 0; i "lt" numInput; ++i)
            for (int j = 0; j "lt" numHiddenA; ++j)
              this.iaWeights[i][j] -=
                iaGrads[i][j] * lrnRate;

          for (int i = 0; i "lt" numHiddenA; ++i)
            for (int j = 0; j "lt" numHiddenB; ++j)
              this.abWeights[i][j] -=
                abGrads[i][j] * lrnRate;

          for (int i = 0; i "lt" numHiddenB; ++i)
            for (int j = 0; j "lt" numOutput; ++j)
              this.boWeights[i][j] -=
                boGrads[i][j] * lrnRate;

          // ------------------------------------------------
          // update biases
          // ------------------------------------------------

          for (int j = 0; j "lt" numHiddenA; ++j)
            this.aBiases[j] -=
              aBiasGrads[j] * lrnRate;

          for (int j = 0; j "lt" numHiddenB; ++j)
            this.bBiases[j] -=
              bBiasGrads[j] * lrnRate;

          for (int j = 0; j "lt" numOutput; ++j)
            this.oBiases[j] -=
              oBiasGrads[j] * lrnRate;

        } // ii each train item

        // --------------------------------------------------
        // display progress
        // --------------------------------------------------

        if (epoch % freq == 0 && epoch "lt" maxEpochs)
        {
          double mse = this.MSE(trainX, trainY);
          double acc = this.Accuracy(trainX, trainY, 0.05);

          string s1 = "epoch: " +
            epoch.ToString().PadLeft(6);

          string s2 = "  MSE = " +
            mse.ToString("F4");

          string s3 = "  acc = " +
            acc.ToString("F4");

          Console.WriteLine(s1 + s2 + s3);
        }

      } // epoch

      return;
    }

    // ------------------------------------------------------

    private void Shuffle(int[] indices)
    {
      // Fisher-Yates algorithm

      for (int i = 0; i "lt" indices.Length; ++i)
      {
        int r = this.rnd.Next(i, indices.Length);

        int tmp = indices[r];
        indices[r] = indices[i];
        indices[i] = tmp;
      }
    }

    // ------------------------------------------------------

    public double Accuracy(double[][] dataX, double[] dataY,
      double pctClose)
    {
      // in this version, if target y is 0, only pred equal
      // exactly 0 is scored as a correct prediction

      int n = dataX.Length;

      int nCorrect = 0;
      int nWrong = 0;

      for (int i = 0; i "lt" n; ++i)
      {
        double predY = this.Predict(dataX[i]);
        double actualY = dataY[i];

        if (Math.Abs(predY - actualY) "lte"
          pctClose * Math.Abs(actualY))
          ++nCorrect;
        else
          ++nWrong;
      }

      return (nCorrect * 1.0) /
        (nCorrect + nWrong);
    }

    // ------------------------------------------------------

    public double MSE(double[][] dataX, double[] dataY)
    {
      // mean squared error

      int n = dataX.Length;
      double sum = 0.0;

      for (int i = 0; i "lt" n; ++i)
      {
        double predY = this.Predict(dataX[i]);
        double actualY = dataY[i];

        sum +=
          (predY - actualY) *
          (predY - actualY);
      }

      return sum / n;
    }

    // ------------------------------------------------------
    
  } // class NeuralNetworkBistratumRegressor

} // ns

Training data.

# synthetic_train_200.txt
#
-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
-0.8299, -0.9219, -0.6603,  0.7563, -0.8033,  0.7955
 0.0663,  0.3838, -0.3690,  0.3730,  0.6693,  0.3206
-0.9634,  0.5003,  0.9777,  0.4963, -0.4391,  0.7377
-0.1042,  0.8172, -0.4128, -0.4244, -0.7399,  0.4801
-0.9613,  0.3577, -0.5767, -0.4689, -0.0169,  0.6861
-0.7065,  0.1786,  0.3995, -0.7953, -0.1719,  0.5569
 0.3888, -0.1716, -0.9001,  0.0718,  0.3276,  0.2500
 0.1731,  0.8068, -0.7251, -0.7214,  0.6148,  0.3297
-0.2046, -0.6693,  0.8550, -0.3045,  0.5016,  0.2129
 0.2473,  0.5019, -0.3022, -0.4601,  0.7918,  0.2613
-0.1438,  0.9297,  0.3269,  0.2434, -0.7705,  0.5171
 0.1568, -0.1837, -0.5259,  0.8068,  0.1474,  0.3307
-0.9943,  0.2343, -0.3467,  0.0541,  0.7719,  0.5581
 0.2467, -0.9684,  0.8589,  0.3818,  0.9946,  0.1092
-0.6553, -0.7257,  0.8652,  0.3936, -0.8680,  0.7018
 0.8460,  0.4230, -0.7515, -0.9602, -0.9476,  0.1996
-0.9434, -0.5076,  0.7201,  0.0777,  0.1056,  0.5664
 0.9392,  0.1221, -0.9627,  0.6013, -0.5341,  0.1533
 0.6142, -0.2243,  0.7271,  0.4942,  0.1125,  0.1661
 0.4260,  0.1194, -0.9749, -0.8561,  0.9346,  0.2230
 0.1362, -0.5934, -0.4953,  0.4877, -0.6091,  0.3810
 0.6937, -0.5203, -0.0125,  0.2399,  0.6580,  0.1460
-0.6864, -0.9628, -0.8600, -0.0273,  0.2127,  0.5387
 0.9772,  0.1595, -0.2397,  0.1019,  0.4907,  0.1611
 0.3385, -0.4702, -0.8673, -0.2598,  0.2594,  0.2270
-0.8669, -0.4794,  0.6095, -0.6131,  0.2789,  0.4700
 0.0493,  0.8496, -0.4734, -0.8681,  0.4701,  0.3516
 0.8639, -0.9721, -0.5313,  0.2336,  0.8980,  0.1412
 0.9004,  0.1133,  0.8312,  0.2831, -0.2200,  0.1782
 0.0991,  0.8524,  0.8375, -0.2102,  0.9265,  0.2150
-0.6521, -0.7473, -0.7298,  0.0113, -0.9570,  0.7422
 0.6190, -0.3105,  0.8802,  0.1640,  0.7577,  0.1056
 0.6895,  0.8108, -0.0802,  0.0927,  0.5972,  0.2214
 0.1982, -0.9689,  0.1870, -0.1326,  0.6147,  0.1310
-0.3695,  0.7858,  0.1557, -0.6320,  0.5759,  0.3773
-0.1596,  0.3581,  0.8372, -0.9992,  0.9535,  0.2071
-0.2468,  0.9476,  0.2094,  0.6577,  0.1494,  0.4132
 0.1737,  0.5000,  0.7166,  0.5102,  0.3961,  0.2611
 0.7290, -0.3546,  0.3416, -0.0983, -0.2358,  0.1332
-0.3652,  0.2438, -0.1395,  0.9476,  0.3556,  0.4170
-0.6029, -0.1466, -0.3133,  0.5953,  0.7600,  0.4334
-0.4596, -0.4953,  0.7098,  0.0554,  0.6043,  0.2775
 0.1450,  0.4663,  0.0380,  0.5418,  0.1377,  0.2931
-0.8636, -0.2442, -0.8407,  0.9656, -0.6368,  0.7429
 0.6237,  0.7499,  0.3768,  0.1390, -0.6781,  0.2185
-0.5499,  0.1850, -0.3755,  0.8326,  0.8193,  0.4399
-0.4858, -0.7782, -0.6141, -0.0008,  0.4572,  0.4197
 0.7033, -0.1683,  0.2334, -0.5327, -0.7961,  0.1776
 0.0317, -0.0457, -0.6947,  0.2436,  0.0880,  0.3345
 0.5031, -0.5559,  0.0387,  0.5706, -0.9553,  0.3107
-0.3513,  0.7458,  0.6894,  0.0769,  0.7332,  0.3170
 0.2205,  0.5992, -0.9309,  0.5405,  0.4635,  0.3532
-0.4806, -0.4859,  0.2646, -0.3094,  0.5932,  0.3202
 0.9809, -0.3995, -0.7140,  0.8026,  0.0831,  0.1600
 0.9495,  0.2732,  0.9878,  0.0921,  0.0529,  0.1289
-0.9476, -0.6792,  0.4913, -0.9392, -0.2669,  0.5966
 0.7247,  0.3854,  0.3819, -0.6227, -0.1162,  0.1550
-0.5922, -0.5045, -0.4757,  0.5003, -0.0860,  0.5863
-0.8861,  0.0170, -0.5761,  0.5972, -0.4053,  0.7301
 0.6877, -0.2380,  0.4997,  0.0223,  0.0819,  0.1404
 0.9189,  0.6079, -0.9354,  0.4188, -0.0700,  0.1907
-0.1428, -0.7820,  0.2676,  0.6059,  0.3936,  0.2790
 0.5324, -0.3151,  0.6917, -0.1425,  0.6480,  0.1071
-0.8432, -0.9633, -0.8666, -0.0828, -0.7733,  0.7784
-0.9444,  0.5097, -0.2103,  0.4939, -0.0952,  0.6787
-0.0520,  0.6063, -0.1952,  0.8094, -0.9259,  0.4836
 0.5477, -0.7487,  0.2370, -0.9793,  0.0773,  0.1241
 0.2450,  0.8116,  0.9799,  0.4222,  0.4636,  0.2355
 0.8186, -0.1983, -0.5003, -0.6531, -0.7611,  0.1511
-0.4714,  0.6382, -0.3788,  0.9648, -0.4667,  0.5950
 0.0673, -0.3711,  0.8215, -0.2669, -0.1328,  0.2677
-0.9381,  0.4338,  0.7820, -0.9454,  0.0441,  0.5518
-0.3480,  0.7190,  0.1170,  0.3805, -0.0943,  0.4724
-0.9813,  0.1535, -0.3771,  0.0345,  0.8328,  0.5438
-0.1471, -0.5052, -0.2574,  0.8637,  0.8737,  0.3042
-0.5454, -0.3712, -0.6505,  0.2142, -0.1728,  0.5783
 0.6327, -0.6297,  0.4038, -0.5193,  0.1484,  0.1153
-0.5424,  0.3282, -0.0055,  0.0380, -0.6506,  0.6613
 0.1414,  0.9935,  0.6337,  0.1887,  0.9520,  0.2540
-0.9351, -0.8128, -0.8693, -0.0965, -0.2491,  0.7353
 0.9507, -0.6640,  0.9456,  0.5349,  0.6485,  0.1059
-0.0462, -0.9737, -0.2940, -0.0159,  0.4602,  0.2606
-0.0627, -0.0852, -0.7247, -0.9782,  0.5166,  0.2977
 0.0478,  0.5098, -0.0723, -0.7504, -0.3750,  0.3335
 0.0090,  0.3477,  0.5403, -0.7393, -0.9542,  0.4415
-0.9748,  0.3449,  0.3736, -0.1015,  0.8296,  0.4358
 0.2887, -0.9895, -0.0311,  0.7186,  0.6608,  0.2057
 0.1570, -0.4518,  0.1211,  0.3435, -0.2951,  0.3244
 0.7117, -0.6099,  0.4946, -0.4208,  0.5476,  0.1096
-0.2929, -0.5726,  0.5346, -0.3827,  0.4665,  0.2465
 0.4889, -0.5572, -0.5718, -0.6021, -0.7150,  0.2163
-0.7782,  0.3491,  0.5996, -0.8389, -0.5366,  0.6516
-0.5847,  0.8347,  0.4226,  0.1078, -0.3910,  0.6134
 0.8469,  0.4121, -0.0439, -0.7476,  0.9521,  0.1571
-0.6803, -0.5948, -0.1376, -0.1916, -0.7065,  0.7156
 0.2878,  0.5086, -0.5785,  0.2019,  0.4979,  0.2980
 0.2764,  0.1943, -0.4090,  0.4632,  0.8906,  0.2960
-0.8877,  0.6705, -0.6155, -0.2098, -0.3998,  0.7107
-0.8398,  0.8093, -0.2597,  0.0614, -0.0118,  0.6502
-0.8476,  0.0158, -0.4769, -0.2859, -0.7839,  0.7715
 0.5751, -0.7868,  0.9714, -0.6457,  0.1448,  0.1175
 0.4802, -0.7001,  0.1022, -0.5668,  0.5184,  0.1090
 0.4458, -0.6469,  0.7239, -0.9604,  0.7205,  0.0779
 0.5175,  0.4339,  0.9747, -0.4438, -0.9924,  0.2879
 0.8678,  0.7158,  0.4577,  0.0334,  0.4139,  0.1678
 0.5406,  0.5012,  0.2264, -0.1963,  0.3946,  0.2088
-0.9938,  0.5498,  0.7928, -0.5214, -0.7585,  0.7687
 0.7661,  0.0863, -0.4266, -0.7233, -0.4197,  0.1466
 0.2277, -0.3517, -0.0853, -0.1118,  0.6563,  0.1767
 0.3499, -0.5570, -0.0655, -0.3705,  0.2537,  0.1632
 0.7547, -0.1046,  0.5689, -0.0861,  0.3125,  0.1257
 0.8186,  0.2110,  0.5335,  0.0094, -0.0039,  0.1391
 0.6858, -0.8644,  0.1465,  0.8855,  0.0357,  0.1845
-0.4967,  0.4015,  0.0805,  0.8977,  0.2487,  0.4663
 0.6760, -0.9841,  0.9787, -0.8446, -0.3557,  0.1509
-0.1203, -0.4885,  0.6054, -0.0443, -0.7313,  0.4854
 0.8557,  0.7919, -0.0169,  0.7134, -0.1628,  0.2002
 0.0115, -0.6209,  0.9300, -0.4116, -0.7931,  0.4052
-0.7114, -0.9718,  0.4319,  0.1290,  0.5892,  0.3661
 0.3915,  0.5557, -0.1870,  0.2955, -0.6404,  0.2954
-0.3564, -0.6548, -0.1827, -0.5172, -0.1862,  0.4622
 0.2392, -0.4959,  0.5857, -0.1341, -0.2850,  0.2470
-0.3394,  0.3947, -0.4627,  0.6166, -0.4094,  0.5325
 0.7107,  0.7768, -0.6312,  0.1707,  0.7964,  0.2757
-0.1078,  0.8437, -0.4420,  0.2177,  0.3649,  0.4028
-0.3139,  0.5595, -0.6505, -0.3161, -0.7108,  0.5546
 0.4335,  0.3986,  0.3770, -0.4932,  0.3847,  0.1810
-0.2562, -0.2894, -0.8847,  0.2633,  0.4146,  0.4036
 0.2272,  0.2966, -0.6601, -0.7011,  0.0284,  0.2778
-0.0743, -0.1421, -0.0054, -0.6770, -0.3151,  0.3597
-0.4762,  0.6891,  0.6007, -0.1467,  0.2140,  0.4266
-0.4061,  0.7193,  0.3432,  0.2669, -0.7505,  0.6147
-0.0588,  0.9731,  0.8966,  0.2902, -0.6966,  0.4955
-0.0627, -0.1439,  0.1985,  0.6999,  0.5022,  0.3077
 0.1587,  0.8494, -0.8705,  0.9827, -0.8940,  0.4263
-0.7850,  0.2473, -0.9040, -0.4308, -0.8779,  0.7199
 0.4070,  0.3369, -0.2428, -0.6236,  0.4940,  0.2215
-0.0242,  0.0513, -0.9430,  0.2885, -0.2987,  0.3947
-0.5416, -0.1322, -0.2351, -0.0604,  0.9590,  0.3683
 0.1055,  0.7783, -0.2901, -0.5090,  0.8220,  0.2984
-0.9129,  0.9015,  0.1128, -0.2473,  0.9901,  0.4776
-0.9378,  0.1424, -0.6391,  0.2619,  0.9618,  0.5368
 0.7498, -0.0963,  0.4169,  0.5549, -0.0103,  0.1614
-0.2612, -0.7156,  0.4538, -0.0460, -0.1022,  0.3717
 0.7720,  0.0552, -0.1818, -0.4622, -0.8560,  0.1685
-0.4177,  0.0070,  0.9319, -0.7812,  0.3461,  0.3052
-0.0001,  0.5542, -0.7128, -0.8336, -0.2016,  0.3803
 0.5356, -0.4194, -0.5662, -0.9666, -0.2027,  0.1776
-0.2378,  0.3187, -0.8582, -0.6948, -0.9668,  0.5474
-0.1947, -0.3579,  0.1158,  0.9869,  0.6690,  0.2992
 0.3992,  0.8365, -0.9205, -0.8593, -0.0520,  0.3154
-0.0209,  0.0793,  0.7905, -0.1067,  0.7541,  0.1864
-0.4928, -0.4524, -0.3433,  0.0951, -0.5597,  0.6261
-0.8118,  0.7404, -0.5263, -0.2280,  0.1431,  0.6349
 0.0516, -0.8480,  0.7483,  0.9023,  0.6250,  0.1959
-0.3212,  0.1093,  0.9488, -0.3766,  0.3376,  0.2735
-0.3481,  0.5490, -0.3484,  0.7797,  0.5034,  0.4379
-0.5785, -0.9170, -0.3563, -0.9258,  0.3877,  0.4121
 0.3407, -0.1391,  0.5356,  0.0720, -0.9203,  0.3458
-0.3287, -0.8954,  0.2102,  0.0241,  0.2349,  0.3247
-0.1353,  0.6954, -0.0919, -0.9692,  0.7461,  0.3338
 0.9036, -0.8982, -0.5299, -0.8733, -0.1567,  0.1187
 0.7277, -0.8368, -0.0538, -0.7489,  0.5458,  0.0830
 0.9049,  0.8878,  0.2279,  0.9470, -0.3103,  0.2194
 0.7957, -0.1308, -0.5284,  0.8817,  0.3684,  0.2172
 0.4647, -0.4931,  0.2010,  0.6292, -0.8918,  0.3371
-0.7390,  0.6849,  0.2367,  0.0626, -0.5034,  0.7039
-0.1567, -0.8711,  0.7940, -0.5932,  0.6525,  0.1710
 0.7635, -0.0265,  0.1969,  0.0545,  0.2496,  0.1445
 0.7675,  0.1354, -0.7698, -0.5460,  0.1920,  0.1728
-0.5211, -0.7372, -0.6763,  0.6897,  0.2044,  0.5217
 0.1913,  0.1980,  0.2314, -0.8816,  0.5006,  0.1998
 0.8964,  0.0694, -0.6149,  0.5059, -0.9854,  0.1825
 0.1767,  0.7104,  0.2093,  0.6452,  0.7590,  0.2832
-0.3580, -0.7541,  0.4426, -0.1193, -0.7465,  0.5657
-0.5996,  0.5766, -0.9758, -0.3933, -0.9572,  0.6800
 0.9950,  0.1641, -0.4132,  0.8579,  0.0142,  0.2003
-0.4717, -0.3894, -0.2567, -0.5111,  0.1691,  0.4266
 0.3917, -0.8561,  0.9422,  0.5061,  0.6123,  0.1212
-0.0366, -0.1087,  0.3449, -0.1025,  0.4086,  0.2475
 0.3633,  0.3943,  0.2372, -0.6980,  0.5216,  0.1925
-0.5325, -0.6466, -0.2178, -0.3589,  0.6310,  0.3568
 0.2271,  0.5200, -0.1447, -0.8011, -0.7699,  0.3128
 0.6415,  0.1993,  0.3777, -0.0178, -0.8237,  0.2181
-0.5298, -0.0768, -0.6028, -0.9490,  0.4588,  0.4356
 0.6870, -0.1431,  0.7294,  0.3141,  0.1621,  0.1632
-0.5985,  0.0591,  0.7889, -0.3900,  0.7419,  0.2945
 0.3661,  0.7984, -0.8486,  0.7572, -0.6183,  0.3449
 0.6995,  0.3342, -0.3113, -0.6972,  0.2707,  0.1712
 0.2565,  0.9126,  0.1798, -0.6043, -0.1413,  0.2893
-0.3265,  0.9839, -0.2395,  0.9854,  0.0376,  0.4770
 0.2690, -0.1722,  0.9818,  0.8599, -0.7015,  0.3954
-0.2102, -0.0768,  0.1219,  0.5607, -0.0256,  0.3949
 0.8216, -0.9555,  0.6422, -0.6231,  0.3715,  0.0801
-0.2896,  0.9484, -0.7545, -0.6249,  0.7789,  0.4370
-0.9985, -0.5448, -0.7092, -0.5931,  0.7926,  0.5402

Test data.

# synthetic_test_40.txt
#
 0.7462,  0.4006, -0.0590,  0.6543, -0.0083,  0.1935
 0.8495, -0.2260, -0.0142, -0.4911,  0.7699,  0.1078
-0.2335, -0.4049,  0.4352, -0.6183, -0.7636,  0.5088
 0.1810, -0.5142,  0.2465,  0.2767, -0.3449,  0.3136
-0.8650,  0.7611, -0.0801,  0.5277, -0.4922,  0.7140
-0.2358, -0.7466, -0.5115, -0.8413, -0.3943,  0.4533
 0.4834,  0.2300,  0.3448, -0.9832,  0.3568,  0.1360
-0.6502, -0.6300,  0.6885,  0.9652,  0.8275,  0.3046
-0.3053,  0.5604,  0.0929,  0.6329, -0.0325,  0.4756
-0.7995,  0.0740, -0.2680,  0.2086,  0.9176,  0.4565
-0.2144, -0.2141,  0.5813,  0.2902, -0.2122,  0.4119
-0.7278, -0.0987, -0.3312, -0.5641,  0.8515,  0.4438
 0.3793,  0.1976,  0.4933,  0.0839,  0.4011,  0.1905
-0.8568,  0.9573, -0.5272,  0.3212, -0.8207,  0.7415
-0.5785,  0.0056, -0.7901, -0.2223,  0.0760,  0.5551
 0.0735, -0.2188,  0.3925,  0.3570,  0.3746,  0.2191
 0.1230, -0.2838,  0.2262,  0.8715,  0.1938,  0.2878
 0.4792, -0.9248,  0.5295,  0.0366, -0.9894,  0.3149
-0.4456,  0.0697,  0.5359, -0.8938,  0.0981,  0.3879
 0.8629, -0.8505, -0.4464,  0.8385,  0.5300,  0.1769
 0.1995,  0.6659,  0.7921,  0.9454,  0.9970,  0.2330
-0.0249, -0.3066, -0.2927, -0.4923,  0.8220,  0.2437
 0.4513, -0.9481, -0.0770, -0.4374, -0.9421,  0.2879
-0.3405,  0.5931, -0.3507, -0.3842,  0.8562,  0.3987
 0.9538,  0.0471,  0.9039,  0.7760,  0.0361,  0.1706
-0.0887,  0.2104,  0.9808,  0.5478, -0.3314,  0.4128
-0.8220, -0.6302,  0.0537, -0.1658,  0.6013,  0.4306
-0.4123, -0.2880,  0.9074, -0.0461, -0.4435,  0.5144
 0.0060,  0.2867, -0.7775,  0.5161,  0.7039,  0.3599
-0.7968, -0.5484,  0.9426, -0.4308,  0.8148,  0.2979
 0.7811,  0.8450, -0.6877,  0.7594,  0.2640,  0.2362
-0.6802, -0.1113, -0.8325, -0.6694, -0.6056,  0.6544
 0.3821,  0.1476,  0.7466, -0.5107,  0.2592,  0.1648
 0.7265,  0.9683, -0.9803, -0.4943, -0.5523,  0.2454
-0.9049, -0.9797, -0.0196, -0.9090, -0.4433,  0.6447
-0.4607,  0.1811, -0.2389,  0.4050, -0.0078,  0.5229
 0.2664, -0.2932, -0.4259, -0.7336,  0.8742,  0.1834
-0.4507,  0.1029, -0.6294, -0.1158, -0.6294,  0.6081
 0.8948, -0.0124,  0.9278,  0.2899, -0.0314,  0.1534
-0.1323, -0.8813, -0.0146, -0.0697,  0.6135,  0.2386
Posted in Machine Learning | Leave a comment

“The AI Trust Problem: Risks and Business Opportunities” on the Pure AI Web Site

I contributed some technical content and opinions to an article titled “The AI Trust Problem: Risks and Business Opportunities” in the November 2026 edition of the Pure AI web site. See https://pureai.com/articles/2026/11/03/the-ai-trust-problem-risks-and-business-opportunities.aspx.

AI deepfakes threaten our ability to trust photographs and other digital information, making independent verification and digital evidence authentication increasingly important.

As AI-generated content becomes more difficult to detect, new business opportunities are emerging for services that help organizations verify the authenticity of digital evidence.

Niche authentication services could combine forensic software, cryptographic verification and human review to assess insurance claims.

The Pure AI editors asked AI expert Dr. James McCaffrey to comment. McCaffrey was one of the original members of the Microsoft Deep Learning team — the group that evolved into Microsoft’s current AI efforts. McCaffrey observed, “One of the troubling consequences of the rise of increasingly difficult-to-detect deepfakes is that AI could weaken our overall trust in information.”

“When convincing fabrications become commonplace, people will doubt genuine photographs, audio recordings, and news reports. Someone caught making a false statement might even claim that the evidence against them was generated by AI.”

McCaffrey added, “A central challenge, therefore, is not simply learning to recognize fake content. It is developing better ways to establish what is true in a world where convincing evidence can be manufactured.”

“But the emergence of AI deepfakes provides several new business opportunities. Only the tech giants — Microsoft, Google, OpenAI, Meta, and so on, have the resources to create a universal digital trust authority. But there is room for small niche services.”

“For example, a startup with just two or three people could create a Digital Evidence Authentication Service for insurance claims. The idea is to specialize in verifying whether photographs and videos submitted for vehicle accidents and other claims are genuine or AI-generated. The startup could combine existing forensic software, cryptographic verification, and human review to sell subscription services to small and mid-sized insurance companies without developing its own AI detection technology.”



Most of my friends, and I, are fascinated by all kinds of strange things that don’t interest normal people. For reasons that are essentially unknown to me, I am intrigued by movies that feature a creature that makes noises that are translated to “chittering” in the closed caption text. The are an amazing number of chittering instances in science fiction movies, but you’ll only notice these chitterings if you’re actively looking for them.

Left: “No One Will Save You” (2023) is one of the most interesting science fiction films I have ever seen. Aliens invade a small town. A young woman named Brynn fights off several of the aliens and associated creatures. In a surprise twist ending, the aliens succeed in taking over the entire town using parasites, but decide to leave Brynn alone and she happily coexists with the infected townspeople. There are only five words of dialog in the entire movie. My overall grade is B.

Right: In “Splinter” (2008), a young couple, Seth and Polly, stop at remote gas station late at night. A deadly parasite can infect just about any living thing (including a severed hand). The parasite places weird biological splinters all over the infected animal. Seth and Polly survive in the end, but the station attendant and an escaped convict and his girlfriend are not so fortunate.


Posted in Miscellaneous | Leave a comment

NFL 2026 Week 11 Predictions – Zoltar Likes the Broncos to Destroy the Raiders

Zoltar is my NFL football prediction computer program. It uses a neural network and a kind of quasi-reinforcement learning. Here are Zoltar’s predictions for week #11 of the 2026 season.

Zoltar:      texans  by    6  opp =       colts    | Vegas:      texans  by  9.5
Zoltar:      chiefs  by   10  opp =   cardinals    | Vegas:      chiefs  by 15.5
Zoltar:       bears  by   11  opp =      saints    | Vegas:       bears  by  9.5
Zoltar:       lions  by    7  opp =  buccaneers    | Vegas:       lions  by  5.5
Zoltar:     cowboys  by    8  opp =      titans    | Vegas:     cowboys  by  8.5
Zoltar:    panthers  by    6  opp =      ravens    | Vegas:    panthers  by  2.5
Zoltar:       bills  by   15  opp =    dolphins    | Vegas:       bills  by 16.5
Zoltar:    steelers  by    0  opp =      eagles    | Vegas:    steelers  by  4.5
Zoltar:    chargers  by    7  opp =        jets    | Vegas:    chargers  by  3.5
Zoltar:     broncos  by   15  opp =     raiders    | Vegas:     broncos  by  8.5
Zoltar:     jaguars  by    5  opp =      giants    | Vegas:     jaguars  by  6.5
Zoltar: fortyniners  by    3  opp =     vikings    | Vegas: fortyniners  by  3.5
Zoltar:     bengals  by    0  opp =  commanders    | Vegas:     bengals  by  5.5

Zoltar theoretically suggests betting when the Vegas line is “significantly” different from Zoltar’s prediction. During the middle part of the season, I typically use a threshold of 4.5 points.

For week #11 Zoltar agrees closely with Las Vegas. Zoltar likes two Vegas underdogs and one favorite:

cardinals    at       chiefs: Bet on Vegas underdog cardinals
raiders      at      broncos: Bet on Vegas favorite broncos
bengals      at   commanders: Bet on Vegas underdog commanders

A bet on the Vegas underdog Cardinals against the Chiefs will pay off if the Cardinals win by any score, or if the favored Chiefs win but by less than 15.5 points (i.e., 15 points or less). If a favored team wins by exactly the point spread, the wager is a push. This is why point spreads often have a 0.5 added — called “the hook” — to eliminate pushes.

I use the early Vegas point spreads, usually posted on late Monday night, right after the Monday Night Football game. By the time you read this, the point spreads will almost certainly have changed. I’ve noticed that, compared to previous years, point spreads are changing more dramatically by late Tuesdays than they used to. A swing of 10 points is not uncommon.

I speculate that this is due to a huge increase in betting. When a lot of money is bet on one team in a matchup, the bookmakers must make a huge change in the point spread to encourage betting on the other team. Bookmakers only make money when the betting amounts are close to equal on both teams.

Theoretically, if you must bet $110 to win $100 (typical in Vegas) then you’ll make money if you predict at 53% accuracy or better. But realistically, you need to predict at 60% accuracy or better, to account for logistics and careless errors (data entry, etc.)

In week #10, against the Vegas point spread, Zoltar went 1-1 (using 4.5 points as the advice threshold). For the season, against the spread, Zoltar is 16-9 (64% accuracy).

Just for fun, I track how well Zoltar does when just trying to predict just which team will win a game. This isn’t useful except for parlay betting. In week #10, just predicting the winning team, Zoltar went 6-4 (60% correct) with 4 games too close for Zoltar to call, which is very poor. Vegas went 10-4 at just predicting the winning team, which is typical for Vegas in late-mid season.



My system is named after the Zoltar fortune teller machine you can find in arcades. While browsing Internet images, I came across a very nice “Ask the Brain” fortune teller machine.

There is almost no information available about “Ask the Brain” fortune teller machine. As best I can tell, “Ask the Brain” was created by the LifeFormations company sometime in the early 2000s.

An Internet search for “Ask the Brain” returned an AI result that said the machine was made by the Exhibit Supply Company (ESCO), but this is completely incorrect. ESCO was a big manufacturer of arcade machines in the 1930s through 1950s, but ESCO certainly did not make “Ask the Brain”.

“Ask the Brain” is similar in concept, and was clearly inspired by “Zoltar”, but “Ask the Brain” has much more complex movement, and the face of The Brain is stunningly realistic. The audio fortune runs about 30 seconds. It begins with some humorous chit-chat or pun, and ends with a geneic fortune like, “I say go for it!”

LifeFormations renamed itself to LF Studios in 2023. The company designs and manufactures wonderful animatronics. Their web site at lfstudios.com is fantastic. If LF Studios made “Ask the Brain”, it looks like they no longer do so.


Posted in Zoltar | Leave a comment

I Use AI to Improve My Random Forest Regression System Using JavaScript

A simple decision tree for regression will almost always overfit the training data where the model accuracy on the training data is near 100% but accuracy on new, previously unseen data is very poor.

The simple idea of random forest regression is to create a collection/ensemble of many simple decision trees, where each tree is trained on a different subset of the rows of the training data, and either all columns, or optionally a subset of the predictor columns, of the training data.

For example, if the source set of training data has 200 rows, you could create 50 simple trees, where each tree is trained on 180 randomly selected rows. The selection is done “with replacement” so for a random subset of training data, some rows will be used more than once and some rows won’t be used at all. An option is to use a random subset of the training data columns for each tree. For example, if a set of training data has 5 predictor columns, and you specify using 4 columns, the first decision tree might use columns [0], [2], [3], [4]. The second tree might use columns [0], [1], [3], [4]. And so on.

Actually, the column selection process is a bit tricky and involves selecting columns at every tree node split, during tree creation.

After all the trees are trained, a prediction for an input vector x is just the average of the predictions of the trees in the collection. Random forest regression is a very simple and crude idea that sometimes works well, and sometimes doesn’t work very well.

Note that bagging (bootstrap aggregation) tree regression is a specific kind of random forest regression. In bagging tree regression, the subsets of the training data all use all the columns. In random forest regression, a random subset of columns is used (or you can use all columns).

Even though bagging tree regression is a type of random forest regression, the two techniques are usually considered distinct.

Bagging tree regression was first described in 1994 paper by a fellow named Leo Breiman, even though the idea had been used for at least a decade before. Then the 1995 paper “Random Decision Forests” by Tin Kam Ho introduced the key ideas for random forest regression. Breiman later developed random forest regression ideas in a 2001 paper.

So, even though bagging tree regression is just a special case of random forest regression, the term bagging tree regression had been in use so there was no getting rid of that term.

Bagging tree regression (1994) evolved into random forest regression (1995), which evolved into AdaBoost.R and AdaBoost.R2 regression (1997), which evolved into gradient boosting regression (1999) — a fascinating (but very long) story. Maybe I’ll post that story some day.

I used AI to optimize my random forest regression system. AI was remarkably accurate and found many optimizations. I made a demo. The demo data is synthetic. It was generated by a 5-10-1 neural network with random weight and bias values. The data looks like:

-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
. . .

The first five values on each line are predictors. The last value on each line is the target value to predict.

Here is the output of a demo run:

JavaScript random forest regression

Loading synthetic train (200), test (40)
Done

First three train x:
-0.1660   0.4406  -0.9998  -0.3953  -0.7065
 0.0776  -0.1616   0.3704  -0.5911   0.7562
-0.9452   0.3409  -0.1654   0.1174  -0.7192

First three train y:
0.484 0.1568 0.8054

Setting numTrees = 100
Setting maxDepth = 6
Setting minSamples = 2
Setting minLeaf = 1
Setting numSplitCols = 4
Setting nRows = 200

Creating and training random forest regression model
Done

Evaluating model

Train acc (within 0.10) = 0.8450
Test acc (within 0.10) = 0.5750

Train MSE = 0.0003
Test MSE = 0.0016

Train R2 = 0.9907
Test R2 = 0.9433

Predicting for x =
-0.1660   0.4406  -0.9998  -0.3953  -0.7065
y = 0.4888

End random forest regression demo

Notice that the model still overfits significantly. This is because the synthetic data is more or less spread evenly, which is not a good use case for bagging tree regression or its close cousin random forest regression.

I validated my JavaScript random forest regression by feeding the demo data to the Python language scikit-learn library RandomForestRegressor module. The results are very close. The results are not exactly the same because of the random column selection at each split. Interestingly, the scikit RandomForestRegressor module does not accept a num_rows parameter of some kind — all rows in the training data are always selected from (with replacement).



I find models of all kinds fascinationg — including machine learning models and model trains. When I was growing up, I loved model trains. Here are two specialized cars that I remember well.

Left: The automatic milk car was very cool. You would park the milk car next to a special platform track. Then every time you pressed an electric button, a man would unload a milk pail. The pails could be reloaded through a hidden door in the roof.

Right: The turbo missile car could fire a spring-loaded flying sauce disc weapon. Excellent for terrorizing cats and younger sisters.


Demo program. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols (my blog editor often chokes on symbols).

// random_forest_regression.js
// node.js v22.16.0
// verified by OpenAI GPT-x

const FS = require('fs');

// ==========================================================

class RandomForestRegressor
{
  constructor(nTrees, nRows, maxDepth, minSamples,
    minLeaf, numSplitCols, seed = 0)
  {
    this.nTrees = nTrees;
    this.nRows = nRows;  // num train rows, each tree
    this.maxDepth = maxDepth;
    this.minSamples = minSamples;
    this.minLeaf = minLeaf;
    this.numSplitCols = numSplitCols;
    this.trees = [];

    this.seed = seed + 0.5; // avoid 0
  }

  // --------------------------------------------------------
  // primary: ctor(), train(), predict(), accuracy(), mse()
  // helper: next(), nextInt()
  // --------------------------------------------------------

  train(trainX, trainY)
  {
    let totalRows = trainX.length;
    let nCols = trainX[0].length;

    // reusable buffers for all trees
    let subsetX = new Array(this.nRows);
    for (let i = 0; i "lt" this.nRows; ++i)
      subsetX[i] = new Array(nCols);

    let subsetY = new Array(this.nRows);

    // train each tree using buffers
    for (let t = 0; t "lt" this.nTrees; ++t) {

      // populate the buffers
      for (let r = 0; r "lt" this.nRows; ++r) {
        let rndIndex = this.nextInt(0, totalRows);

        // Copy row values into reusable buffer.
        // analogous to Array.Copy() in the C#.
        for (let c = 0; c "lt" nCols; ++c)
          subsetX[r][c] = trainX[rndIndex][c];

        subsetY[r] = trainY[rndIndex];
      }

      let dtr = new DecisionTreeRegressor(
        this.maxDepth,
        this.minSamples,
        this.minLeaf,
        this.numSplitCols, // sample columns per split
        false,              // don't save rows info
        this.nextInt(0, 1000000)
      );

      dtr.train(subsetX, subsetY);
      this.trees.push(dtr);
    }
  } // train()

  // --------------------------------------------------------

  predict(x)
  {
    let sum = 0.0;

    for (let t = 0; t "lt" this.nTrees; ++t)
      sum += this.trees[t].predict(x);

    return sum / this.nTrees;
  }

  // --------------------------------------------------------

  accuracy(dataX, dataY, pctClose)
  {
    let numCorrect = 0;
    let numWrong = 0;

    for (let i = 0; i "lt" dataX.length; ++i) {
      let actualY = dataY[i];
      let predY = this.predict(dataX[i]);

      if (
        Math.abs(predY - actualY) "lt"
        (pctClose * Math.abs(actualY))
      )
        ++numCorrect;
      else
        ++numWrong;
    }

    return numCorrect / (numWrong + numCorrect);
  }

  // --------------------------------------------------------

  mse(dataX, dataY)
  {
    let n = dataX.length;
    let sum = 0.0;

    for (let i = 0; i "lt" n; ++i) {
      let actualY = dataY[i];
      let predY = this.predict(dataX[i]);

      sum += (actualY - predY) *
             (actualY - predY);
    }

    return sum / n;
  }

  // --------------------------------------------------------
  // Random-number helpers
  // --------------------------------------------------------

  next()
  {
    let x = Math.sin(this.seed++) * 10000;
    return x - Math.floor(x);
  }

  // --------------------------------------------------------

  nextInt(lo, hi)
  {
    return Math.trunc((hi - lo) * this.next() + lo);
  }

} // class RandomForestRegressor

// ==========================================================
// ==========================================================

class DecisionTreeRegressor
{
  // internal Node factory
  createNode()
  {
    // return an object literal
    return {
      id: -1,
      colIdx: -1,
      thresh: 0.0,
      left: -1,
      right: -1,
      value: 0.0,
      isLeaf: false,
      rows: null,
    };
  }

  // --------------------------------------------------------
  // primary: ctor(), train(), predict()
  // helpers: bestSplit(), treeTargetMean(), next(),
  //  nextInt(), sortParallel()
  // -------------------------------------------------------

  constructor(maxDepth, minSamples, minLeaf, numSplitCols,
    saveRows, seed)
  {
    this.maxDepth = maxDepth;
    this.minSamples = minSamples;
    this.minLeaf = minLeaf;
    this.numSplitCols = numSplitCols;
    this.saveRows = saveRows;

    let numNodes = Math.pow(2, maxDepth + 1) - 1;
    this.tree = new Array(numNodes).fill(null);

    this.seed = seed + 0.5; // avoid 0
    this.trainX = null;
    this.trainY = null;
  }

  // --------------------------------------------------------

  train(trainX, trainY)
  {
    this.trainX = trainX;
    this.trainY = trainY;

    let maxParentID = Math.pow(2, this.maxDepth) - 2;
    
    let allRows = new Array(this.trainX.length);
    for (let i = 0; i "lt" this.trainX.length; ++i)
      allRows[i] = i;

    let root = this.createNode();
    root.id = 0;
    root.value = this.treeTargetMean(allRows);
    root.rows = allRows;
    this.tree[0] = root;

    for (let i = 0; i "lt" this.tree.length; ++i) {
      let currNode = this.tree[i];
      if (currNode == null) continue;

      let leftID = currNode.id * 2 + 1;
      let rightID = currNode.id * 2 + 2;

      if (currNode.id "gt" maxParentID || 
        currNode.rows.length "lt" this.minSamples ||
        rightID "gte" this.tree.length) {
        currNode.isLeaf = true;
        continue;
      }

      let splitInfo = this.bestSplit(currNode.rows);
      let colIdx = splitInfo[0];
      let splitVal = splitInfo[1];

      // split failure
      if (colIdx == -1) {
        currNode.isLeaf = true;
        continue;
      }

      // split success
      currNode.colIdx = colIdx;
      currNode.thresh = splitVal;

      let leftIdxs = [];
      let rightIdxs = [];

      for (let k = 0; k "lt" currNode.rows.length; ++k) {
        let r = currNode.rows[k];
        if (this.trainX[r][colIdx] "lte" splitVal)
          leftIdxs.push(r);
        else
          rightIdxs.push(r);
      }

      // create and attach children
      currNode.left = leftID;
      let n1 = this.createNode();
      n1.id = leftID;
      n1.rows = leftIdxs;
      n1.value = this.treeTargetMean(leftIdxs);
      this.tree[leftID] = n1;

      currNode.right = rightID;
      let n2 = this.createNode();
      n2.id = rightID;
      n2.rows = rightIdxs;
      n2.value = this.treeTargetMean(rightIdxs);
      this.tree[rightID] = n2;
    } // main loop

    // delete row info to save memory
    if (this.saveRows == false) {
      for (let i = 0; i "lt" this.tree.length; ++i) {
        if (this.tree[i] != null)
          this.tree[i].rows = null;
      }
    }
  } // train()

  // --------------------------------------------------------

  predict(x)
  {
    let p = 0;
    let lastValidValue = 0.0;

    while (p != -1 && p "lt" this.tree.length) {
      let currNode = this.tree[p];
      if (currNode == null) break;

      lastValidValue = currNode.value;

      if (
        currNode.isLeaf ||
        currNode.colIdx == -1 ||
        currNode.left "gte" this.tree.length ||
        currNode.right "gte" this.tree.length
      )
        break;

      if (x[currNode.colIdx] "lte" currNode.thresh)
        p = currNode.left;
      else
        p = currNode.right;
    }
    return lastValidValue;
  }

  // --------------------------------------------------------
  // helpers
  // --------------------------------------------------------

  bestSplit(rows)
  {
    let bestColIdx = -1;
    let bestThresh = 0.0;
    let bestGain = -1.0;
    let nRows = rows.length;
    let nCols = this.trainX[0].length;

    if (nRows === 0)
      throw new Error("Empty data in bestSplit()");

    let colIndices = new Array(nCols);
    for (let k = 0; k "lt" nCols; ++k)
      colIndices[k] = k;

    // Fisher-Yates column selection
    for (let i = 0; i "lt" nCols; ++i) {
      let ri = this.nextInt(i, nCols);
      let tmp = colIndices[i];
      colIndices[i] = colIndices[ri];
      colIndices[ri] = tmp;
    }

    let numColsToUse;
    if (this.numSplitCols === -1)
      numColsToUse = nCols;
    else
      numColsToUse = Math.min(this.numSplitCols, nCols);

    let sortedRows = new Array(nRows);
    let featureKeys = new Array(nRows);

    let totalSum = 0.0;
    for (let i = 0; i "lt" nRows; ++i)
      totalSum += this.trainY[rows[i]];

    for (let j = 0; j "lt" numColsToUse; ++j) {
      let colIdx =colIndices[j];
      for (let i = 0; i "lt" nRows; ++i) {
        let r = rows[i];
        sortedRows[i] = r;
        featureKeys[i] = this.trainX[r][colIdx];
      } // i

      // sort featureKeys, sortedRows in parallel
      // based on order of featureKeys
      this.sortParallel(featureKeys, sortedRows);
      
      let leftSum = 0.0;
      for (let i = 0; i "lt" nRows-1; ++i) {
        leftSum += this.trainY[sortedRows[i]];
        let leftCount = i + 1;
        let rightCount = nRows - leftCount;

        if (featureKeys[i] == featureKeys[i+1])
          continue;  // duplicate

        if (leftCount "lt" this.minLeaf ||
          rightCount "lt" this.minLeaf)
          continue;

        let rightSum = totalSum - leftSum;

        let gain = (leftSum * leftSum / leftCount) +
          (rightSum * rightSum / rightCount);

        if (gain "gt" bestGain) {
          bestGain = gain;
          bestColIdx = colIdx;
          bestThresh = featureKeys[i] + 
            (featureKeys[i+1] - featureKeys[i]) / 2.0;
        }
      } // i
    } // j

    return [bestColIdx, bestThresh];
  } // bestSplit()

  // --------------------------------------------------------

  next()
  {
    let x = Math.sin(this.seed++) * 10000; // avoid stuck 0
    return x - Math.floor(x);
  }

  // -------------------------------------------------------

  nextInt(lo, hi)
  {
    return Math.trunc((hi - lo) * this.next() + lo);
  }

  // --------------------------------------------------------

  treeTargetMean(rows)
  {
    let sum = 0.0;
    for (let i = 0; i "lt" rows.length; ++i) {
      sum += this.trainY[rows[i]];
    }
    return sum / rows.length;
  }

  // --------------------------------------------------------

  sortParallel(keys, values)
  {
    // sort keys and values in-place based on order of keys
    let n = keys.length;
    // let idxs = this.argsort(keys);

    let augmented = [];
    for (let i = 0; i "lt" n; ++i) {
      augmented[i] = [keys[i], i]; // add indices to values
    }

    augmented.sort(function(a, b){return a[0] - b[0]});

    let idxs = [];
    for (let i = 0; i "lt" n; ++i) {
      idxs[i] = augmented[i][1];  // fetch the indexes
    }

    let sortedKeys = new Array(n);
    let sortedValues = new Array(n);
    for (let i = 0; i "lt" n; ++i) {
      let idx = idxs[i]; 
      sortedKeys[i] = keys[idx];
      sortedValues[i] = values[idx];
    }
    for (let i = 0; i "lt" n; ++i) {
      keys[i] = sortedKeys[i];
      values[i] = sortedValues[i];
    }

  } // sortParallel()

} // class DecisionTreeRegressor

// ==========================================================
// ==========================================================

function loadTxt(fn, delimit, usecols, comment)
{
  let all = FS.readFileSync(fn, "utf8").trim();
  let lines = all.split("\n");

  let nRows = 0;
  for (let i = 0; i "lt" lines.length; ++i) {
    if (!lines[i].startsWith(comment)) ++nRows;
  }

  let nCols = usecols.length;
  // let result = Array.from({ length: nRows },
  //   () ="gt" new Array(nCols));

  let result = [];
  for (let i = 0; i "lt" nRows; ++i) {
    result[i] = [];
    for (let j = 0; j "lt" nCols; ++j) {
      result[i][j] = 0.0;
    }
  }

  let r = 0; let i = 0;
  while (r "lt" lines.length) {
    if (lines[r].startsWith(comment)) {
      ++r;
    } 
    else {
      let tokens = lines[r].split(delimit);
      for (let j = 0; j "lt" nCols; ++j) {
        result[i][j] = parseFloat(tokens[usecols[j]]);
      }
      ++r;
      ++i;
    }
  }

  if (usecols.length === 1) {
    let vec = new Array(nRows);
    for (let k = 0; k "lt" nRows; ++k)
      vec[k] = result[k][0];
    return vec;
  }

  return result;
}

// ----------------------------------------------------------

function vecShow(v, dec, len)
{
  for (let i = 0; i "lt" v.length; ++i) {
    if (i != 0 && i % len == 0) {
      process.stdout.write("\n");
    }
    if (v[i] "gte" 0.0) {
      process.stdout.write(" ");  // + or - space
    }
    process.stdout.write(v[i].toFixed(dec));
    process.stdout.write("  ");
  }
  process.stdout.write("\n");
}

// ----------------------------------------------------------

function main()
{
  console.log("\nJavaScript random forest regression ");

  // 1. load data
  console.log("\nLoading synthetic train (200), test (40)");
  let trainFile = ".\\Data\\synthetic_train_200.txt";
  let testFile = ".\\Data\\synthetic_test_40.txt";

  let trainX = loadTxt(trainFile, ",", [0,1,2,3,4], "#");
  let trainY = loadTxt(trainFile, ",", [5], "#");
  let testX = loadTxt(testFile, ",", [0,1,2,3,4], "#");
  let testY = loadTxt(testFile, ",", [5], "#");
  console.log("Done ");

  console.log("\nFirst three train x: ");
  for (let i = 0; i "lt" 3; ++i)
    vecShow(trainX[i], 4, 8);  // 4 decimals 8 wide
  console.log("\nFirst three train y: ");
  for (let i = 0; i "lt" 3; ++i)
    process.stdout.write(trainY[i].toString() + " ");
  console.log("");

  // 2. create and train model
  let numTrees = 100;
  let maxDepth = 6;
  let minSamples = 2;
  let minLeaf = 1;
  let numSplitCols = 4; // default is p / 3
  let nRows = 200 // default is all rows
  let seed = 0;

  console.log("\nSetting numTrees = " +
    numTrees.toString());
  console.log("Setting maxDepth = " +
    maxDepth.toString());
  console.log("Setting minSamples = " +
    minSamples.toString());
  console.log("Setting minLeaf = " +
    minLeaf.toString());
  console.log("Setting numSplitCols = " +
    numSplitCols.toString());
  console.log("Setting nRows = " +
    nRows.toString());

  console.log("\nCreating and training random " +
    "forest regression model ");
  let rfr = 
    new RandomForestRegressor(numTrees, nRows, maxDepth,
      minSamples, minLeaf, numSplitCols, seed);

  rfr.train(trainX, trainY);
  console.log("Done ");

  // 3. evaluate model
  console.log("\nEvaluating model ");
  let trainAcc = rfr.accuracy(trainX, trainY, 0.10);
  let testAcc = rfr.accuracy(testX, testY, 0.10);
  console.log("\nTrain acc (within 0.10) = " +
    trainAcc.toFixed(4).toString());
  console.log("Test acc (within 0.10) = " +
    testAcc.toFixed(4).toString());

  let trainMSE = rfr.mse(trainX, trainY);
  let testMSE = rfr.mse(testX, testY);
  console.log("\nTrain MSE = " +
    trainMSE.toFixed(4).toString());
  console.log("Test MSE = " +
    testMSE.toFixed(4).toString());

  // 4. use tree to make prediction
  let x = trainX[0];
  console.log("\nPredicting for x = ");
  vecShow(x, 4, 9);
  let predY = rfr.predict(x);
  console.log("y = " + predY.toFixed(4).toString());

  console.log("\nEnd random forest regression demo ");
}

main();

Training data:

# synthetic_train_200.txt
#
-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
-0.8299, -0.9219, -0.6603,  0.7563, -0.8033,  0.7955
 0.0663,  0.3838, -0.3690,  0.3730,  0.6693,  0.3206
-0.9634,  0.5003,  0.9777,  0.4963, -0.4391,  0.7377
-0.1042,  0.8172, -0.4128, -0.4244, -0.7399,  0.4801
-0.9613,  0.3577, -0.5767, -0.4689, -0.0169,  0.6861
-0.7065,  0.1786,  0.3995, -0.7953, -0.1719,  0.5569
 0.3888, -0.1716, -0.9001,  0.0718,  0.3276,  0.2500
 0.1731,  0.8068, -0.7251, -0.7214,  0.6148,  0.3297
-0.2046, -0.6693,  0.8550, -0.3045,  0.5016,  0.2129
 0.2473,  0.5019, -0.3022, -0.4601,  0.7918,  0.2613
-0.1438,  0.9297,  0.3269,  0.2434, -0.7705,  0.5171
 0.1568, -0.1837, -0.5259,  0.8068,  0.1474,  0.3307
-0.9943,  0.2343, -0.3467,  0.0541,  0.7719,  0.5581
 0.2467, -0.9684,  0.8589,  0.3818,  0.9946,  0.1092
-0.6553, -0.7257,  0.8652,  0.3936, -0.8680,  0.7018
 0.8460,  0.4230, -0.7515, -0.9602, -0.9476,  0.1996
-0.9434, -0.5076,  0.7201,  0.0777,  0.1056,  0.5664
 0.9392,  0.1221, -0.9627,  0.6013, -0.5341,  0.1533
 0.6142, -0.2243,  0.7271,  0.4942,  0.1125,  0.1661
 0.4260,  0.1194, -0.9749, -0.8561,  0.9346,  0.2230
 0.1362, -0.5934, -0.4953,  0.4877, -0.6091,  0.3810
 0.6937, -0.5203, -0.0125,  0.2399,  0.6580,  0.1460
-0.6864, -0.9628, -0.8600, -0.0273,  0.2127,  0.5387
 0.9772,  0.1595, -0.2397,  0.1019,  0.4907,  0.1611
 0.3385, -0.4702, -0.8673, -0.2598,  0.2594,  0.2270
-0.8669, -0.4794,  0.6095, -0.6131,  0.2789,  0.4700
 0.0493,  0.8496, -0.4734, -0.8681,  0.4701,  0.3516
 0.8639, -0.9721, -0.5313,  0.2336,  0.8980,  0.1412
 0.9004,  0.1133,  0.8312,  0.2831, -0.2200,  0.1782
 0.0991,  0.8524,  0.8375, -0.2102,  0.9265,  0.2150
-0.6521, -0.7473, -0.7298,  0.0113, -0.9570,  0.7422
 0.6190, -0.3105,  0.8802,  0.1640,  0.7577,  0.1056
 0.6895,  0.8108, -0.0802,  0.0927,  0.5972,  0.2214
 0.1982, -0.9689,  0.1870, -0.1326,  0.6147,  0.1310
-0.3695,  0.7858,  0.1557, -0.6320,  0.5759,  0.3773
-0.1596,  0.3581,  0.8372, -0.9992,  0.9535,  0.2071
-0.2468,  0.9476,  0.2094,  0.6577,  0.1494,  0.4132
 0.1737,  0.5000,  0.7166,  0.5102,  0.3961,  0.2611
 0.7290, -0.3546,  0.3416, -0.0983, -0.2358,  0.1332
-0.3652,  0.2438, -0.1395,  0.9476,  0.3556,  0.4170
-0.6029, -0.1466, -0.3133,  0.5953,  0.7600,  0.4334
-0.4596, -0.4953,  0.7098,  0.0554,  0.6043,  0.2775
 0.1450,  0.4663,  0.0380,  0.5418,  0.1377,  0.2931
-0.8636, -0.2442, -0.8407,  0.9656, -0.6368,  0.7429
 0.6237,  0.7499,  0.3768,  0.1390, -0.6781,  0.2185
-0.5499,  0.1850, -0.3755,  0.8326,  0.8193,  0.4399
-0.4858, -0.7782, -0.6141, -0.0008,  0.4572,  0.4197
 0.7033, -0.1683,  0.2334, -0.5327, -0.7961,  0.1776
 0.0317, -0.0457, -0.6947,  0.2436,  0.0880,  0.3345
 0.5031, -0.5559,  0.0387,  0.5706, -0.9553,  0.3107
-0.3513,  0.7458,  0.6894,  0.0769,  0.7332,  0.3170
 0.2205,  0.5992, -0.9309,  0.5405,  0.4635,  0.3532
-0.4806, -0.4859,  0.2646, -0.3094,  0.5932,  0.3202
 0.9809, -0.3995, -0.7140,  0.8026,  0.0831,  0.1600
 0.9495,  0.2732,  0.9878,  0.0921,  0.0529,  0.1289
-0.9476, -0.6792,  0.4913, -0.9392, -0.2669,  0.5966
 0.7247,  0.3854,  0.3819, -0.6227, -0.1162,  0.1550
-0.5922, -0.5045, -0.4757,  0.5003, -0.0860,  0.5863
-0.8861,  0.0170, -0.5761,  0.5972, -0.4053,  0.7301
 0.6877, -0.2380,  0.4997,  0.0223,  0.0819,  0.1404
 0.9189,  0.6079, -0.9354,  0.4188, -0.0700,  0.1907
-0.1428, -0.7820,  0.2676,  0.6059,  0.3936,  0.2790
 0.5324, -0.3151,  0.6917, -0.1425,  0.6480,  0.1071
-0.8432, -0.9633, -0.8666, -0.0828, -0.7733,  0.7784
-0.9444,  0.5097, -0.2103,  0.4939, -0.0952,  0.6787
-0.0520,  0.6063, -0.1952,  0.8094, -0.9259,  0.4836
 0.5477, -0.7487,  0.2370, -0.9793,  0.0773,  0.1241
 0.2450,  0.8116,  0.9799,  0.4222,  0.4636,  0.2355
 0.8186, -0.1983, -0.5003, -0.6531, -0.7611,  0.1511
-0.4714,  0.6382, -0.3788,  0.9648, -0.4667,  0.5950
 0.0673, -0.3711,  0.8215, -0.2669, -0.1328,  0.2677
-0.9381,  0.4338,  0.7820, -0.9454,  0.0441,  0.5518
-0.3480,  0.7190,  0.1170,  0.3805, -0.0943,  0.4724
-0.9813,  0.1535, -0.3771,  0.0345,  0.8328,  0.5438
-0.1471, -0.5052, -0.2574,  0.8637,  0.8737,  0.3042
-0.5454, -0.3712, -0.6505,  0.2142, -0.1728,  0.5783
 0.6327, -0.6297,  0.4038, -0.5193,  0.1484,  0.1153
-0.5424,  0.3282, -0.0055,  0.0380, -0.6506,  0.6613
 0.1414,  0.9935,  0.6337,  0.1887,  0.9520,  0.2540
-0.9351, -0.8128, -0.8693, -0.0965, -0.2491,  0.7353
 0.9507, -0.6640,  0.9456,  0.5349,  0.6485,  0.1059
-0.0462, -0.9737, -0.2940, -0.0159,  0.4602,  0.2606
-0.0627, -0.0852, -0.7247, -0.9782,  0.5166,  0.2977
 0.0478,  0.5098, -0.0723, -0.7504, -0.3750,  0.3335
 0.0090,  0.3477,  0.5403, -0.7393, -0.9542,  0.4415
-0.9748,  0.3449,  0.3736, -0.1015,  0.8296,  0.4358
 0.2887, -0.9895, -0.0311,  0.7186,  0.6608,  0.2057
 0.1570, -0.4518,  0.1211,  0.3435, -0.2951,  0.3244
 0.7117, -0.6099,  0.4946, -0.4208,  0.5476,  0.1096
-0.2929, -0.5726,  0.5346, -0.3827,  0.4665,  0.2465
 0.4889, -0.5572, -0.5718, -0.6021, -0.7150,  0.2163
-0.7782,  0.3491,  0.5996, -0.8389, -0.5366,  0.6516
-0.5847,  0.8347,  0.4226,  0.1078, -0.3910,  0.6134
 0.8469,  0.4121, -0.0439, -0.7476,  0.9521,  0.1571
-0.6803, -0.5948, -0.1376, -0.1916, -0.7065,  0.7156
 0.2878,  0.5086, -0.5785,  0.2019,  0.4979,  0.2980
 0.2764,  0.1943, -0.4090,  0.4632,  0.8906,  0.2960
-0.8877,  0.6705, -0.6155, -0.2098, -0.3998,  0.7107
-0.8398,  0.8093, -0.2597,  0.0614, -0.0118,  0.6502
-0.8476,  0.0158, -0.4769, -0.2859, -0.7839,  0.7715
 0.5751, -0.7868,  0.9714, -0.6457,  0.1448,  0.1175
 0.4802, -0.7001,  0.1022, -0.5668,  0.5184,  0.1090
 0.4458, -0.6469,  0.7239, -0.9604,  0.7205,  0.0779
 0.5175,  0.4339,  0.9747, -0.4438, -0.9924,  0.2879
 0.8678,  0.7158,  0.4577,  0.0334,  0.4139,  0.1678
 0.5406,  0.5012,  0.2264, -0.1963,  0.3946,  0.2088
-0.9938,  0.5498,  0.7928, -0.5214, -0.7585,  0.7687
 0.7661,  0.0863, -0.4266, -0.7233, -0.4197,  0.1466
 0.2277, -0.3517, -0.0853, -0.1118,  0.6563,  0.1767
 0.3499, -0.5570, -0.0655, -0.3705,  0.2537,  0.1632
 0.7547, -0.1046,  0.5689, -0.0861,  0.3125,  0.1257
 0.8186,  0.2110,  0.5335,  0.0094, -0.0039,  0.1391
 0.6858, -0.8644,  0.1465,  0.8855,  0.0357,  0.1845
-0.4967,  0.4015,  0.0805,  0.8977,  0.2487,  0.4663
 0.6760, -0.9841,  0.9787, -0.8446, -0.3557,  0.1509
-0.1203, -0.4885,  0.6054, -0.0443, -0.7313,  0.4854
 0.8557,  0.7919, -0.0169,  0.7134, -0.1628,  0.2002
 0.0115, -0.6209,  0.9300, -0.4116, -0.7931,  0.4052
-0.7114, -0.9718,  0.4319,  0.1290,  0.5892,  0.3661
 0.3915,  0.5557, -0.1870,  0.2955, -0.6404,  0.2954
-0.3564, -0.6548, -0.1827, -0.5172, -0.1862,  0.4622
 0.2392, -0.4959,  0.5857, -0.1341, -0.2850,  0.2470
-0.3394,  0.3947, -0.4627,  0.6166, -0.4094,  0.5325
 0.7107,  0.7768, -0.6312,  0.1707,  0.7964,  0.2757
-0.1078,  0.8437, -0.4420,  0.2177,  0.3649,  0.4028
-0.3139,  0.5595, -0.6505, -0.3161, -0.7108,  0.5546
 0.4335,  0.3986,  0.3770, -0.4932,  0.3847,  0.1810
-0.2562, -0.2894, -0.8847,  0.2633,  0.4146,  0.4036
 0.2272,  0.2966, -0.6601, -0.7011,  0.0284,  0.2778
-0.0743, -0.1421, -0.0054, -0.6770, -0.3151,  0.3597
-0.4762,  0.6891,  0.6007, -0.1467,  0.2140,  0.4266
-0.4061,  0.7193,  0.3432,  0.2669, -0.7505,  0.6147
-0.0588,  0.9731,  0.8966,  0.2902, -0.6966,  0.4955
-0.0627, -0.1439,  0.1985,  0.6999,  0.5022,  0.3077
 0.1587,  0.8494, -0.8705,  0.9827, -0.8940,  0.4263
-0.7850,  0.2473, -0.9040, -0.4308, -0.8779,  0.7199
 0.4070,  0.3369, -0.2428, -0.6236,  0.4940,  0.2215
-0.0242,  0.0513, -0.9430,  0.2885, -0.2987,  0.3947
-0.5416, -0.1322, -0.2351, -0.0604,  0.9590,  0.3683
 0.1055,  0.7783, -0.2901, -0.5090,  0.8220,  0.2984
-0.9129,  0.9015,  0.1128, -0.2473,  0.9901,  0.4776
-0.9378,  0.1424, -0.6391,  0.2619,  0.9618,  0.5368
 0.7498, -0.0963,  0.4169,  0.5549, -0.0103,  0.1614
-0.2612, -0.7156,  0.4538, -0.0460, -0.1022,  0.3717
 0.7720,  0.0552, -0.1818, -0.4622, -0.8560,  0.1685
-0.4177,  0.0070,  0.9319, -0.7812,  0.3461,  0.3052
-0.0001,  0.5542, -0.7128, -0.8336, -0.2016,  0.3803
 0.5356, -0.4194, -0.5662, -0.9666, -0.2027,  0.1776
-0.2378,  0.3187, -0.8582, -0.6948, -0.9668,  0.5474
-0.1947, -0.3579,  0.1158,  0.9869,  0.6690,  0.2992
 0.3992,  0.8365, -0.9205, -0.8593, -0.0520,  0.3154
-0.0209,  0.0793,  0.7905, -0.1067,  0.7541,  0.1864
-0.4928, -0.4524, -0.3433,  0.0951, -0.5597,  0.6261
-0.8118,  0.7404, -0.5263, -0.2280,  0.1431,  0.6349
 0.0516, -0.8480,  0.7483,  0.9023,  0.6250,  0.1959
-0.3212,  0.1093,  0.9488, -0.3766,  0.3376,  0.2735
-0.3481,  0.5490, -0.3484,  0.7797,  0.5034,  0.4379
-0.5785, -0.9170, -0.3563, -0.9258,  0.3877,  0.4121
 0.3407, -0.1391,  0.5356,  0.0720, -0.9203,  0.3458
-0.3287, -0.8954,  0.2102,  0.0241,  0.2349,  0.3247
-0.1353,  0.6954, -0.0919, -0.9692,  0.7461,  0.3338
 0.9036, -0.8982, -0.5299, -0.8733, -0.1567,  0.1187
 0.7277, -0.8368, -0.0538, -0.7489,  0.5458,  0.0830
 0.9049,  0.8878,  0.2279,  0.9470, -0.3103,  0.2194
 0.7957, -0.1308, -0.5284,  0.8817,  0.3684,  0.2172
 0.4647, -0.4931,  0.2010,  0.6292, -0.8918,  0.3371
-0.7390,  0.6849,  0.2367,  0.0626, -0.5034,  0.7039
-0.1567, -0.8711,  0.7940, -0.5932,  0.6525,  0.1710
 0.7635, -0.0265,  0.1969,  0.0545,  0.2496,  0.1445
 0.7675,  0.1354, -0.7698, -0.5460,  0.1920,  0.1728
-0.5211, -0.7372, -0.6763,  0.6897,  0.2044,  0.5217
 0.1913,  0.1980,  0.2314, -0.8816,  0.5006,  0.1998
 0.8964,  0.0694, -0.6149,  0.5059, -0.9854,  0.1825
 0.1767,  0.7104,  0.2093,  0.6452,  0.7590,  0.2832
-0.3580, -0.7541,  0.4426, -0.1193, -0.7465,  0.5657
-0.5996,  0.5766, -0.9758, -0.3933, -0.9572,  0.6800
 0.9950,  0.1641, -0.4132,  0.8579,  0.0142,  0.2003
-0.4717, -0.3894, -0.2567, -0.5111,  0.1691,  0.4266
 0.3917, -0.8561,  0.9422,  0.5061,  0.6123,  0.1212
-0.0366, -0.1087,  0.3449, -0.1025,  0.4086,  0.2475
 0.3633,  0.3943,  0.2372, -0.6980,  0.5216,  0.1925
-0.5325, -0.6466, -0.2178, -0.3589,  0.6310,  0.3568
 0.2271,  0.5200, -0.1447, -0.8011, -0.7699,  0.3128
 0.6415,  0.1993,  0.3777, -0.0178, -0.8237,  0.2181
-0.5298, -0.0768, -0.6028, -0.9490,  0.4588,  0.4356
 0.6870, -0.1431,  0.7294,  0.3141,  0.1621,  0.1632
-0.5985,  0.0591,  0.7889, -0.3900,  0.7419,  0.2945
 0.3661,  0.7984, -0.8486,  0.7572, -0.6183,  0.3449
 0.6995,  0.3342, -0.3113, -0.6972,  0.2707,  0.1712
 0.2565,  0.9126,  0.1798, -0.6043, -0.1413,  0.2893
-0.3265,  0.9839, -0.2395,  0.9854,  0.0376,  0.4770
 0.2690, -0.1722,  0.9818,  0.8599, -0.7015,  0.3954
-0.2102, -0.0768,  0.1219,  0.5607, -0.0256,  0.3949
 0.8216, -0.9555,  0.6422, -0.6231,  0.3715,  0.0801
-0.2896,  0.9484, -0.7545, -0.6249,  0.7789,  0.4370
-0.9985, -0.5448, -0.7092, -0.5931,  0.7926,  0.5402

Test data:

# synthetic_test_40.txt
#
 0.7462,  0.4006, -0.0590,  0.6543, -0.0083,  0.1935
 0.8495, -0.2260, -0.0142, -0.4911,  0.7699,  0.1078
-0.2335, -0.4049,  0.4352, -0.6183, -0.7636,  0.5088
 0.1810, -0.5142,  0.2465,  0.2767, -0.3449,  0.3136
-0.8650,  0.7611, -0.0801,  0.5277, -0.4922,  0.7140
-0.2358, -0.7466, -0.5115, -0.8413, -0.3943,  0.4533
 0.4834,  0.2300,  0.3448, -0.9832,  0.3568,  0.1360
-0.6502, -0.6300,  0.6885,  0.9652,  0.8275,  0.3046
-0.3053,  0.5604,  0.0929,  0.6329, -0.0325,  0.4756
-0.7995,  0.0740, -0.2680,  0.2086,  0.9176,  0.4565
-0.2144, -0.2141,  0.5813,  0.2902, -0.2122,  0.4119
-0.7278, -0.0987, -0.3312, -0.5641,  0.8515,  0.4438
 0.3793,  0.1976,  0.4933,  0.0839,  0.4011,  0.1905
-0.8568,  0.9573, -0.5272,  0.3212, -0.8207,  0.7415
-0.5785,  0.0056, -0.7901, -0.2223,  0.0760,  0.5551
 0.0735, -0.2188,  0.3925,  0.3570,  0.3746,  0.2191
 0.1230, -0.2838,  0.2262,  0.8715,  0.1938,  0.2878
 0.4792, -0.9248,  0.5295,  0.0366, -0.9894,  0.3149
-0.4456,  0.0697,  0.5359, -0.8938,  0.0981,  0.3879
 0.8629, -0.8505, -0.4464,  0.8385,  0.5300,  0.1769
 0.1995,  0.6659,  0.7921,  0.9454,  0.9970,  0.2330
-0.0249, -0.3066, -0.2927, -0.4923,  0.8220,  0.2437
 0.4513, -0.9481, -0.0770, -0.4374, -0.9421,  0.2879
-0.3405,  0.5931, -0.3507, -0.3842,  0.8562,  0.3987
 0.9538,  0.0471,  0.9039,  0.7760,  0.0361,  0.1706
-0.0887,  0.2104,  0.9808,  0.5478, -0.3314,  0.4128
-0.8220, -0.6302,  0.0537, -0.1658,  0.6013,  0.4306
-0.4123, -0.2880,  0.9074, -0.0461, -0.4435,  0.5144
 0.0060,  0.2867, -0.7775,  0.5161,  0.7039,  0.3599
-0.7968, -0.5484,  0.9426, -0.4308,  0.8148,  0.2979
 0.7811,  0.8450, -0.6877,  0.7594,  0.2640,  0.2362
-0.6802, -0.1113, -0.8325, -0.6694, -0.6056,  0.6544
 0.3821,  0.1476,  0.7466, -0.5107,  0.2592,  0.1648
 0.7265,  0.9683, -0.9803, -0.4943, -0.5523,  0.2454
-0.9049, -0.9797, -0.0196, -0.9090, -0.4433,  0.6447
-0.4607,  0.1811, -0.2389,  0.4050, -0.0078,  0.5229
 0.2664, -0.2932, -0.4259, -0.7336,  0.8742,  0.1834
-0.4507,  0.1029, -0.6294, -0.1158, -0.6294,  0.6081
 0.8948, -0.0124,  0.9278,  0.2899, -0.0314,  0.1534
-0.1323, -0.8813, -0.0146, -0.0697,  0.6135,  0.2386
Posted in JavaScript, Machine Learning | Leave a comment

Computing Precision, Recall, and F1 Score for a Binary Classification Model Using C#

The goal of a machine learning binary classification system is to predict the value of a variable that can have two discrete values. For example, you might want to predict the sex (male = 0, female = 1) of a person based on age, State of residence, income and political leaning (conservative, moderate, liberal).

There are approximately a dozen binary classification techniques. Techniques include logistic regression, nearest neighbors classification, naive Bayes classification, winnow classification, neural network classification, and support vector classification. Each technique has several variations, based mostly on how the prediction model is trained.

To evaluate a trained binary classification model, two common techniques are to compute a confusion matrix, and to compute precision, recall, and F1 score metrics. For example:

Confusion matrix:
 actual   0 |     16    10  | 0.6154
 actual   1 |      0    14  | 1.0000
----------
predicted:         0     1  | 0.7500

Computing precision recall
precision = 0.7907
recall =    0.9623
F1 score =  0.8681

In binary classification, the two possible values to predict are almost always labeled 0 and 1. In the confusion matrix above, there are a total of 40 data items. Of these, 16 were actual class = 0, and predicted class = 0. These are “true negatives” (correctly predicted class = 0 items). The “false positives” are those items incorrectly predicted as positive (1) — so actual class = 0, predicted class = 1. There are 10 false positives.

Similarly, there are 14 “true positives” (correctly predicted class = 1). And there are 0 “false negatives” — items incorrectly predicted as negative class 0 — actual class = 1, predicted class = 0.

Note that years ago, when I was a university student, there was no consensus on what a “false positive” was. Some references defined a false positive to be items “incorrectly predicted as positive” (actual = 0, predicted = 1). And some references defined a false positive to be items “incorrectly predicted positives” (actual = 1, predicted = 0). But when Wikipedia posted an article on all this, they defined a false positive as those items “incorrectly predicted as positive” (actual = 0, predicted = 1), and that definition quickly became the standard. However, you might still run into the non-standard definitions of false positive and false negative.

The confusion matrix above shows that the overall accuracy of the binary classification model is 75.00% (30 out of 40 correct). But this is slightly misleading because the model predicts class 0 poorly (16 out of 26 correct, 61.54% accuracy), and predicts class 1 perfectly (14 out of 14 correct). This is the point of a confusion matrix — to give more granular accuracy information than a single overall value.

The precision, recall, and F1 scores are useful when you have imbalanced datasets (many more of one of the two classes) or when the impact of different types of errors (false positives vs. false negatives) is different. The F1 score is just the harmonic mean of precision and recall, so it is a summary of precision and recall.

The definitions of precision, recall, and F1 are:

precision = TP / (TP + FP)
recall    = TP / (TP + FN)
F1 score  = (2 * TP) / ((2 * TP) + FP + FN)
or
F1 score  = 2 * (precision * recall) / (precision + recall)

There are other equivalent definitions of F1 score than the two above.

I put together a demo using the C# language. For binary classification, I used the unusual approach of applying linear regression. The idea is that the target variable to predict in the training data is 0 or 1. If you use standard linear regression the output will be a single numeric value from negative infinity to positive infinity. If the predicted output is less than 0.5 (close to 0), the predicted class is 0. If the predicted output is greater than or equal to 0.5 (close to 1), the predicted class is 1.

For my demo data, I used one of my standard synthetic datasets. The raw data looks like:

F, 24, michigan, 29500.00, liberal
M, 39, oklahoma, 51200.00, moderate
F, 63, nebraska, 75800.00, conservative
M, 36, michigan, 44500.00, moderate
. . .

The fields are sex, age, State (just three), income, political leaning. The goal is to predict sex (male = 0) from age, State, income, and politics.

The normalized and encoded data is:

1, 0.24, 1,0,0, 0.2950, 0,0,1
0, 0.39, 0,0,1, 0.5120, 0,1,0
1, 0.63, 0,1,0, 0.7580, 1,0,0
0, 0.36, 1,0,0, 0.4450, 0,1,0
. . .

There are 200 training items and 40 test items. The output of my demo is:

Linear regression binary classification with coordinate
 descent training
Predict sex (male=0) from age, State, income, politics

Loading people train (200) and test (40) data
Done

First three train X:
  0.2400  1.00  0.00  0.00  0.2950  0.00  0.00  1.00
  0.3900  0.00  0.00  1.00  0.5120  0.00  1.00  0.00
  0.6300  0.00  1.00  0.00  0.7580  1.00  0.00  0.00

First three train y:
  1.0000
  0.0000
  1.0000

Creating and training model
Done

Weights/coefficients:
9.70  0.13  0.12  -0.06  -9.57  0.47  0.17  -0.12
Bias/constant: 1.1020

Confusion matrix:
 actual   0 |     16    10  | 0.6154
 actual   1 |      0    14  | 1.0000
----------
predicted:         0     1  | 0.7500

Computing precision recall
precision = 0.7907
recall =    0.9623
F1 score =  0.8681

End demo

There are many ways (at least a dozen) to train a linear regression model. I used coordinate descent training. Alternatives include stochastic gradient descent, and several types of closed-form solutions such as QR decomposition with back-substitution solve, and Gauss-Jordan elimination.

Good fun.



The definitions of precision and recall were cAs a young man, I watched the very first James Bond films, including “Dr. No” (1962). I also read most of the Bond novels. The book “Dr. No” was published in 1958.

Left: This is the illustration on the dust cover of the first edition. Very beautiful to my eye. Mysterious and ominous. By artist Pat Marriott.

Center: A very interesting paperback cover. By artist Richie Fahey.

Right: This is the Signet Publishing version I read in the early 1960s. It’s not really very attractive but it is interesting. Cover art by Barye Phillips, who was prolific in the 1960s.


Demo program. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols (my blog editor chokes on symbols).

using System;
using System.IO;
using System.Collections.Generic;

namespace LinearRegressionBinaryCoordinate
{
  internal class LinearRegressionProgram
  {
    static void Main(string[] args)
    {
      Console.WriteLine("\nLinear regression" +
        " binary classification with coordinate" +
        " descent training ");

      // sex age State income politics
      // 1, 0.24, 1,0,0, 0.2950, 0,0,1
      // 0, 0.39, 0,0,1, 0.5120, 0,1,0
      // 0    1   2 3 4     5    6 7 8
      //
      // predict sex from age State income politics
      Console.WriteLine("Predict sex (male=0) from age," +
        " State, income, politics ");

      // 1. load data
      Console.WriteLine("\nLoading people train" +
        " (200) and test (40) data");
      string trainFile =
        "..\\..\\..\\Data\\people_train_200.txt";
      int[] colsX = new int[] { 1, 2, 3, 4, 5, 6, 7, 8 };
      double[][] trainX =
        MatLoad(trainFile, colsX, ',', "#");
      double[] trainY =
        MatToVec(MatLoad(trainFile,
        new int[] { 0 }, ',', "#"));

      string testFile =
        "..\\..\\..\\Data\\people_test_40.txt";
      double[][] testX =
        MatLoad(testFile, colsX, ',', "#");
      double[] testY =
        MatToVec(MatLoad(testFile,
        new int[] { 0 }, ',', "#"));
      Console.WriteLine("Done ");

      Console.WriteLine("\nFirst three train X: ");
      for (int i = 0; i "lt" 3; ++i)
        VecShow(trainX[i], 4, 8);

      Console.WriteLine("\nFirst three train y: ");
      for (int i = 0; i "lt" 3; ++i)
        Console.WriteLine(trainY[i].ToString("F4").
          PadLeft(8));

      // 2. create and train model
      Console.WriteLine("\nCreating and training model");
      LinearRegressor model = new LinearRegressor();
      model.Train(trainX, trainY);
      Console.WriteLine("Done ");

      // 2b.show model parameters
      Console.WriteLine("\nWeights/coefficients: ");
      for (int i = 0; i "lt" model.weights.Length; ++i)
        Console.Write(model.weights[i].ToString("F4") + "  ");
      Console.WriteLine("\nBias/constant: " +
        model.bias.ToString("F2"));

      // 3. evaluate model
      int[][] cm = model.ConfusionMatrix(testX, testY);
      Console.WriteLine("\nConfusion matrix: ");
      model.ShowConfusion(cm);

      Console.WriteLine("\nComputing precision recall ");
      PrecisionRecall(model, trainX, trainY);

      Console.WriteLine("\nEnd demo ");
      Console.ReadLine();
    } // Main

    // ------------------------------------------------------
    // helpers for Main()
    // ------------------------------------------------------

    static double[][] MatLoad(string fn, int[] usecols,
      char sep, string comment)
    {
      List"lt"double[]"gt" result = 
        new List"lt"double[]"gt"();
      string line = "";
      FileStream ifs = new FileStream(fn, FileMode.Open);
      StreamReader sr = new StreamReader(ifs);
      while ((line = sr.ReadLine()) != null)
      {
        if (line.StartsWith(comment) == true)
          continue;
        string[] tokens = line.Split(sep);
        List"lt"double"gt" lst = new List"lt"double"gt"();
        for (int j = 0; j "lt" usecols.Length; ++j)
          lst.Add(double.Parse(tokens[usecols[j]]));
        double[] row = lst.ToArray();
        result.Add(row);
      }
      sr.Close(); ifs.Close();
      return result.ToArray();
    }

    static double[] MatToVec(double[][] M)
    {
      int nRows = M.Length;
      int nCols = M[0].Length;
      double[] result = new double[nRows * nCols];
      int k = 0;
      for (int i = 0; i "lt" nRows; ++i)
        for (int j = 0; j "lt" nCols; ++j)
          result[k++] = M[i][j];
      return result;
    }

    static void VecShow(double[] vec, int dec, int wid)
    {
      for (int i = 0; i "lt" vec.Length; ++i)
        Console.Write(vec[i].ToString("F" + dec).
          PadLeft(wid));
      Console.WriteLine("");
    }

    // ------------------------------------------------------

    static void PrecisionRecall(dynamic model, 
      double[][] dataX, double[] dataY)
    {
      // compute and display precision, recall, F1
      // for a binary classifer
      // assumes a Predict() method for model
      // A false positive occurs when the model predicts
      // the positive class (1), but the true actual label
      // is the negative class (0) -- Wikipedia
      int TP = 0; // correctly predicted as positive (1)
      int TN = 0;
      int FP = 0;  // falsely predicted as positive (1)
      int FN = 0;  
      int n = dataX.Length;
      for (int i = 0; i "lt" n; ++i)
      {
        double[] x = dataX[i];
        double actY = (int)(dataY[i]);
        int predY = model.Predict(x);
        
        if (actY == 0 && predY == 0)
          ++TN;
        else if (actY == 1 && predY == 1)
          ++TP;
        else if (actY == 0 && predY == 1)
          ++FP;
        else if (actY == 1 && predY == 0)
          ++FN;
      }
      double precision = (TP * 1.0) / (TP + FP);
      double recall = (TP * 1.0) / (TP + FN);
      
      double F1 = 2.0 * (precision * recall) / 
        (precision + recall);
      // double F1 = (2.0 * TP) /
      //   ((2 * TP) + FP + FN);
      // double F1 = 1.0 /
      //   (((1.0 / precision) + (1.0 / recall)) / 2.0);

      Console.WriteLine("precision = " +
        precision.ToString("F4"));
      Console.WriteLine("recall =    " +
        recall.ToString("F4"));
      Console.WriteLine("F1 score =  " +
        F1.ToString("F4"));
    }

    // ------------------------------------------------------

  } // class Program

  public class LinearRegressor
  {
    public double[] weights;
    public double bias;

    public LinearRegressor()
    {
      this.weights = new double[0]; // keep compiler happy
      this.bias = 0;
    }
    // Compute coefficients through a fixed number of
    // coordinate-descent passes, updating each coefficient
    // to minimize the squared prediction error while
    // holding the other coefficients constant. After each
    // pass, the coefficients increasingly account for
    // relationships between features, typically producing
    // a more accurate model than the initial covariance
    // estimates alone. Finally, the algorithm calculates
    // the bias, yielding an efficient approximation of
    // ordinary least-squares regression without matrix
    // inversion, QR decomposition, or a learning rate.

    public void Train(double[][] trainX, double[] trainY)
    {
      //  coordinate descent
      int n = trainX.Length;
      int dim = trainX[0].Length;
      this.weights = new double[dim];

      // 1. compute predictor means and target mean
      double[] xMean = new double[dim];
      double yMean = 0.0;
      for (int i = 0; i "lt" n; ++i)
      {
        yMean += trainY[i];
        for (int j = 0; j "lt" dim; ++j)
          xMean[j] += trainX[i][j];
      }
      yMean /= n;
      for (int j = 0; j "lt" dim; ++j)
        xMean[j] /= n;

      // 3. coordinate descent
      int maxPasses = n * 50;  //10000;
      double exitTol = 1.0e-8;

      for (int pass = 0; pass "lt" maxPasses; ++pass)
      {
        double maxChange = 0.0;

        for (int j = 0; j "lt" dim; ++j)
        {
          double numerator = 0.0;
          double denominator = 0.0;

          for (int i = 0; i "lt" n; ++i)
          {
            // residual excluding feature j
            double residual = trainY[i] - yMean;

            for (int k = 0; k "lt" dim; ++k)
            {
              if (k != j)
                residual -= this.weights[k] *
                  (trainX[i][k] - xMean[k]);
            }

            double dx = trainX[i][j] - xMean[j];

            numerator += dx * residual;
            denominator += dx * dx;
          }

          double oldWeight = this.weights[j];

          if (denominator "gt" 0.0)
            this.weights[j] = numerator / denominator;
          else
            this.weights[j] = 0.0;

          double change = Math.Abs(
            this.weights[j] - oldWeight);

          if (change "gt" maxChange)
            maxChange = change;
        }

        // stop if a pass barely changes the weights
        if (maxChange "lt" exitTol)
          break;
      }

      // 4. compute bias
      this.bias = yMean;
      for (int j = 0; j "lt" dim; ++j)
        this.bias -= this.weights[j] * xMean[j];

    } // Train

    // ------------------------------------------------------

    public int Predict(double[] x)
    {
      double result = 0.0;
      for (int j = 0; j "lt" x.Length; ++j)
        result += x[j] * this.weights[j];
      result += this.bias;
      if (result "lt" 0.5)
        return 0;
      else
        return 1;
    }

    // ------------------------------------------------------

    public int[][] ConfusionMatrix(double[][] dataX,
      double[] dataY)
    {
      // assumes binary classification
      int[][] result = new int[2][];  // 2x2
      result[0] = new int[2];
      result[1] = new int[2];
      int n = dataX.Length;
      for (int i = 0; i "lt" n; ++i) // each data item
      {
        int predY = this.Predict(dataX[i]); // 0 or 1
        int actualY = (int)(dataY[i]);  // 0 or 1

        ++result[actualY][predY];

        //if (actualY == 0 && predY == 0)
        //  ++result[0][0];
        //else if (actualY == 0 && predY == 1)
        //  ++result[0][1];
        //else if (actualY == 1 && predY == 0)
        //  ++result[1][0];
        //else if (actualY == 1 && predY == 1)
        //  ++result[1][1];
      }
      return result;
    }

    public void ShowConfusion(int[][] cm)
    {
      int n = cm.Length; // number classes
      double[] accuracies = new double[n];
      for (int i = 0; i "lt" n; ++i)
      {
        int rowSum = 0;
        for (int j = 0; j "lt" n; ++j)
          rowSum += cm[i][j];
        accuracies[i] = (cm[i][i] * 1.0) / rowSum;
      }

      // assume binary classification
      int numCorrect = cm[0][0] + cm[1][1];
      int numItems =
        cm[0][0] + cm[0][1] + cm[1][0] + cm[1][1];
      double overallAcc = (numCorrect * 1.0) / numItems;

      for (int i = 0; i "lt" n; ++i)
      {
        Console.Write(" actual " +
          i.ToString().PadLeft(3) + " | ");
        for (int j = 0; j "lt" n; ++j)
          Console.Write(cm[i][j].ToString().PadLeft(6));
        Console.WriteLine("  | " +
          accuracies[i].ToString("F4"));
      }
      Console.WriteLine("----------");
      Console.Write("predicted:    ");
      for (int j = 0; j "lt" cm[0].Length; ++j)
        Console.Write(j.ToString().PadLeft(6));
      Console.Write("  | " + overallAcc.ToString("F4"));
      Console.WriteLine("");
    }

    // ------------------------------------------------------

  } // class LinearRegressor

} // ns

Training data:

# people_train.txt
# sex (0 = male, 1 = female)
# age, state (michigan, nebraska, oklahoma), income,
# politics type (conservative, moderate, liberal)
#
1, 0.24, 1, 0, 0, 0.2950, 0, 0, 1
0, 0.39, 0, 0, 1, 0.5120, 0, 1, 0
1, 0.63, 0, 1, 0, 0.7580, 1, 0, 0
0, 0.36, 1, 0, 0, 0.4450, 0, 1, 0
1, 0.27, 0, 1, 0, 0.2860, 0, 0, 1
1, 0.50, 0, 1, 0, 0.5650, 0, 1, 0
1, 0.50, 0, 0, 1, 0.5500, 0, 1, 0
0, 0.19, 0, 0, 1, 0.3270, 1, 0, 0
1, 0.22, 0, 1, 0, 0.2770, 0, 1, 0
0, 0.39, 0, 0, 1, 0.4710, 0, 0, 1
1, 0.34, 1, 0, 0, 0.3940, 0, 1, 0
0, 0.22, 1, 0, 0, 0.3350, 1, 0, 0
1, 0.35, 0, 0, 1, 0.3520, 0, 0, 1
0, 0.33, 0, 1, 0, 0.4640, 0, 1, 0
1, 0.45, 0, 1, 0, 0.5410, 0, 1, 0
1, 0.42, 0, 1, 0, 0.5070, 0, 1, 0
0, 0.33, 0, 1, 0, 0.4680, 0, 1, 0
1, 0.25, 0, 0, 1, 0.3000, 0, 1, 0
0, 0.31, 0, 1, 0, 0.4640, 1, 0, 0
1, 0.27, 1, 0, 0, 0.3250, 0, 0, 1
1, 0.48, 1, 0, 0, 0.5400, 0, 1, 0
0, 0.64, 0, 1, 0, 0.7130, 0, 0, 1
1, 0.61, 0, 1, 0, 0.7240, 1, 0, 0
1, 0.54, 0, 0, 1, 0.6100, 1, 0, 0
1, 0.29, 1, 0, 0, 0.3630, 1, 0, 0
1, 0.50, 0, 0, 1, 0.5500, 0, 1, 0
1, 0.55, 0, 0, 1, 0.6250, 1, 0, 0
1, 0.40, 1, 0, 0, 0.5240, 1, 0, 0
1, 0.22, 1, 0, 0, 0.2360, 0, 0, 1
1, 0.68, 0, 1, 0, 0.7840, 1, 0, 0
0, 0.60, 1, 0, 0, 0.7170, 0, 0, 1
0, 0.34, 0, 0, 1, 0.4650, 0, 1, 0
0, 0.25, 0, 0, 1, 0.3710, 1, 0, 0
0, 0.31, 0, 1, 0, 0.4890, 0, 1, 0
1, 0.43, 0, 0, 1, 0.4800, 0, 1, 0
1, 0.58, 0, 1, 0, 0.6540, 0, 0, 1
0, 0.55, 0, 1, 0, 0.6070, 0, 0, 1
0, 0.43, 0, 1, 0, 0.5110, 0, 1, 0
0, 0.43, 0, 0, 1, 0.5320, 0, 1, 0
0, 0.21, 1, 0, 0, 0.3720, 1, 0, 0
1, 0.55, 0, 0, 1, 0.6460, 1, 0, 0
1, 0.64, 0, 1, 0, 0.7480, 1, 0, 0
0, 0.41, 1, 0, 0, 0.5880, 0, 1, 0
1, 0.64, 0, 0, 1, 0.7270, 1, 0, 0
0, 0.56, 0, 0, 1, 0.6660, 0, 0, 1
1, 0.31, 0, 0, 1, 0.3600, 0, 1, 0
0, 0.65, 0, 0, 1, 0.7010, 0, 0, 1
1, 0.55, 0, 0, 1, 0.6430, 1, 0, 0
0, 0.25, 1, 0, 0, 0.4030, 1, 0, 0
1, 0.46, 0, 0, 1, 0.5100, 0, 1, 0
0, 0.36, 1, 0, 0, 0.5350, 1, 0, 0
1, 0.52, 0, 1, 0, 0.5810, 0, 1, 0
1, 0.61, 0, 0, 1, 0.6790, 1, 0, 0
1, 0.57, 0, 0, 1, 0.6570, 1, 0, 0
0, 0.46, 0, 1, 0, 0.5260, 0, 1, 0
0, 0.62, 1, 0, 0, 0.6680, 0, 0, 1
1, 0.55, 0, 0, 1, 0.6270, 1, 0, 0
0, 0.22, 0, 0, 1, 0.2770, 0, 1, 0
0, 0.50, 1, 0, 0, 0.6290, 1, 0, 0
0, 0.32, 0, 1, 0, 0.4180, 0, 1, 0
0, 0.21, 0, 0, 1, 0.3560, 1, 0, 0
1, 0.44, 0, 1, 0, 0.5200, 0, 1, 0
1, 0.46, 0, 1, 0, 0.5170, 0, 1, 0
1, 0.62, 0, 1, 0, 0.6970, 1, 0, 0
1, 0.57, 0, 1, 0, 0.6640, 1, 0, 0
0, 0.67, 0, 0, 1, 0.7580, 0, 0, 1
1, 0.29, 1, 0, 0, 0.3430, 0, 0, 1
1, 0.53, 1, 0, 0, 0.6010, 1, 0, 0
0, 0.44, 1, 0, 0, 0.5480, 0, 1, 0
1, 0.46, 0, 1, 0, 0.5230, 0, 1, 0
0, 0.20, 0, 1, 0, 0.3010, 0, 1, 0
0, 0.38, 1, 0, 0, 0.5350, 0, 1, 0
1, 0.50, 0, 1, 0, 0.5860, 0, 1, 0
1, 0.33, 0, 1, 0, 0.4250, 0, 1, 0
0, 0.33, 0, 1, 0, 0.3930, 0, 1, 0
1, 0.26, 0, 1, 0, 0.4040, 1, 0, 0
1, 0.58, 1, 0, 0, 0.7070, 1, 0, 0
1, 0.43, 0, 0, 1, 0.4800, 0, 1, 0
0, 0.46, 1, 0, 0, 0.6440, 1, 0, 0
1, 0.60, 1, 0, 0, 0.7170, 1, 0, 0
0, 0.42, 1, 0, 0, 0.4890, 0, 1, 0
0, 0.56, 0, 0, 1, 0.5640, 0, 0, 1
0, 0.62, 0, 1, 0, 0.6630, 0, 0, 1
0, 0.50, 1, 0, 0, 0.6480, 0, 1, 0
1, 0.47, 0, 0, 1, 0.5200, 0, 1, 0
0, 0.67, 0, 1, 0, 0.8040, 0, 0, 1
0, 0.40, 0, 0, 1, 0.5040, 0, 1, 0
1, 0.42, 0, 1, 0, 0.4840, 0, 1, 0
1, 0.64, 1, 0, 0, 0.7200, 1, 0, 0
0, 0.47, 1, 0, 0, 0.5870, 0, 0, 1
1, 0.45, 0, 1, 0, 0.5280, 0, 1, 0
0, 0.25, 0, 0, 1, 0.4090, 1, 0, 0
1, 0.38, 1, 0, 0, 0.4840, 1, 0, 0
1, 0.55, 0, 0, 1, 0.6000, 0, 1, 0
0, 0.44, 1, 0, 0, 0.6060, 0, 1, 0
1, 0.33, 1, 0, 0, 0.4100, 0, 1, 0
1, 0.34, 0, 0, 1, 0.3900, 0, 1, 0
1, 0.27, 0, 1, 0, 0.3370, 0, 0, 1
1, 0.32, 0, 1, 0, 0.4070, 0, 1, 0
1, 0.42, 0, 0, 1, 0.4700, 0, 1, 0
0, 0.24, 0, 0, 1, 0.4030, 1, 0, 0
1, 0.42, 0, 1, 0, 0.5030, 0, 1, 0
1, 0.25, 0, 0, 1, 0.2800, 0, 0, 1
1, 0.51, 0, 1, 0, 0.5800, 0, 1, 0
0, 0.55, 0, 1, 0, 0.6350, 0, 0, 1
1, 0.44, 1, 0, 0, 0.4780, 0, 0, 1
0, 0.18, 1, 0, 0, 0.3980, 1, 0, 0
0, 0.67, 0, 1, 0, 0.7160, 0, 0, 1
1, 0.45, 0, 0, 1, 0.5000, 0, 1, 0
1, 0.48, 1, 0, 0, 0.5580, 0, 1, 0
0, 0.25, 0, 1, 0, 0.3900, 0, 1, 0
0, 0.67, 1, 0, 0, 0.7830, 0, 1, 0
1, 0.37, 0, 0, 1, 0.4200, 0, 1, 0
0, 0.32, 1, 0, 0, 0.4270, 0, 1, 0
1, 0.48, 1, 0, 0, 0.5700, 0, 1, 0
0, 0.66, 0, 0, 1, 0.7500, 0, 0, 1
1, 0.61, 1, 0, 0, 0.7000, 1, 0, 0
0, 0.58, 0, 0, 1, 0.6890, 0, 1, 0
1, 0.19, 1, 0, 0, 0.2400, 0, 0, 1
1, 0.38, 0, 0, 1, 0.4300, 0, 1, 0
0, 0.27, 1, 0, 0, 0.3640, 0, 1, 0
1, 0.42, 1, 0, 0, 0.4800, 0, 1, 0
1, 0.60, 1, 0, 0, 0.7130, 1, 0, 0
0, 0.27, 0, 0, 1, 0.3480, 1, 0, 0
1, 0.29, 0, 1, 0, 0.3710, 1, 0, 0
0, 0.43, 1, 0, 0, 0.5670, 0, 1, 0
1, 0.48, 1, 0, 0, 0.5670, 0, 1, 0
1, 0.27, 0, 0, 1, 0.2940, 0, 0, 1
0, 0.44, 1, 0, 0, 0.5520, 1, 0, 0
1, 0.23, 0, 1, 0, 0.2630, 0, 0, 1
0, 0.36, 0, 1, 0, 0.5300, 0, 0, 1
1, 0.64, 0, 0, 1, 0.7250, 1, 0, 0
1, 0.29, 0, 0, 1, 0.3000, 0, 0, 1
0, 0.33, 1, 0, 0, 0.4930, 0, 1, 0
0, 0.66, 0, 1, 0, 0.7500, 0, 0, 1
0, 0.21, 0, 0, 1, 0.3430, 1, 0, 0
1, 0.27, 1, 0, 0, 0.3270, 0, 0, 1
1, 0.29, 1, 0, 0, 0.3180, 0, 0, 1
0, 0.31, 1, 0, 0, 0.4860, 0, 1, 0
1, 0.36, 0, 0, 1, 0.4100, 0, 1, 0
1, 0.49, 0, 1, 0, 0.5570, 0, 1, 0
0, 0.28, 1, 0, 0, 0.3840, 1, 0, 0
0, 0.43, 0, 0, 1, 0.5660, 0, 1, 0
0, 0.46, 0, 1, 0, 0.5880, 0, 1, 0
1, 0.57, 1, 0, 0, 0.6980, 1, 0, 0
0, 0.52, 0, 0, 1, 0.5940, 0, 1, 0
0, 0.31, 0, 0, 1, 0.4350, 0, 1, 0
0, 0.55, 1, 0, 0, 0.6200, 0, 0, 1
1, 0.50, 1, 0, 0, 0.5640, 0, 1, 0
1, 0.48, 0, 1, 0, 0.5590, 0, 1, 0
0, 0.22, 0, 0, 1, 0.3450, 1, 0, 0
1, 0.59, 0, 0, 1, 0.6670, 1, 0, 0
1, 0.34, 1, 0, 0, 0.4280, 0, 0, 1
0, 0.64, 1, 0, 0, 0.7720, 0, 0, 1
1, 0.29, 0, 0, 1, 0.3350, 0, 0, 1
0, 0.34, 0, 1, 0, 0.4320, 0, 1, 0
0, 0.61, 1, 0, 0, 0.7500, 0, 0, 1
1, 0.64, 0, 0, 1, 0.7110, 1, 0, 0
0, 0.29, 1, 0, 0, 0.4130, 1, 0, 0
1, 0.63, 0, 1, 0, 0.7060, 1, 0, 0
0, 0.29, 0, 1, 0, 0.4000, 1, 0, 0
0, 0.51, 1, 0, 0, 0.6270, 0, 1, 0
0, 0.24, 0, 0, 1, 0.3770, 1, 0, 0
1, 0.48, 0, 1, 0, 0.5750, 0, 1, 0
1, 0.18, 1, 0, 0, 0.2740, 1, 0, 0
1, 0.18, 1, 0, 0, 0.2030, 0, 0, 1
1, 0.33, 0, 1, 0, 0.3820, 0, 0, 1
0, 0.20, 0, 0, 1, 0.3480, 1, 0, 0
1, 0.29, 0, 0, 1, 0.3300, 0, 0, 1
0, 0.44, 0, 0, 1, 0.6300, 1, 0, 0
0, 0.65, 0, 0, 1, 0.8180, 1, 0, 0
0, 0.56, 1, 0, 0, 0.6370, 0, 0, 1
0, 0.52, 0, 0, 1, 0.5840, 0, 1, 0
0, 0.29, 0, 1, 0, 0.4860, 1, 0, 0
0, 0.47, 0, 1, 0, 0.5890, 0, 1, 0
1, 0.68, 1, 0, 0, 0.7260, 0, 0, 1
1, 0.31, 0, 0, 1, 0.3600, 0, 1, 0
1, 0.61, 0, 1, 0, 0.6250, 0, 0, 1
1, 0.19, 0, 1, 0, 0.2150, 0, 0, 1
1, 0.38, 0, 0, 1, 0.4300, 0, 1, 0
0, 0.26, 1, 0, 0, 0.4230, 1, 0, 0
1, 0.61, 0, 1, 0, 0.6740, 1, 0, 0
1, 0.40, 1, 0, 0, 0.4650, 0, 1, 0
0, 0.49, 1, 0, 0, 0.6520, 0, 1, 0
1, 0.56, 1, 0, 0, 0.6750, 1, 0, 0
0, 0.48, 0, 1, 0, 0.6600, 0, 1, 0
1, 0.52, 1, 0, 0, 0.5630, 0, 0, 1
0, 0.18, 1, 0, 0, 0.2980, 1, 0, 0
0, 0.56, 0, 0, 1, 0.5930, 0, 0, 1
0, 0.52, 0, 1, 0, 0.6440, 0, 1, 0
0, 0.18, 0, 1, 0, 0.2860, 0, 1, 0
0, 0.58, 1, 0, 0, 0.6620, 0, 0, 1
0, 0.39, 0, 1, 0, 0.5510, 0, 1, 0
0, 0.46, 1, 0, 0, 0.6290, 0, 1, 0
0, 0.40, 0, 1, 0, 0.4620, 0, 1, 0
0, 0.60, 1, 0, 0, 0.7270, 0, 0, 1
1, 0.36, 0, 1, 0, 0.4070, 0, 0, 1
1, 0.44, 1, 0, 0, 0.5230, 0, 1, 0
1, 0.28, 1, 0, 0, 0.3130, 0, 0, 1
1, 0.54, 0, 0, 1, 0.6260, 1, 0, 0

Test data:

# people_test_40.txt
#
0, 0.51, 1, 0, 0, 0.6120, 0, 1, 0
0, 0.32, 0, 1, 0, 0.4610, 0, 1, 0
1, 0.55, 1, 0, 0, 0.6270, 1, 0, 0
1, 0.25, 0, 0, 1, 0.2620, 0, 0, 1
1, 0.33, 0, 0, 1, 0.3730, 0, 0, 1
0, 0.29, 0, 1, 0, 0.4620, 1, 0, 0
1, 0.65, 1, 0, 0, 0.7270, 1, 0, 0
0, 0.43, 0, 1, 0, 0.5140, 0, 1, 0
0, 0.54, 0, 1, 0, 0.6480, 0, 0, 1
1, 0.61, 0, 1, 0, 0.7270, 1, 0, 0
1, 0.52, 0, 1, 0, 0.6360, 1, 0, 0
1, 0.30, 0, 1, 0, 0.3350, 0, 0, 1
1, 0.29, 1, 0, 0, 0.3140, 0, 0, 1
0, 0.47, 0, 0, 1, 0.5940, 0, 1, 0
1, 0.39, 0, 1, 0, 0.4780, 0, 1, 0
1, 0.47, 0, 0, 1, 0.5200, 0, 1, 0
0, 0.49, 1, 0, 0, 0.5860, 0, 1, 0
0, 0.63, 0, 0, 1, 0.6740, 0, 0, 1
0, 0.30, 1, 0, 0, 0.3920, 1, 0, 0
0, 0.61, 0, 0, 1, 0.6960, 0, 0, 1
0, 0.47, 0, 0, 1, 0.5870, 0, 1, 0
1, 0.30, 0, 0, 1, 0.3450, 0, 0, 1
0, 0.51, 0, 0, 1, 0.5800, 0, 1, 0
0, 0.24, 1, 0, 0, 0.3880, 0, 1, 0
0, 0.49, 1, 0, 0, 0.6450, 0, 1, 0
1, 0.66, 0, 0, 1, 0.7450, 1, 0, 0
0, 0.65, 1, 0, 0, 0.7690, 1, 0, 0
0, 0.46, 0, 1, 0, 0.5800, 1, 0, 0
0, 0.45, 0, 0, 1, 0.5180, 0, 1, 0
0, 0.47, 1, 0, 0, 0.6360, 1, 0, 0
0, 0.29, 1, 0, 0, 0.4480, 1, 0, 0
0, 0.57, 0, 0, 1, 0.6930, 0, 0, 1
0, 0.20, 1, 0, 0, 0.2870, 0, 0, 1
0, 0.35, 1, 0, 0, 0.4340, 0, 1, 0
0, 0.61, 0, 0, 1, 0.6700, 0, 0, 1
0, 0.31, 0, 0, 1, 0.3730, 0, 1, 0
1, 0.18, 1, 0, 0, 0.2080, 0, 0, 1
1, 0.26, 0, 0, 1, 0.2920, 0, 0, 1
0, 0.28, 1, 0, 0, 0.3640, 0, 0, 1
0, 0.59, 0, 0, 1, 0.6940, 0, 0, 1
Posted in Machine Learning | Leave a comment

Logistic Regression from Scratch with Newton-Raphson Training (Cholesky Decomp Solve, No Early-Exit) Using C#

Logistic regression is a technique for binary classification. For example, you might want to predict the sex of a person (0 = male, 1 = female) based on age, State, income, and political leaning. I enjoy writing code, so one rainy evening (I live in the Pacific Northwest so this wasn’t unusual), while I was waiting for the rain to stop so I could walk my dogs, I decided to implement logistic regression, from scratch, using C#.

In logistic regression, the prediction output is a pseudo-probability value between 0.0 and 1.0. If the p-value is less than 0.5, the prediction is class 0. If the p-value is greater than or equal to 0.5, the prediction is class 1. For any problem, the choice of which label is 0 and which is 1, is arbitrary.

The prediction equation for logistic regression is:

z = (wo * x0) + (w1 * x1) + . . . + b
p = 1.0 / (1.0 + exp(-z))

The xi are predictor values. The wi are model weights. The b is a bias constant. Training the model is the process of finding values for the weights and the bias so that computed output probabilities closely match the target y (0 / 1) values.

For my demo, I used one of my standard artificial datasets. The raw data looks like:

F, 24, michigan, 29500.00, liberal
M, 39, oklahoma, 51200.00, moderate
F, 63, nebraska, 75800.00, conservative
. . .

The normalized and encoded data looks like:

1, 0.24, 1,0,0, 0.2950, 0,0,1
0, 0.39, 0,0,1, 0.5120, 0,1,0
1, 0.63, 0,1,0, 0.7580, 1,0,0
. . .

There are only three State values. The output of my demo is:

Begin C# linear regression demo with Newton-Raphson training

Loading people train (200) and test (40) data
Done

First three train X:
  0.2400  1.00  0.00  0.00  0.2950  0.00  0.00  1.00
  0.3900  0.00  0.00  1.00  0.5120  0.00  1.00  0.00
  0.6300  0.00  1.00  0.00  0.7580  1.00  0.00  0.00

First three train y:
  1.0000
  0.0000
  1.0000

Creating Logistic Regression model
Done

Start Newton-Raphson train
epoch =     0  MSE =   0.1274
epoch =    40  MSE =   0.1145
epoch =    80  MSE =   0.1145
epoch =   120  MSE =   0.1145
epoch =   160  MSE =   0.1145
Done

Weights/coefficients:
81.4712 1.8590 1.8903 0.2694 -80.7779 3.9275 1.1614 -1.0702
Bias/constant: 4.0187

Evaluating model

Accuracy train = 0.8450
Accuracy test = 0.7250

MSE train = 0.1145
MSE test = 0.1305

Confusion matrix:
 actual   0 |     17     9  | 0.6538
 actual   1 |      2    12  | 0.8571
----------
predicted:         0     1  | 0.7250

End demo

There are dozens of design alternatives for logistic regression. I used Newton-Raphson training with binary cross entropy (BCE) loss gradient updates, combined with mean squared error loss progress messages.

Newton-Raphson training is appealing because it does not require any training parameters such as a learning rate and maximum epochs, that must be determined by trial and error. The main alternative, stochastic gradient descent (SGD), does require a learning rate and max epochs. Newton-Raphson computes an nxn Hessian matrix, where n is the number of predictors plus 1, which isn’t feasible for rare scenarios with datasets that have a huge number of predictors, but Newton-Raphson is very fast compared to SGD training. L-BFGS training is possible too, but it isn’t well-suited for scratch implementation.

There are several variations of Newton-Raphson. I used Cholesky decomposition with a solve function that uses back-substitution and forward-substitution to implicitly find the inverse of the Hessian matrix. An alternative is to explicitly compute the inverse of the Hessian matrix, using any standard technique, such as Cholesky inverse or Gauss-Jordan elimination. The Cholesky-decomp-solve variation I used is the most robust (least likely to fail due to numerical instability).

All versions of Newton-Raphson training for logistic regression are iterative. This requires a max epochs value. I use the number of training items as a heuristic. An alternative approach is to early-exit the main training loop by monitoring the change in model weights, or the change in loss (mean squared error or binary cross entropy error). My demo does not implement an early-exit condition.

To evaluate the trained model, I implemented code to compute and display a confusion matrix. This is more or less standard procedure for binary classification problems.

I validated my logistic regression system by running the demo data through the scikit-learn library LogisticRegression module. I got essentially the same results. See https://jamesmccaffreyblog.com/2023/02/10/logistic-regression-using-the-scikit-library-in-visual-studio-magazine/.

My implementation can be improved. Currently, the training loop iterates a fixed number of iterations (maxEpochs). It’s possible to monitor the change in model weights and bias, and early-exit training when the largest change is less than some small value, like 1.0e-6 or so. But that’s a task for another day.

Good fun on a rainy Pacific Northwest night.



I am a big fan of science fiction films of the 1950s. Some of these movies have really bad special effects, but pretty good acting and story, make them entertaining (to me anyway).

Left: In “Beginning of the End” (1957), scientists experiment with radioactive nutrients to grow giant food. Bad idea. Tens of thousands of car-sized locusts/grasshoppers make their way to Chicago. Here, the locusts crawl over a photo of a building. The locusts are lured into Lake Michigan using a synthetic mating call, where they drown. My grade for the movie = B.

Right: In “The Giant Claw” (1957), a giant bird creature somehow manages to get to Earth from an antimatter galaxy, which makes it invulnerable to all missiles and weapons. Scientists devise a way to remove the creature’s antimatter protection, and kill it with rockets. This is a pretty good movie but the production ran out of money before the special effects could be added, and resorted to low-budget Mexican special effects company. My grade for the movie = B.


Demo program. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols. My blog editor chokes on symbols.

using System;
using System.IO;
using System.Collections.Generic;

namespace LogisticRegressionPeopleNR
{
  internal class LogisticRegressionNRProgram
  {
    static void Main(string[] args)
    {
      Console.WriteLine("\nBegin C# linear regression" +
        " demo with Newton-Raphson training ");
      // sex age State income politics
      // 1, 0.24, 1,0,0, 0.2950, 0,0,1
      // 0, 0.39, 0,0,1, 0.5120, 0,1,0
      // 0    1   2 3 4     5    6 7 8
      //
      // predict sex from age State income politics

      // 1. load data
      Console.WriteLine("\nLoading people train" +
        " (200) and test (40) data");
      string trainFile =
        "..\\..\\..\\Data\\people_train_200.txt";
      int[] colsX = new int[] { 1, 2, 3, 4, 5, 6, 7, 8 };
      double[][] trainX =
        MatLoad(trainFile, colsX, ',', "#");
      double[] trainY =
        MatToVec(MatLoad(trainFile,
        new int[] { 0 }, ',', "#"));

      string testFile =
        "..\\..\\..\\Data\\people_test_40.txt";
      double[][] testX =
        MatLoad(testFile, colsX, ',', "#");
      double[] testY =
        MatToVec(MatLoad(testFile,
        new int[] { 0 }, ',', "#"));
      Console.WriteLine("Done ");

      Console.WriteLine("\nFirst three train X: ");
      for (int i = 0; i "lt" 3; ++i)
        VecShow(trainX[i], 4, 8);

      Console.WriteLine("\nFirst three train y: ");
      for (int i = 0; i "lt" 3; ++i)
        Console.WriteLine(trainY[i].ToString("F4").
          PadLeft(8));

      // 2. create and train model
      Console.WriteLine("\nCreating Logistic " +
        "Regression model ");
      LogisticRegression model =
        new LogisticRegression();
      Console.WriteLine("Done ");

      Console.WriteLine("\nStart Newton-Raphson train ");
      model.Train(trainX, trainY);
      Console.WriteLine("Done ");

      // 2b.show model parameters
      Console.WriteLine("\nWeights/coefficients: ");
      for (int i = 0; i "lt" model.weights.Length; ++i)
        Console.Write(model.weights[i].ToString("F4") + " ");
      Console.WriteLine("\nBias/constant: " +
        model.bias.ToString("F4"));

      // 3. evaluate model
      Console.WriteLine("\nEvaluating model ");

      double accTrain = model.Accuracy(trainX, trainY);
      Console.WriteLine("\nAccuracy train = " +
        accTrain.ToString("F4"));
      double accTest = model.Accuracy(testX, testY);
      Console.WriteLine("Accuracy test = " +
        accTest.ToString("F4"));

      double mseTrain = model.MSE(trainX, trainY);
      Console.WriteLine("\nMSE train = " +
        mseTrain.ToString("F4"));
      double mseTest = model.MSE(testX, testY);
      Console.WriteLine("MSE test = " +
        mseTest.ToString("F4"));

      Console.WriteLine("\nConfusion matrix: ");
      int[][] cm = model.ConfusionMatrix(testX, testY);
      model.ShowConfusion(cm);

      Console.WriteLine("\nEnd demo ");
      Console.ReadLine();

    } // Main

    // ------------------------------------------------------
    // helpers for Main()
    // ------------------------------------------------------

    static double[][] MatLoad(string fn, int[] usecols,
      char sep, string comment)
    {
      List"lt"double[]"gt" result = 
        new List"lt"double[]"gt"();
      string line = "";
      FileStream ifs = new FileStream(fn, FileMode.Open);
      StreamReader sr = new StreamReader(ifs);
      while ((line = sr.ReadLine()) != null)
      {
        if (line.StartsWith(comment) == true)
          continue;
        string[] tokens = line.Split(sep);
        List"lt"double"gt" lst = new List"lt"double"gt"();
        for (int j = 0; j "lt" usecols.Length; ++j)
          lst.Add(double.Parse(tokens[usecols[j]]));
        double[] row = lst.ToArray();
        result.Add(row);
      }
      sr.Close(); ifs.Close();
      return result.ToArray();
    }

    static double[] MatToVec(double[][] M)
    {
      int nRows = M.Length;
      int nCols = M[0].Length;
      double[] result = new double[nRows * nCols];
      int k = 0;
      for (int i = 0; i "lt" nRows; ++i)
        for (int j = 0; j "lt" nCols; ++j)
          result[k++] = M[i][j];
      return result;
    }

    static void VecShow(double[] vec, int dec, int wid)
    {
      for (int i = 0; i "lt" vec.Length; ++i)
        Console.Write(vec[i].ToString("F" + dec).
          PadLeft(wid));
      Console.WriteLine("");
    }

  } // class Program

  // ========================================================

  public class LogisticRegression
  {
    public double[] weights;
    public double bias;
    private Random rnd;

    public LogisticRegression(int seed = 0)
    {
      this.weights = new double[0]; // compiler happy
      this.bias = 0.0;
      this.rnd = new Random(seed);
    }

    // ------------------------------------------------------

    public int Predict(double[] x)
    {
      double p = ComputeP(x);
      if (p "lt" 0.5)
        return 0;
      else
        return 1;
    }

    // ------------------------------------------------------

    public double ComputeP(double[] x)
    {
      double z = 0.0;
      for (int i = 0; i "lt" x.Length; ++i)
        z += this.weights[i] * x[i];
      z += this.bias;
      double p = this.Sigmoid(z);
      return p;
    }

    // ------------------------------------------------------

    public double Sigmoid(double z)
    {
      if (z "lt" -6.0)
        return 0.0;
      else if (z "gt" 6.0)
        return 1.0;
      else
        return 1.0 / (1.0 + Math.Exp(-z));
    }

    // ------------------------------------------------------

    public void Train(double[][] trainX, double[] trainY)
    {
      // Newton-Raphson technique
      // initialize weights and bias
      int dim = trainX[0].Length;
      int n = trainX.Length;
      int m = dim + 1;  // include bias as final parameter

      this.weights = new double[dim];
      this.bias = 0.0;
      int maxEpochs = trainX.Length; // heuristic

      // Newton-Raphson iterations
      for (int epoch = 0; epoch "lt" maxEpochs; ++epoch)
      {
        // gradient and Hessian
        double[] grad = new double[m]; // features + bias
        double[][] H = new double[m][]; // H is mxm
        for (int i = 0; i "lt" m; ++i)
          H[i] = new double[m];

        // accumulate gradient and Hessian
        for (int i = 0; i "lt" n; ++i) // each train item
        {
          double[] x = trainX[i];
          double y = trainY[i];

          double p = this.ComputeP(x);
          double error = p - y; // order is important
          double v = p * (1.0 - p);

          // include bias as a final feature
          double[] xBias = new double[m];
          for (int j = 0; j "lt" dim; ++j)
            xBias[j] = x[j];
          xBias[dim] = 1.0;

          // gradient
          for (int j = 0; j "lt" m; ++j)
            grad[j] += error * xBias[j];

          // Hessian
          for (int j = 0; j "lt" m; ++j)
            for (int k = 0; k "lt" m; ++k)
              H[j][k] += v * xBias[j] * xBias[k];
        }

        // average gradient and Hessian
        for (int j = 0; j "lt" m; ++j)
        {
          grad[j] /= n;
          for (int k = 0; k "lt" m; ++k)
            H[j][k] /= n;
        }

        // small diagonal for numerical stability
        for (int j = 0; j "lt" m; ++j)
          H[j][j] += 1.0e-8;

        // Cholesky decomposition: H = L * L^T
        double[][] L = new double[m][];
        for (int i = 0; i "lt" m; ++i)
          L[i] = new double[m];

        for (int i = 0; i "lt" m; ++i)
        {
          for (int j = 0; j "lte" i; ++j)
          {
            double sum = H[i][j];

            for (int k = 0; k "lt" j; ++k)
              sum -= L[i][k] * L[j][k];

            if (i == j)
              L[i][j] = Math.Sqrt(sum);
            else
              L[i][j] = sum / L[j][j]; // not 0
          }
        } // each item

        // solve L * z = -gradient
        // (forward substitution)
        double[] z = new double[m];

        for (int i = 0; i "lt" m; ++i)
        {
          double sum = -grad[i];
          for (int j = 0; j "lt" i; ++j)
            sum -= L[i][j] * z[j];
          z[i] = sum / L[i][i];
        }

        // Solve L^T * delta = z
        // (back substitution)
        double[] delta = new double[m];

        for (int i = m - 1; i "gte" 0; --i)
        {
          double sum = z[i];
          for (int j = i + 1; j "lt" m; ++j)
            sum -= L[j][i] * delta[j];
          delta[i] = sum / L[i][i];
        }

        // update weights and bias
        for (int j = 0; j "lt" dim; ++j)
          this.weights[j] += delta[j];
        this.bias += delta[dim];

        // show progress
        if (epoch % Math.Max(1, maxEpochs / 5) == 0)
        {
          double mse = this.MSE(trainX, trainY);
          Console.WriteLine("epoch = " +
            epoch.ToString().PadLeft(5) +
            "  MSE = " + mse.ToString("F4").PadLeft(8));
        }
      } // each epoch
    } // Train()

    // ------------------------------------------------------

    public double MSE(double[][] dataX, double[] dataY)
    {
      int n = dataX.Length;
      double sum = 0.0;
      for (int i = 0; i "lt" n; ++i)
      {
        double[] x = dataX[i];
        double y = dataY[i];
        double p = this.ComputeP(x);
        sum += (p - y) * (p - y);
      }
      return sum / n;
    }

    // ------------------------------------------------------

    public double Accuracy(double[][] dataX, double[] dataY)
    {
      int n = dataX.Length;
      int numCorrect = 0; int numWrong = 0;
      for (int i = 0; i "lt" n; ++i)
      {
        double[] x = dataX[i];
        double y = (int)(dataY[i]);  // 0 or 1
        int predY = this.Predict(x);  // 0 or 1

        if (predY == y)
          ++numCorrect;
        else
          ++numWrong;
      }
      return (numCorrect * 1.0) / (numCorrect + numWrong);
    }

    // ------------------------------------------------------

    public int[][] ConfusionMatrix(double[][] dataX,
      double[] dataY)
    {
      // assumes binary classification
      int[][] result = new int[2][];  // 2x2
      result[0] = new int[2];
      result[1] = new int[2];
      int n = dataX.Length;
      for (int i = 0; i "lt" n; ++i) // each data item
      {
        int predY = this.Predict(dataX[i]); // 0 or 1
        int actualY = (int)(dataY[i]);  // 0 or 1

        ++result[actualY][predY];

        //if (actualY == 0 && predY == 0)
        //  ++result[0][0];
        //else if (actualY == 0 && predY == 1)
        //  ++result[0][1];
        //else if (actualY == 1 && predY == 0)
        //  ++result[1][0];
        //else if (actualY == 1 && predY == 1)
        //  ++result[1][1];
      }
      return result;
    }

    public void ShowConfusionX(int[][] cm)
    {
      // simple display version
      Console.Write("actual = 0: ");
      Console.Write(cm[0][0].ToString().PadLeft(4) + " ");
      Console.WriteLine(cm[0][1].ToString().PadLeft(4));
      Console.Write("actual = 1: ");
      Console.Write(cm[1][0].ToString().PadLeft(4) + " ");
      Console.WriteLine(cm[1][1].ToString().PadLeft(4));
    }

    public void ShowConfusion(int[][] cm)
    {
      int n = cm.Length; // number classes
      double[] accuracies = new double[n];
      for (int i = 0; i "lt" n; ++i)
      {
        int rowSum = 0;
        for (int j = 0; j "lt" n; ++j)
          rowSum += cm[i][j];
        accuracies[i] = (cm[i][i] * 1.0) / rowSum;
      }

      // assume binary classification
      int numCorrect = cm[0][0] + cm[1][1];
      int numItems = 
        cm[0][0] + cm[0][1] + cm[1][0] + cm[1][1];
      double overallAcc = (numCorrect * 1.0) / numItems;
     
      for (int i = 0; i "lt" n; ++i)
      {
        Console.Write(" actual " + 
          i.ToString().PadLeft(3) + " | ");
        for (int j = 0; j "lt" n; ++j)
          Console.Write(cm[i][j].ToString().PadLeft(6));
        Console.WriteLine("  | " + 
          accuracies[i].ToString("F4"));
      }
      Console.WriteLine("----------");
      Console.Write("predicted:    ");
      for (int j = 0; j "lt" cm[0].Length; ++j)
        Console.Write(j.ToString().PadLeft(6));
      Console.Write("  | " + overallAcc.ToString("F4"));
      Console.WriteLine("");
    }

  } // class LogisticRegression

  // ========================================================

} // ns

Training data:

# people_train_200.txt
# sex (0 = male, 1 = female)
# age, state (michigan, nebraska, oklahoma), income,
# politics type (conservative, moderate, liberal)
#
1, 0.24, 1, 0, 0, 0.2950, 0, 0, 1
0, 0.39, 0, 0, 1, 0.5120, 0, 1, 0
1, 0.63, 0, 1, 0, 0.7580, 1, 0, 0
0, 0.36, 1, 0, 0, 0.4450, 0, 1, 0
1, 0.27, 0, 1, 0, 0.2860, 0, 0, 1
1, 0.50, 0, 1, 0, 0.5650, 0, 1, 0
1, 0.50, 0, 0, 1, 0.5500, 0, 1, 0
0, 0.19, 0, 0, 1, 0.3270, 1, 0, 0
1, 0.22, 0, 1, 0, 0.2770, 0, 1, 0
0, 0.39, 0, 0, 1, 0.4710, 0, 0, 1
1, 0.34, 1, 0, 0, 0.3940, 0, 1, 0
0, 0.22, 1, 0, 0, 0.3350, 1, 0, 0
1, 0.35, 0, 0, 1, 0.3520, 0, 0, 1
0, 0.33, 0, 1, 0, 0.4640, 0, 1, 0
1, 0.45, 0, 1, 0, 0.5410, 0, 1, 0
1, 0.42, 0, 1, 0, 0.5070, 0, 1, 0
0, 0.33, 0, 1, 0, 0.4680, 0, 1, 0
1, 0.25, 0, 0, 1, 0.3000, 0, 1, 0
0, 0.31, 0, 1, 0, 0.4640, 1, 0, 0
1, 0.27, 1, 0, 0, 0.3250, 0, 0, 1
1, 0.48, 1, 0, 0, 0.5400, 0, 1, 0
0, 0.64, 0, 1, 0, 0.7130, 0, 0, 1
1, 0.61, 0, 1, 0, 0.7240, 1, 0, 0
1, 0.54, 0, 0, 1, 0.6100, 1, 0, 0
1, 0.29, 1, 0, 0, 0.3630, 1, 0, 0
1, 0.50, 0, 0, 1, 0.5500, 0, 1, 0
1, 0.55, 0, 0, 1, 0.6250, 1, 0, 0
1, 0.40, 1, 0, 0, 0.5240, 1, 0, 0
1, 0.22, 1, 0, 0, 0.2360, 0, 0, 1
1, 0.68, 0, 1, 0, 0.7840, 1, 0, 0
0, 0.60, 1, 0, 0, 0.7170, 0, 0, 1
0, 0.34, 0, 0, 1, 0.4650, 0, 1, 0
0, 0.25, 0, 0, 1, 0.3710, 1, 0, 0
0, 0.31, 0, 1, 0, 0.4890, 0, 1, 0
1, 0.43, 0, 0, 1, 0.4800, 0, 1, 0
1, 0.58, 0, 1, 0, 0.6540, 0, 0, 1
0, 0.55, 0, 1, 0, 0.6070, 0, 0, 1
0, 0.43, 0, 1, 0, 0.5110, 0, 1, 0
0, 0.43, 0, 0, 1, 0.5320, 0, 1, 0
0, 0.21, 1, 0, 0, 0.3720, 1, 0, 0
1, 0.55, 0, 0, 1, 0.6460, 1, 0, 0
1, 0.64, 0, 1, 0, 0.7480, 1, 0, 0
0, 0.41, 1, 0, 0, 0.5880, 0, 1, 0
1, 0.64, 0, 0, 1, 0.7270, 1, 0, 0
0, 0.56, 0, 0, 1, 0.6660, 0, 0, 1
1, 0.31, 0, 0, 1, 0.3600, 0, 1, 0
0, 0.65, 0, 0, 1, 0.7010, 0, 0, 1
1, 0.55, 0, 0, 1, 0.6430, 1, 0, 0
0, 0.25, 1, 0, 0, 0.4030, 1, 0, 0
1, 0.46, 0, 0, 1, 0.5100, 0, 1, 0
0, 0.36, 1, 0, 0, 0.5350, 1, 0, 0
1, 0.52, 0, 1, 0, 0.5810, 0, 1, 0
1, 0.61, 0, 0, 1, 0.6790, 1, 0, 0
1, 0.57, 0, 0, 1, 0.6570, 1, 0, 0
0, 0.46, 0, 1, 0, 0.5260, 0, 1, 0
0, 0.62, 1, 0, 0, 0.6680, 0, 0, 1
1, 0.55, 0, 0, 1, 0.6270, 1, 0, 0
0, 0.22, 0, 0, 1, 0.2770, 0, 1, 0
0, 0.50, 1, 0, 0, 0.6290, 1, 0, 0
0, 0.32, 0, 1, 0, 0.4180, 0, 1, 0
0, 0.21, 0, 0, 1, 0.3560, 1, 0, 0
1, 0.44, 0, 1, 0, 0.5200, 0, 1, 0
1, 0.46, 0, 1, 0, 0.5170, 0, 1, 0
1, 0.62, 0, 1, 0, 0.6970, 1, 0, 0
1, 0.57, 0, 1, 0, 0.6640, 1, 0, 0
0, 0.67, 0, 0, 1, 0.7580, 0, 0, 1
1, 0.29, 1, 0, 0, 0.3430, 0, 0, 1
1, 0.53, 1, 0, 0, 0.6010, 1, 0, 0
0, 0.44, 1, 0, 0, 0.5480, 0, 1, 0
1, 0.46, 0, 1, 0, 0.5230, 0, 1, 0
0, 0.20, 0, 1, 0, 0.3010, 0, 1, 0
0, 0.38, 1, 0, 0, 0.5350, 0, 1, 0
1, 0.50, 0, 1, 0, 0.5860, 0, 1, 0
1, 0.33, 0, 1, 0, 0.4250, 0, 1, 0
0, 0.33, 0, 1, 0, 0.3930, 0, 1, 0
1, 0.26, 0, 1, 0, 0.4040, 1, 0, 0
1, 0.58, 1, 0, 0, 0.7070, 1, 0, 0
1, 0.43, 0, 0, 1, 0.4800, 0, 1, 0
0, 0.46, 1, 0, 0, 0.6440, 1, 0, 0
1, 0.60, 1, 0, 0, 0.7170, 1, 0, 0
0, 0.42, 1, 0, 0, 0.4890, 0, 1, 0
0, 0.56, 0, 0, 1, 0.5640, 0, 0, 1
0, 0.62, 0, 1, 0, 0.6630, 0, 0, 1
0, 0.50, 1, 0, 0, 0.6480, 0, 1, 0
1, 0.47, 0, 0, 1, 0.5200, 0, 1, 0
0, 0.67, 0, 1, 0, 0.8040, 0, 0, 1
0, 0.40, 0, 0, 1, 0.5040, 0, 1, 0
1, 0.42, 0, 1, 0, 0.4840, 0, 1, 0
1, 0.64, 1, 0, 0, 0.7200, 1, 0, 0
0, 0.47, 1, 0, 0, 0.5870, 0, 0, 1
1, 0.45, 0, 1, 0, 0.5280, 0, 1, 0
0, 0.25, 0, 0, 1, 0.4090, 1, 0, 0
1, 0.38, 1, 0, 0, 0.4840, 1, 0, 0
1, 0.55, 0, 0, 1, 0.6000, 0, 1, 0
0, 0.44, 1, 0, 0, 0.6060, 0, 1, 0
1, 0.33, 1, 0, 0, 0.4100, 0, 1, 0
1, 0.34, 0, 0, 1, 0.3900, 0, 1, 0
1, 0.27, 0, 1, 0, 0.3370, 0, 0, 1
1, 0.32, 0, 1, 0, 0.4070, 0, 1, 0
1, 0.42, 0, 0, 1, 0.4700, 0, 1, 0
0, 0.24, 0, 0, 1, 0.4030, 1, 0, 0
1, 0.42, 0, 1, 0, 0.5030, 0, 1, 0
1, 0.25, 0, 0, 1, 0.2800, 0, 0, 1
1, 0.51, 0, 1, 0, 0.5800, 0, 1, 0
0, 0.55, 0, 1, 0, 0.6350, 0, 0, 1
1, 0.44, 1, 0, 0, 0.4780, 0, 0, 1
0, 0.18, 1, 0, 0, 0.3980, 1, 0, 0
0, 0.67, 0, 1, 0, 0.7160, 0, 0, 1
1, 0.45, 0, 0, 1, 0.5000, 0, 1, 0
1, 0.48, 1, 0, 0, 0.5580, 0, 1, 0
0, 0.25, 0, 1, 0, 0.3900, 0, 1, 0
0, 0.67, 1, 0, 0, 0.7830, 0, 1, 0
1, 0.37, 0, 0, 1, 0.4200, 0, 1, 0
0, 0.32, 1, 0, 0, 0.4270, 0, 1, 0
1, 0.48, 1, 0, 0, 0.5700, 0, 1, 0
0, 0.66, 0, 0, 1, 0.7500, 0, 0, 1
1, 0.61, 1, 0, 0, 0.7000, 1, 0, 0
0, 0.58, 0, 0, 1, 0.6890, 0, 1, 0
1, 0.19, 1, 0, 0, 0.2400, 0, 0, 1
1, 0.38, 0, 0, 1, 0.4300, 0, 1, 0
0, 0.27, 1, 0, 0, 0.3640, 0, 1, 0
1, 0.42, 1, 0, 0, 0.4800, 0, 1, 0
1, 0.60, 1, 0, 0, 0.7130, 1, 0, 0
0, 0.27, 0, 0, 1, 0.3480, 1, 0, 0
1, 0.29, 0, 1, 0, 0.3710, 1, 0, 0
0, 0.43, 1, 0, 0, 0.5670, 0, 1, 0
1, 0.48, 1, 0, 0, 0.5670, 0, 1, 0
1, 0.27, 0, 0, 1, 0.2940, 0, 0, 1
0, 0.44, 1, 0, 0, 0.5520, 1, 0, 0
1, 0.23, 0, 1, 0, 0.2630, 0, 0, 1
0, 0.36, 0, 1, 0, 0.5300, 0, 0, 1
1, 0.64, 0, 0, 1, 0.7250, 1, 0, 0
1, 0.29, 0, 0, 1, 0.3000, 0, 0, 1
0, 0.33, 1, 0, 0, 0.4930, 0, 1, 0
0, 0.66, 0, 1, 0, 0.7500, 0, 0, 1
0, 0.21, 0, 0, 1, 0.3430, 1, 0, 0
1, 0.27, 1, 0, 0, 0.3270, 0, 0, 1
1, 0.29, 1, 0, 0, 0.3180, 0, 0, 1
0, 0.31, 1, 0, 0, 0.4860, 0, 1, 0
1, 0.36, 0, 0, 1, 0.4100, 0, 1, 0
1, 0.49, 0, 1, 0, 0.5570, 0, 1, 0
0, 0.28, 1, 0, 0, 0.3840, 1, 0, 0
0, 0.43, 0, 0, 1, 0.5660, 0, 1, 0
0, 0.46, 0, 1, 0, 0.5880, 0, 1, 0
1, 0.57, 1, 0, 0, 0.6980, 1, 0, 0
0, 0.52, 0, 0, 1, 0.5940, 0, 1, 0
0, 0.31, 0, 0, 1, 0.4350, 0, 1, 0
0, 0.55, 1, 0, 0, 0.6200, 0, 0, 1
1, 0.50, 1, 0, 0, 0.5640, 0, 1, 0
1, 0.48, 0, 1, 0, 0.5590, 0, 1, 0
0, 0.22, 0, 0, 1, 0.3450, 1, 0, 0
1, 0.59, 0, 0, 1, 0.6670, 1, 0, 0
1, 0.34, 1, 0, 0, 0.4280, 0, 0, 1
0, 0.64, 1, 0, 0, 0.7720, 0, 0, 1
1, 0.29, 0, 0, 1, 0.3350, 0, 0, 1
0, 0.34, 0, 1, 0, 0.4320, 0, 1, 0
0, 0.61, 1, 0, 0, 0.7500, 0, 0, 1
1, 0.64, 0, 0, 1, 0.7110, 1, 0, 0
0, 0.29, 1, 0, 0, 0.4130, 1, 0, 0
1, 0.63, 0, 1, 0, 0.7060, 1, 0, 0
0, 0.29, 0, 1, 0, 0.4000, 1, 0, 0
0, 0.51, 1, 0, 0, 0.6270, 0, 1, 0
0, 0.24, 0, 0, 1, 0.3770, 1, 0, 0
1, 0.48, 0, 1, 0, 0.5750, 0, 1, 0
1, 0.18, 1, 0, 0, 0.2740, 1, 0, 0
1, 0.18, 1, 0, 0, 0.2030, 0, 0, 1
1, 0.33, 0, 1, 0, 0.3820, 0, 0, 1
0, 0.20, 0, 0, 1, 0.3480, 1, 0, 0
1, 0.29, 0, 0, 1, 0.3300, 0, 0, 1
0, 0.44, 0, 0, 1, 0.6300, 1, 0, 0
0, 0.65, 0, 0, 1, 0.8180, 1, 0, 0
0, 0.56, 1, 0, 0, 0.6370, 0, 0, 1
0, 0.52, 0, 0, 1, 0.5840, 0, 1, 0
0, 0.29, 0, 1, 0, 0.4860, 1, 0, 0
0, 0.47, 0, 1, 0, 0.5890, 0, 1, 0
1, 0.68, 1, 0, 0, 0.7260, 0, 0, 1
1, 0.31, 0, 0, 1, 0.3600, 0, 1, 0
1, 0.61, 0, 1, 0, 0.6250, 0, 0, 1
1, 0.19, 0, 1, 0, 0.2150, 0, 0, 1
1, 0.38, 0, 0, 1, 0.4300, 0, 1, 0
0, 0.26, 1, 0, 0, 0.4230, 1, 0, 0
1, 0.61, 0, 1, 0, 0.6740, 1, 0, 0
1, 0.40, 1, 0, 0, 0.4650, 0, 1, 0
0, 0.49, 1, 0, 0, 0.6520, 0, 1, 0
1, 0.56, 1, 0, 0, 0.6750, 1, 0, 0
0, 0.48, 0, 1, 0, 0.6600, 0, 1, 0
1, 0.52, 1, 0, 0, 0.5630, 0, 0, 1
0, 0.18, 1, 0, 0, 0.2980, 1, 0, 0
0, 0.56, 0, 0, 1, 0.5930, 0, 0, 1
0, 0.52, 0, 1, 0, 0.6440, 0, 1, 0
0, 0.18, 0, 1, 0, 0.2860, 0, 1, 0
0, 0.58, 1, 0, 0, 0.6620, 0, 0, 1
0, 0.39, 0, 1, 0, 0.5510, 0, 1, 0
0, 0.46, 1, 0, 0, 0.6290, 0, 1, 0
0, 0.40, 0, 1, 0, 0.4620, 0, 1, 0
0, 0.60, 1, 0, 0, 0.7270, 0, 0, 1
1, 0.36, 0, 1, 0, 0.4070, 0, 0, 1
1, 0.44, 1, 0, 0, 0.5230, 0, 1, 0
1, 0.28, 1, 0, 0, 0.3130, 0, 0, 1
1, 0.54, 0, 0, 1, 0.6260, 1, 0, 0

Test data:

# people_test_40.txt
#
0, 0.51, 1, 0, 0, 0.6120, 0, 1, 0
0, 0.32, 0, 1, 0, 0.4610, 0, 1, 0
1, 0.55, 1, 0, 0, 0.6270, 1, 0, 0
1, 0.25, 0, 0, 1, 0.2620, 0, 0, 1
1, 0.33, 0, 0, 1, 0.3730, 0, 0, 1
0, 0.29, 0, 1, 0, 0.4620, 1, 0, 0
1, 0.65, 1, 0, 0, 0.7270, 1, 0, 0
0, 0.43, 0, 1, 0, 0.5140, 0, 1, 0
0, 0.54, 0, 1, 0, 0.6480, 0, 0, 1
1, 0.61, 0, 1, 0, 0.7270, 1, 0, 0
1, 0.52, 0, 1, 0, 0.6360, 1, 0, 0
1, 0.30, 0, 1, 0, 0.3350, 0, 0, 1
1, 0.29, 1, 0, 0, 0.3140, 0, 0, 1
0, 0.47, 0, 0, 1, 0.5940, 0, 1, 0
1, 0.39, 0, 1, 0, 0.4780, 0, 1, 0
1, 0.47, 0, 0, 1, 0.5200, 0, 1, 0
0, 0.49, 1, 0, 0, 0.5860, 0, 1, 0
0, 0.63, 0, 0, 1, 0.6740, 0, 0, 1
0, 0.30, 1, 0, 0, 0.3920, 1, 0, 0
0, 0.61, 0, 0, 1, 0.6960, 0, 0, 1
0, 0.47, 0, 0, 1, 0.5870, 0, 1, 0
1, 0.30, 0, 0, 1, 0.3450, 0, 0, 1
0, 0.51, 0, 0, 1, 0.5800, 0, 1, 0
0, 0.24, 1, 0, 0, 0.3880, 0, 1, 0
0, 0.49, 1, 0, 0, 0.6450, 0, 1, 0
1, 0.66, 0, 0, 1, 0.7450, 1, 0, 0
0, 0.65, 1, 0, 0, 0.7690, 1, 0, 0
0, 0.46, 0, 1, 0, 0.5800, 1, 0, 0
0, 0.45, 0, 0, 1, 0.5180, 0, 1, 0
0, 0.47, 1, 0, 0, 0.6360, 1, 0, 0
0, 0.29, 1, 0, 0, 0.4480, 1, 0, 0
0, 0.57, 0, 0, 1, 0.6930, 0, 0, 1
0, 0.20, 1, 0, 0, 0.2870, 0, 0, 1
0, 0.35, 1, 0, 0, 0.4340, 0, 1, 0
0, 0.61, 0, 0, 1, 0.6700, 0, 0, 1
0, 0.31, 0, 0, 1, 0.3730, 0, 1, 0
1, 0.18, 1, 0, 0, 0.2080, 0, 0, 1
1, 0.26, 0, 0, 1, 0.2920, 0, 0, 1
0, 0.28, 1, 0, 0, 0.3640, 0, 0, 1
0, 0.59, 0, 0, 1, 0.6940, 0, 0, 1
Posted in Machine Learning | Leave a comment

“Data Smoothing with the Nadaraya-Watson Kernel Technique Using C#” in Visual Studio Magazine

I wrote an article titled “Data Smoothing with the Nadaraya-Watson Kernel Technique Using C#” in the November 2026 edition of Microsoft Visual Studio Magazine. See https://visualstudiomagazine.com/articles/2026/11/03/data-smoothing-with-the-nadaraya-watson-kernel-technique-using-c-sharp.aspx.

Nadaraya-Watson (NW) kernel smoothing is a classical statistics technique that removes noise in a dataset by using weighted averages to reveal patterns.

NW kernel smoothing uses a kernel function (often RBF) that measures the similarity between two vectors. The RBF kernel function bandwidth parameter controls the degree of smoothing.

NW kernel smoothing can be applied to both single-dimensional and multidimensional data.

I put together a demo to explain how the technique works. The demo source data is:

0.1000, 0.1000, 0.0998
0.2000, 0.2000, 0.3987
0.3000, 0.3000, 0.0955
0.5000, 0.5000, 0.5794
0.6000, 0.6000, 0.5646
0.7000, 0.7000, 0.9442
0.9000, 0.9000, 0.4833
1.0000, 1.0000, 0.5415

The programmatic output of the demo is:

Begin Nadaraya-Watson kernel smoothing demo

Loading simple XXy data
Done

First three data X:
  0.1000  0.1000
  0.2000  0.2000
  0.3000  0.3000

First three data y:
  0.0998
  0.3987
  0.0955

Creating NW kernel smoother with RBF sigma = 0.1600
Done

Computing smoothed y values
Done

   X             y        smooth y
-------------------------------------
  0.10  0.10    0.0998    0.2069
  0.20  0.20    0.3987    0.2303
  0.30  0.30    0.0955    0.2474
  0.50  0.50    0.5794    0.5599
  0.60  0.60    0.5646    0.6681
  0.70  0.70    0.9442    0.7357
  0.90  0.90    0.4833    0.5556
  1.00  1.00    0.5415    0.5255

End NW kernel smoothing demo

Data smoothing is most often used with small datasets that have a single x value and a single y value. This allows the source data and the smoothed data to be visualized using a graph. However, I used data with two x values, such as (0.10, 0.10), where both values are artificially the same, to demonstrate that NW kernel smoothing can use multidimensional X data with several values. The following graph uses just the first element of each x vector so that the data can be displayed in a 2D graph:

When using NW kernel smoothing, you must specify a kernel function and its parameter or parameters. There are many different kernel functions.

The demo uses the radial basis function (RBF) kernel, with a sigma parameter set to 0.16. Smaller values of sigma produce a more jagged approximation of the source data, and larger values produce a smoother approximation. Data smoothing is subjective, and the choice of a good RBF sigma value will vary from problem to problem.

Nadaraya-Watson kernel smoothing is named after Elizbar Nadaraya (born 1936, Soviet Georgia) and Geoffrey Watson (1921-1998, Australia), who came up with the idea independently in 1964. Their original intent was to use the technique for regression rather than data smoothing.

It is possible to use the ideas in NW kernel smoothing to perform regression. Instead of computing smoothed values for existing X data points, you can compute a predicted (smoothed) value for a new, previously unseen X data point.



Actor Russell Johnson (1924-2014) is best known for his role as the Professor on the “Gilligan’s Island” TV series (1964-67). Johnson also appeared in three of my favorite science fiction films of the 1950s.

Left: In “Attack of the Crab Monsters” (1957), a scientific expedition on a remote Pacific island runs into a pair of huge, intelligent, evil crabs (accidentally produced by atomic testing). Low-budget but an effective film. My grade = B.

Center: In “This Island Earth” (1955), aliens from the planet Metaluna recruit Earth scientists in an effort to convert lead to uranium, which is deperately needed in a war with the Zagons. My grade = B+.

Right: In “It Came from Outer Space” (1953), an alien spacecraft crash lands in the Southwest desert. Aliens capture people from the local town to help them repair their ship. They succeed and there’s a happy ending. My grade = A-.


Posted in Miscellaneous | Leave a comment

NFL 2026 Week 10 Predictions – Zoltar Likes the Patriots and the Bills

Zoltar is my NFL football prediction computer program. It uses a neural network and a form of reinforcement learning. Here are Zoltar’s predictions for week #10 of the 2026 season.

Zoltar:      giants  by    6  opp =  commanders    | Vegas:      giants  by  1.5
Zoltar:    patriots  by    3  opp =       lions    | Vegas:       lions  by  2.5
Zoltar:      chiefs  by    0  opp =     falcons    | Vegas:      chiefs  by  3.5
Zoltar:       bills  by   12  opp =        jets    | Vegas:       bills  by    7
Zoltar:       colts  by    4  opp =    dolphins    | Vegas:       colts  by  2.5
Zoltar:     packers  by    4  opp =     vikings    | Vegas:     packers  by    3
Zoltar:    panthers  by    0  opp =      saints    | Vegas:    panthers  by  1.5
Zoltar:     jaguars  by   12  opp =      titans    | Vegas:     jaguars  by  9.5
Zoltar:      texans  by    0  opp =      browns    | Vegas:      texans  by  4.5
Zoltar:    seahawks  by    5  opp =     raiders    | Vegas:    seahawks  by  3.5
Zoltar:        rams  by    4  opp =   cardinals    | Vegas:        rams  by  7.5
Zoltar: fortyniners  by    4  opp =     cowboys    | Vegas: fortyniners  by  4.5
Zoltar:    steelers  by    0  opp =     bengals    | Vegas:     bengals  by  3.5
Zoltar:      ravens  by    5  opp =    chargers    | Vegas:      ravens  by  3.5

Zoltar theoretically suggests betting when the Vegas line is “significantly” different from Zoltar’s prediction. In mid-season I typically use a threshold of 4.5 points difference but in early and late parts of the season I use a more aggressive threshold of 3.0 or 3.5 points difference.

For week #10, using 4.5 points as the threshold, Zoltar likes one Vegas underdog and one Vegas favorite:

patriots     at        lions: Bet on Vegas underdog patriots
bills        at         jets: Bet on Vegas favorite  bills

A bet on the Vegas underdog Patriots against the Lions will pay off if the Patriots win by any score, or if the favored Lions win but by less than 2.5 points (i.e., 1 point or 2 points). If the favored team wins by exactly the point spread, the wager is a push. This is why point spreads often have a 0.5 added — called “the hook” — to eliminate pushes.

I use the early Vegas point spreads, usually posted on late Monday night, right after the Monday Night Football game. By the time you read this, the point spreads will likely have changed. I’ve noticed that, compared to last year, point spreads are changing more dramatically by late Tuesdays than they used to. A swing of 7 points is not uncommon. Note that when I wrote this post, the Bills line was changing wildly.

I speculate that big point spread changes are due to a huge increase in betting. When a lot of money is bet on one team in a matchup, the bookmakers must make a huge change in the point spread to encourage betting on the other team. Bookmakers only make money when the betting amounts are close to equal on both teams.

This is the part of the season where injuries have a big effect on the point spread. Weeks 10 through 14 are usually Zoltar’s weakest.

Theoretically, if you must bet $110 to win $100 (typical in Vegas) then you’ll make money if you predict at 53% accuracy or better. But realistically, you need to predict at 60% accuracy or better, to take into account overhead and miscellaneous mistakes.

In week #9, against the Vegas point spread, Zoltar went a poor 1-2 using 4.5 points as the advice threshold because. And sadly, the two games Zoltar missed on (underdog Ravens, underdog Eagles) were not close. Although the human-me would never have bet on these two games because the Ravens and Eagles starting QBs were out with injuries, and the backups were particularly weak.

For the season, against the spread, Zoltar is 15-8 (~65% accuracy).

Just for fun, I track how well Zoltar does when just trying to predict just which team will win a game. This isn’t useful except for parlay betting. In week #9, just predicting the winning team, Zoltar went only 9-3 with 3 games to close to call, which is OK (75% correct). Vegas was incredible, going 14-1 at just predicting the winning team — the best result I can remember in recent memory.



My prediction system is math-based in the sense that it computes a numeric rating for each team and then uses ratings to compute the predicted margin of victory of the better team. An entirely different approach for predicting NFL football scores is to use a sophisticated simulation program and then simulate a game many thousands of times.

Left: There are many versions of simple football simulation games where a player rolls two ordinary dice and the result is based on the 21 possible outcomes (because order doesn’t matter). But here is an interesting version with three dice so there are 56 possible outcomes.

Center: “1st and Goal” by R and R Games is a fairly sophisticated game that uses several kinds of specialized dice and cards.

Right: “Half-Time Football” by Lakeside was produced in the late 1970s and early 80s. It’s strictly a dice game and is medium complexity.


Posted in Zoltar | Leave a comment

Experiment: Simple Encoding vs. Sophisticated Encoding for Gradient Boosting (Tree-Based) Regression

Bottom line: Theory says the sophisticated, complex encoding technique should work better than a super simple encoding technique. In one experiment, the simple encoding technique worked slightly better.

I took a set of synthetic source training data that looks like this:

M,29,Austin,expert,68.5,timid,54600.00
F,51,Juneau,novice,60.0,brave,25700.00
M,32,Boston,master,73.5,alert,74900.00
M,19,Irvine,novice,61.0,brave,30300.00
. . .

Each row represents a person. The fields are sex, age, city (10 values), skill (3 values), height (inches), psychology (3 values), income (dollars). The goal is to predict income from the other six variables, using gradient boosting regression with decision tree weak learners.

There are 200 training items and 40 test items.

I normalized the numeric predictors, age and height, and the target variable income by dividing by 100, 100, 100,000 respectively. I did this just for convenience. The categorical predictors had to be converted to numbers because I wasn’t using a CART style tree.

There are two main schemes to encode categorical data for use with tree-based systems. The first encoding scheme is super simple: use ordinal encoding regardless of categorical type. So for sex, M = 0, F = 1. For city, Austin = 0, Boston = 1, Casper = 2, Denver = 3, Eugene = 4, Fresno = 5, Gallup = 6, Helena = 7, Irvine = 8, Juneau = 9. For skill, novice = 0, expert = 1, master = 2. For psychology, timid = 0, alert = 1, brave = 2.

The super simple normalized and encoded data looks like:

0, 0.29, 0, 1, 0.685, 0, 0.54600
1, 0.51, 9, 0, 0.600, 2, 0.25700
0, 0.32, 1, 2, 0.735, 1, 0.74900
0, 0.19, 8, 0, 0.610, 2, 0.30300
. . . 

With the super simple encoding, you can encode and then perform a train-test split. Easy.

The second encoding scheme is quite complicated. The complex normalized and encoded data looks like:

0, 0.29, 0.383784, 1, 0.685, 1, 0, 0, 0.54600
1, 0.51, 0.580719, 0, 0.600, 0, 0, 1, 0.25700
0, 0.32, 0.363510, 2, 0.735, 0, 1, 0, 0.74900
0, 0.19, 0.515110, 0, 0.610, 0, 0, 1, 0.30300
. . .

The city variable is smoothed-target encoded with smoothing parameter m=10. The psychology variable is one-hot encoded. When using smoothed target encoding, there is a risk of leakage. So you must first split, then compute the smoothed encodings on the training data, then apply to both train and test data.

I ran both versions of the encoded data through a from-scratch C# implementation of gradient boosting regression. The key output of the super simple encoding scheme:

Setting numTrees = 200
Setting maxDepth = 4
Setting minSamples = 2
Setting minLeaf = 1
Setting lrnRate = 0.0500

Creating and training GradientBoostRegression model
Done

Evaluating model

Accuracy train (within 0.10) = 1.0000
Accuracy test (within 0.10) = 0.8000

MSE train = 0.0000
MSE test = 0.0017

R2 train = 0.9994
R2 test = 0.9484

The key output of the complicated encoding scheme:

Setting numTrees = 200
Setting maxDepth = 4
Setting minSamples = 2
Setting minLeaf = 1
Setting lrnRate = 0.0500

Creating and training GradientBoostRegression model
Done

Evaluating model

Accuracy train (within 0.10) = 1.0000
Accuracy test (within 0.10) = 0.7750

MSE train = 0.0000
MSE test = 0.0022

R2 train = 0.9993
R2 test = 0.9360

The super simple encoding scheme is inferior to the complex encoding scheme, but the simple scheme worked slightly (but not statistically significant) better. The reasons why super simple encoding works well with gradient boosting regression are very complicated and subtle — quite literally pages of ideas and details.

I have seen this phenomenon many times in machine learning: complicated doesn’t always beat simple.



I have advanced college degrees in math and computer science, but my first degree was in psychology from UC Irvine. One of my requirements was a class in sociology. I remember learning a few things about strip club science.

Sociologists K. Frank and B. Barton studied how dancers manage customer relationships. The key finding is that the primary driver of high earnings for girls in strip clubs isn’t their performance on stage, but the dancer’s ability to create counterfeit intimacy — making a man feel listened to and valued — something often missing from a marriage.

The researchers found that strip club patrons often pay their “regular girl” thousands of dollars primarily for conversation and emotional connection. In many cases there is no physical component of any kind in the relationship.

Strip clubs and their equivalents have been around for literally thousands of years, dating back to ancient Greece and Rome.


Posted in Machine Learning | Leave a comment

I Use AI to Improve My Gradient Boosting Regression System Using JavaScript

Bottom line: I used AI to analyze my existing implementation of gradient boosting regression with JavaScript, and it came up with some significant performance improvements.

The goal of a machine learning regression problem is to predict a single numeric value. For example, a bank might want to predict the maximum safe loan amount to a customer, based on age, account balance, current debt, and so on.

Gradient boosting regression (GBR) is a tree-based technique. GBR creates a collection of small decision trees (typically about 100) where, loosely speaking, each tree learns how to predict training items that were poorly predicted by the previous tree by predicting the amount needed to add (residuals) to a running prediction value. The actual algorithm is very deep and subtle, and is based on predicting “residuals”. If you want to give yourself a headache, read the Wikipedia explanation of Gradient Boosting.

I submitted my existing JavaScript GBR system to AI (ChatGPT was much better than Gemini) and it found several significant performance improvements. I incorporated the performance improvements. For my demo, I used one of my standard datasets. The output of my demo is:

JavaScript gradient boosting regression

Loading synthetic train (200), test (40)
Done

First three train x:
-0.1660   0.4406  -0.9998  -0.3953  -0.7065
 0.0776  -0.1616   0.3704  -0.5911   0.7562
-0.9452   0.3409  -0.1654   0.1174  -0.7192

First three train y:
0.4840 0.1568 0.8054

Setting numTrees = 200

Setting maxDepth = 3
Setting minSamples = 2
Setting minLeaf = 1
Setting lrnRate = 0.1000

Creating and training gradient boosting regression model
Done

Evaluating model

Train acc (within 0.10) = 0.9950
Test acc (within 0.10) = 0.7250

Train MSE = 0.0000
Test MSE = 0.0011

Train R2 = 0.9996
Test R2 = 0.9600

Predicting for x =
-0.1660   0.4406  -0.9998  -0.3953  -0.7065
y = 0.4829

Explaininng for x =
-0.1660   0.4406  -0.9998  -0.3953  -0.7065

Init prediction: 0.3493
t =   0  pred_res =   0.0608  delta =   0.0061  pred =  0.3554
t =   1  pred_res =   0.0840  delta =   0.0084  pred =  0.3638
t =   2  pred_res =   0.0704  delta =   0.0070  pred =  0.3708
t =   3  pred_res =   0.1074  delta =   0.0107  pred =  0.3816
t =   4  pred_res =   0.0243  delta =   0.0024  pred =  0.3840
. . .
t = 198  pred_res =  -0.0001  delta =  -0.0000  pred =  0.4829
t = 199  pred_res =   0.0002  delta =   0.0000  pred =  0.4829

End gradient boost regression demo

The initial prediction is the average of the target y values. Each tree adds a fraction (the learning rate) of the predicted residual. By the final few trees, the predicted residuals get close to zero and so the final prediction doesn’t change much.

Notice that the model still overfits quite a bit. This is because the synthetic data is more or less spread evenly, which is not a good scenario for tree-based regression techniques.

The accuracy of the new revised system is the same as the the accuracy of the old system. But the new system is faster and can scale to datasets with hundreds of thousands of rows.

I validated the new performant gradient boosting system by running the demo data through the scikit-learn GradientBoostingRegressor module. I got essentially the same results:

Begin scikit gradient boost regression demo

Loading synthetic train (200), test (400) data
Done

Setting n_estimators = 200
Setting max_depth = 3
Setting min_samples_split = 2
Setting min_samples_leaf = 1
Setting lrn_rate = 0.1000

Training gradient boost regression model
Done

Evaluating model

Acc train (within 0.10): 0.9950
Acc test (within 0.10): 0.7250

MSE train: 0.0000
MSE test: 0.0011

R2 train: 0.9996
R2 test: 0.9603

Good fun.



Machine learning regression algorithms find hidden patterns in data.

Every Playboy Magazine (except for the first issue in December 1953) has the company’s bunny head logo somewhere on the cover. In most cases the logo is large, clear, and obvious. But on some covers, the logo is cleverly hidden, and it’s fun to try and find the logo.

Left: April 1981. The bunny logo is hidden as part of the model’s swimsuit design, in the light colored fabric panel, just above her right hand.

Right: March 1987. The bunny logo is cleverly disguised as folds in the model’s turquoise shirt, on the folded cuff of the right hand (a bit above the bar code).


Demo program. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols (my blog editor often chokes on symbols).

// gradient_boost_regression.js
// node.js v22.16.0
// verified by OpenAI GPT-x

const FS = require('fs');

// ==========================================================

class GradientBoostRegressor
{
  constructor(nTrees, maxDepth, minSamples, minLeaf,
    numSplitCols, lrnRate, seed = 0)
  {
    this.lrnRate = lrnRate;
    this.nTrees = nTrees;
    this.maxDepth = maxDepth;
    this.minSamples = minSamples;
    this.minLeaf = minLeaf;
    this.numSplitCols = numSplitCols;

    this.trees = [];

    this.pred0 = 0.0;  // initial prediction
    this.seed = seed + 0.5;
  }

  // --------------------------------------------------------

  train(trainX, trainY)
  {
    let n = trainX.length;

    // initial pred = mean of training targets
    this.pred0 = this.mean(trainY);

    // Current prediction for each training item
    let preds = new Array(n);
    for (let i = 0; i "lt" n; ++i)
      preds[i] = this.pred0;

    // residuals are target for the current tree
    let residuals = new Array(n);

    // train each tree
    for (let t = 0; t "lt" this.nTrees; ++t) {
      for (let i = 0; i "lt" n; ++i)
        residuals[i] = trainY[i] - preds[i];

      let treeSeed = this.nextInt(0, 2147483647);
      let dtr = new DecisionTreeRegressor(
        this.maxDepth,
        this.minSamples,
        this.minLeaf,
        this.numSplitCols,
        false,       // don't save rows info
        treeSeed
      );

      // train tree to predict residuals
      dtr.train(trainX, residuals);

      for (let i = 0; i "lt" n; ++i) {
        let predResidual = dtr.predict(trainX[i]);
        preds[i] += this.lrnRate * predResidual;
      }

      // add trained tree
      this.trees.push(dtr);
    }
  } // train()

  // --------------------------------------------------------

  predict(x)
  {
    let result = this.pred0;

    for (let t = 0; t "lt" this.nTrees; ++t) {
      let predResidual = this.trees[t].predict(x);
      let delta = this.lrnRate * predResidual;
      result += delta;
    }

    return result;
  }

  // --------------------------------------------------------

  explain(x)
  {
    let result = this.pred0;
    console.log("\nInit prediction: " + result.toFixed(4));

    for (let t = 0; t "lt" this.nTrees; ++t) {
      let predResidual = this.trees[t].predict(x);
      let delta = this.lrnRate * predResidual;

      result += delta;

      // first 5 trees and final 2 trees
      if (
      (t "gte" 0 && t "lte" 4) ||
      (t "gte" this.nTrees - 2 && t "lte" this.nTrees - 1)) {
        let tStr = t.toString().padStart(3, " ");
        let resStr = 
          predResidual.toFixed(4).padStart(8, " ");
        let deltaStr = 
          delta.toFixed(4).padStart(8, " ");
        let predStr = 
          result.toFixed(4).padStart(7, " ");

        console.log(
          "t = " + tStr +
          "  pred_res = " + resStr +
          "  delta = " + deltaStr +
          "  pred = " + predStr
        );
      } // if

      if (t === 5)
        console.log(". . . ");
    } // t

    return result;
  }

  // --------------------------------------------------------

  accuracy(dataX, dataY, pctClose)
  {
    let nCorrect = 0;
    let nWrong = 0;

    for (let i = 0; i "lt" dataX.length; ++i) {
      let actualY = dataY[i];
      let predY = this.predict(dataX[i]);

      if (Math.abs(predY - actualY) "lt" 
        (pctClose * Math.abs(actualY)))
        ++nCorrect;
      else
        ++nWrong;
    }

    return (nCorrect * 1.0) / (nCorrect + nWrong);
  }

  // --------------------------------------------------------

  mse(dataX, dataY)
  {
    let n = dataX.length;
    let sum = 0.0;

    for (let i = 0; i "lt" n; ++i) {
      let actualY = dataY[i];
      let predY = this.predict(dataX[i]);
      sum += (actualY - predY) * (actualY - predY);
    }

    return sum / n;
  }

  // --------------------------------------------------------

  r2(dataX, dataY)
  {
    // coefficient of determination
    let n = dataX.length;

    let sum = 0.0;
    for (let i = 0; i "lt" n; ++i)
      sum += dataY[i];

    let meanActual = sum / n;
    let ssRes = 0.0;
    let ssTot = 0.0;

    for (let i = 0; i "lt" n; ++i) {
      let predY = this.predict(dataX[i]);

      ssRes += (dataY[i] - predY) * (dataY[i] - predY);

      ssTot += (dataY[i] - meanActual) *
        (dataY[i] - meanActual);
    }

    if (Math.abs(ssTot) "lt" 1.0e-12)
      return 0.0;
    else
      return 1.0 - (ssRes / ssTot);
  }

  // --------------------------------------------------------

  mean(data)
  {
    let n = data.length;
    let sum = 0.0;

    for (let i = 0; i "lt" n; ++i)
      sum += data[i];

    return sum / n;
  }

  // --------------------------------------------------------

  next()    // simulated random-number generator
  {
    let x = Math.sin(this.seed++) * 10000;
    return x - Math.floor(x);
  }

  // --------------------------------------------------------

  nextInt(lo, hi)
  {
    return Math.trunc(
      (hi - lo) * this.next() + lo
    );
  }

} // class GradientBoostRegressor

// ==========================================================
// ==========================================================

class DecisionTreeRegressor
{
  // internal Node factory
  createNode()
  {
    // return an object literal
    return {
      id: -1,
      colIdx: -1,
      thresh: 0.0,
      left: -1,
      right: -1,
      value: 0.0,
      isLeaf: false,
      rows: null,
    };
  }

  // --------------------------------------------------------
  // primary: ctor(), train(), predict()
  // help: bestSplit(), treeTargetMean(), next(),
  //  nextInt(), sortParallel()
  // -------------------------------------------------------

  constructor(maxDepth, minSamples, minLeaf, numSplitCols,
    saveRows, seed)
  {
    this.maxDepth = maxDepth;
    this.minSamples = minSamples;
    this.minLeaf = minLeaf;
    this.numSplitCols = numSplitCols;
    this.saveRows = saveRows;

    let numNodes = Math.pow(2, maxDepth + 1) - 1;
    this.tree = new Array(numNodes).fill(null);

    this.seed = seed + 0.5; // avoid 0
    this.trainX = null;
    this.trainY = null;
  }

  // --------------------------------------------------------

  train(trainX, trainY)
  {
    this.trainX = trainX;
    this.trainY = trainY;

    let maxParentID = Math.pow(2, this.maxDepth) - 2;
    
    let allRows = new Array(this.trainX.length);
    for (let i = 0; i "lt" this.trainX.length; ++i)
      allRows[i] = i;

    let root = this.createNode();
    root.id = 0;
    root.value = this.treeTargetMean(allRows);
    root.rows = allRows;
    this.tree[0] = root;

    for (let i = 0; i "lt" this.tree.length; ++i) {
      let currNode = this.tree[i];
      if (currNode == null) continue;

      let leftID = currNode.id * 2 + 1;
      let rightID = currNode.id * 2 + 2;

      if (currNode.id "gt" maxParentID || 
        currNode.rows.length "lt" this.minSamples ||
        rightID "gte" this.tree.length) {
        currNode.isLeaf = true;
        continue;
      }

      let splitInfo = this.bestSplit(currNode.rows);
      let colIdx = splitInfo[0];
      let splitVal = splitInfo[1];

      // split failure
      if (colIdx == -1) {
        currNode.isLeaf = true;
        continue;
      }

      // split success
      currNode.colIdx = colIdx;
      currNode.thresh = splitVal;

      let leftIdxs = [];
      let rightIdxs = [];

      for (let k = 0; k "lt" currNode.rows.length; ++k) {
        let r = currNode.rows[k];
        if (this.trainX[r][colIdx] "lte" splitVal)
          leftIdxs.push(r);
        else
          rightIdxs.push(r);
      }

      // create and attach children
      currNode.left = leftID;
      let n1 = this.createNode();
      n1.id = leftID;
      n1.rows = leftIdxs;
      n1.value = this.treeTargetMean(leftIdxs);
      this.tree[leftID] = n1;

      currNode.right = rightID;
      let n2 = this.createNode();
      n2.id = rightID;
      n2.rows = rightIdxs;
      n2.value = this.treeTargetMean(rightIdxs);
      this.tree[rightID] = n2;
    } // main loop

    // delete row info to save memory
    if (this.saveRows == false) {
      for (let i = 0; i "lt" this.tree.length; ++i) {
        if (this.tree[i] != null)
          this.tree[i].rows = null;
      }
    }
  } // train()

  // --------------------------------------------------------

  predict(x)
  {
    let p = 0;
    let lastValidValue = 0.0;

    while (p != -1 && p "lt" this.tree.length) {
      let currNode = this.tree[p];
      if (currNode == null) break;

      lastValidValue = currNode.value;

      if (
        currNode.isLeaf ||
        currNode.colIdx == -1 ||
        currNode.left "gte" this.tree.length ||
        currNode.right "gte" this.tree.length
      )
        break;

      if (x[currNode.colIdx] "lte" currNode.thresh)
        p = currNode.left;
      else
        p = currNode.right;
    }
    return lastValidValue;
  }

  // --------------------------------------------------------
  // helpers
  // --------------------------------------------------------

  bestSplit(rows)
  {
    let bestColIdx = -1;
    let bestThresh = 0.0;
    let bestGain = -1.0;
    let nRows = rows.length;
    let nCols = this.trainX[0].length;

    if (nRows === 0)
      throw new Error("Empty data in bestSplit()");

    let colIndices = new Array(nCols);
    for (let k = 0; k "lt" nCols; ++k)
      colIndices[k] = k;

    // Fisher-Yates column selection
    for (let i = 0; i "lt" nCols; ++i) {
      let ri = this.nextInt(i, nCols);
      let tmp = colIndices[i];
      colIndices[i] = colIndices[ri];
      colIndices[ri] = tmp;
    }

    let numColsToUse;
    if (this.numSplitCols === -1)
      numColsToUse = nCols;
    else
      numColsToUse = Math.min(this.numSplitCols, nCols);

    let sortedRows = new Array(nRows);
    let featureKeys = new Array(nRows);

    let totalSum = 0.0;
    for (let i = 0; i "lt" nRows; ++i)
      totalSum += this.trainY[rows[i]];

    for (let j = 0; j "lt" numColsToUse; ++j) {
      let colIdx =colIndices[j];
      for (let i = 0; i "lt" nRows; ++i) {
        let r = rows[i];
        sortedRows[i] = r;
        featureKeys[i] = this.trainX[r][colIdx];
      } // i

      // sort featureKeys, sortedRows in parallel
      // based on order of featureKeys
      this.sortParallel(featureKeys, sortedRows);
      
      let leftSum = 0.0;
      for (let i = 0; i "lt" nRows-1; ++i) {
        leftSum += this.trainY[sortedRows[i]];
        let leftCount = i + 1;
        let rightCount = nRows - leftCount;

        if (featureKeys[i] == featureKeys[i+1])
          continue;  // duplicate

        if (leftCount "lt" this.minLeaf ||
          rightCount "lt" this.minLeaf)
          continue;

        let rightSum = totalSum - leftSum;

        let gain = (leftSum * leftSum / leftCount) +
          (rightSum * rightSum / rightCount);

        if (gain "gt" bestGain) {
          bestGain = gain;
          bestColIdx = colIdx;
          bestThresh = featureKeys[i] + 
            (featureKeys[i+1] - featureKeys[i]) / 2.0;
        }
      } // i
    } // j

    return [bestColIdx, bestThresh];
  } // bestSplit()

  // --------------------------------------------------------

  next()
  {
    let x = Math.sin(this.seed++) * 10000; // avoid stuck 0
    return x - Math.floor(x);
  }

  // -------------------------------------------------------

  nextInt(lo, hi)
  {
    return Math.trunc((hi - lo) * this.next() + lo);
  }

  // --------------------------------------------------------

  treeTargetMean(rows)
  {
    let sum = 0.0;
    for (let i = 0; i "lt" rows.length; ++i) {
      sum += this.trainY[rows[i]];
    }
    return sum / rows.length;
  }

  // --------------------------------------------------------

  sortParallel(keys, values)
  {
    // sort keys and values in-place based on order of keys
    let n = keys.length;
    // let idxs = this.argsort(keys);

    let augmented = [];
    for (let i = 0; i "lt" n; ++i) {
      augmented[i] = [keys[i], i]; // add indices to values
    }

    augmented.sort(function(a, b){return a[0] - b[0]});

    let idxs = [];
    for (let i = 0; i "lt" n; ++i) {
      idxs[i] = augmented[i][1];  // fetch the indexes
    }

    let sortedKeys = new Array(n);
    let sortedValues = new Array(n);
    for (let i = 0; i "lt" n; ++i) {
      let idx = idxs[i]; 
      sortedKeys[i] = keys[idx];
      sortedValues[i] = values[idx];
    }
    for (let i = 0; i "lt" n; ++i) {
      keys[i] = sortedKeys[i];
      values[i] = sortedValues[i];
    }

  } // sortParallel()

  // --------------------------------------------------------

} // class DecisionTreeRegressor

// ==========================================================
// ==========================================================

function loadTxt(fn, delimit, usecols, comment)
{
  let all = FS.readFileSync(fn, "utf8").trim();
  let lines = all.split("\n");

  let nRows = 0;
  for (let i = 0; i "lt" lines.length; ++i) {
    if (!lines[i].startsWith(comment)) ++nRows;
  }

  let nCols = usecols.length;
  // let result = Array.from({ length: nRows },
  //   () ="gt" new Array(nCols));

  let result = [];
  for (let i = 0; i "lt" nRows; ++i) {
    result[i] = [];
    for (let j = 0; j "lt" nCols; ++j) {
      result[i][j] = 0.0;
    }
  }

  let r = 0; let i = 0;
  while (r "lt" lines.length) {
    if (lines[r].startsWith(comment)) {
      ++r;
    } 
    else {
      let tokens = lines[r].split(delimit);
      for (let j = 0; j "lt" nCols; ++j) {
        result[i][j] = parseFloat(tokens[usecols[j]]);
      }
      ++r;
      ++i;
    }
  }

  if (usecols.length === 1) {
    let vec = new Array(nRows);
    for (let k = 0; k "lt" nRows; ++k)
      vec[k] = result[k][0];
    return vec;
  }

  return result;
}

// ----------------------------------------------------------

function vecShow(v, dec, len)
{
  for (let i = 0; i "lt" v.length; ++i) {
    if (i != 0 && i % len == 0) {
      process.stdout.write("\n");
    }
    if (v[i] "gte" 0.0) {
      process.stdout.write(" ");  // + or - space
    }
    process.stdout.write(v[i].toFixed(dec));
    process.stdout.write("  ");
  }
  process.stdout.write("\n");
}

// ----------------------------------------------------------

function main()
{
  console.log("\nJavaScript gradient boosting regression ");

  // 1. load data
  console.log("\nLoading synthetic train (200), test (40)");
  let trainFile = ".\\Data\\synthetic_train_200.txt";
  let testFile = ".\\Data\\synthetic_test_40.txt";

  let trainX = loadTxt(trainFile, ",", [0,1,2,3,4], "#");
  let trainY = loadTxt(trainFile, ",", [5], "#");
  let testX = loadTxt(testFile, ",", [0,1,2,3,4], "#");
  let testY = loadTxt(testFile, ",", [5], "#");
  console.log("Done ");

  console.log("\nFirst three train x: ");
  for (let i = 0; i "lt" 3; ++i)
    vecShow(trainX[i], 4, 8);  // 4 decimals 8 wide
  console.log("\nFirst three train y: ");
  for (let i = 0; i "lt" 3; ++i)
    process.stdout.write(trainY[i].toString() + " ");
  console.log("");

  // 2. create and train model
  let numTrees = 200;
  let maxDepth = 3;
  let minSamples = 2;
  let minLeaf = 1;
  let numSplitCols = -1;  // use all columns
  let saveRows = false;
  let lrnRate = 0.10;
  let seed = 0;

  console.log("\nSetting numTrees = " +
    numTrees.toString());
  console.log("\nSetting maxDepth = " +
    maxDepth.toString());
  console.log("Setting minSamples = " +
    minSamples.toString());
  console.log("Setting minLeaf = " +
    minLeaf.toString());
  console.log("Setting lrnRate = " +
    lrnRate.toFixed(4).toString());
  // console.log("Using default numSplitCols = -1 (all)");

  console.log("\nCreating and training gradient " +
    "boosting regression model ");
  let gbr = 
    new GradientBoostRegressor(numTrees, maxDepth,
      minSamples, minLeaf, numSplitCols, lrnRate, seed);

  gbr.train(trainX, trainY);
  console.log("Done ");

  // 3. evaluate model
  console.log("\nEvaluating model ");
  let trainAcc = gbr.accuracy(trainX, trainY, 0.10);
  let testAcc = gbr.accuracy(testX, testY, 0.10);
  console.log("\nTrain acc (within 0.10) = " +
    trainAcc.toFixed(4).toString());
  console.log("Test acc (within 0.10) = " +
    testAcc.toFixed(4).toString());

  let trainMSE = gbr.mse(trainX, trainY);
  let testMSE = gbr.mse(testX, testY);
  console.log("\nTrain MSE = " +
    trainMSE.toFixed(4).toString());
  console.log("Test MSE = " +
    testMSE.toFixed(4).toString());

  let trainR2 = gbr.r2(trainX, trainY);
  let testR2 = gbr.r2(testX, testY);
  console.log("\nTrain R2 = " +
    trainR2.toFixed(4).toString());
  console.log("Test R2 = " +
    testR2.toFixed(4).toString());

  // 4. use tree to make prediction
  let x = trainX[0];
  console.log("\nPredicting for x = ");
  vecShow(x, 4, 9);
  let predY = gbr.predict(x);
  console.log("y = " + predY.toFixed(4).toString());

  console.log("\nExplaininng for x = ");
  vecShow(x, 4, 9);
  gbr.explain(x);

  console.log("\nEnd gradient boost regression demo ");
}

main();

Training data:

# synthetic_train_200.txt
#
-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
-0.8299, -0.9219, -0.6603,  0.7563, -0.8033,  0.7955
 0.0663,  0.3838, -0.3690,  0.3730,  0.6693,  0.3206
-0.9634,  0.5003,  0.9777,  0.4963, -0.4391,  0.7377
-0.1042,  0.8172, -0.4128, -0.4244, -0.7399,  0.4801
-0.9613,  0.3577, -0.5767, -0.4689, -0.0169,  0.6861
-0.7065,  0.1786,  0.3995, -0.7953, -0.1719,  0.5569
 0.3888, -0.1716, -0.9001,  0.0718,  0.3276,  0.2500
 0.1731,  0.8068, -0.7251, -0.7214,  0.6148,  0.3297
-0.2046, -0.6693,  0.8550, -0.3045,  0.5016,  0.2129
 0.2473,  0.5019, -0.3022, -0.4601,  0.7918,  0.2613
-0.1438,  0.9297,  0.3269,  0.2434, -0.7705,  0.5171
 0.1568, -0.1837, -0.5259,  0.8068,  0.1474,  0.3307
-0.9943,  0.2343, -0.3467,  0.0541,  0.7719,  0.5581
 0.2467, -0.9684,  0.8589,  0.3818,  0.9946,  0.1092
-0.6553, -0.7257,  0.8652,  0.3936, -0.8680,  0.7018
 0.8460,  0.4230, -0.7515, -0.9602, -0.9476,  0.1996
-0.9434, -0.5076,  0.7201,  0.0777,  0.1056,  0.5664
 0.9392,  0.1221, -0.9627,  0.6013, -0.5341,  0.1533
 0.6142, -0.2243,  0.7271,  0.4942,  0.1125,  0.1661
 0.4260,  0.1194, -0.9749, -0.8561,  0.9346,  0.2230
 0.1362, -0.5934, -0.4953,  0.4877, -0.6091,  0.3810
 0.6937, -0.5203, -0.0125,  0.2399,  0.6580,  0.1460
-0.6864, -0.9628, -0.8600, -0.0273,  0.2127,  0.5387
 0.9772,  0.1595, -0.2397,  0.1019,  0.4907,  0.1611
 0.3385, -0.4702, -0.8673, -0.2598,  0.2594,  0.2270
-0.8669, -0.4794,  0.6095, -0.6131,  0.2789,  0.4700
 0.0493,  0.8496, -0.4734, -0.8681,  0.4701,  0.3516
 0.8639, -0.9721, -0.5313,  0.2336,  0.8980,  0.1412
 0.9004,  0.1133,  0.8312,  0.2831, -0.2200,  0.1782
 0.0991,  0.8524,  0.8375, -0.2102,  0.9265,  0.2150
-0.6521, -0.7473, -0.7298,  0.0113, -0.9570,  0.7422
 0.6190, -0.3105,  0.8802,  0.1640,  0.7577,  0.1056
 0.6895,  0.8108, -0.0802,  0.0927,  0.5972,  0.2214
 0.1982, -0.9689,  0.1870, -0.1326,  0.6147,  0.1310
-0.3695,  0.7858,  0.1557, -0.6320,  0.5759,  0.3773
-0.1596,  0.3581,  0.8372, -0.9992,  0.9535,  0.2071
-0.2468,  0.9476,  0.2094,  0.6577,  0.1494,  0.4132
 0.1737,  0.5000,  0.7166,  0.5102,  0.3961,  0.2611
 0.7290, -0.3546,  0.3416, -0.0983, -0.2358,  0.1332
-0.3652,  0.2438, -0.1395,  0.9476,  0.3556,  0.4170
-0.6029, -0.1466, -0.3133,  0.5953,  0.7600,  0.4334
-0.4596, -0.4953,  0.7098,  0.0554,  0.6043,  0.2775
 0.1450,  0.4663,  0.0380,  0.5418,  0.1377,  0.2931
-0.8636, -0.2442, -0.8407,  0.9656, -0.6368,  0.7429
 0.6237,  0.7499,  0.3768,  0.1390, -0.6781,  0.2185
-0.5499,  0.1850, -0.3755,  0.8326,  0.8193,  0.4399
-0.4858, -0.7782, -0.6141, -0.0008,  0.4572,  0.4197
 0.7033, -0.1683,  0.2334, -0.5327, -0.7961,  0.1776
 0.0317, -0.0457, -0.6947,  0.2436,  0.0880,  0.3345
 0.5031, -0.5559,  0.0387,  0.5706, -0.9553,  0.3107
-0.3513,  0.7458,  0.6894,  0.0769,  0.7332,  0.3170
 0.2205,  0.5992, -0.9309,  0.5405,  0.4635,  0.3532
-0.4806, -0.4859,  0.2646, -0.3094,  0.5932,  0.3202
 0.9809, -0.3995, -0.7140,  0.8026,  0.0831,  0.1600
 0.9495,  0.2732,  0.9878,  0.0921,  0.0529,  0.1289
-0.9476, -0.6792,  0.4913, -0.9392, -0.2669,  0.5966
 0.7247,  0.3854,  0.3819, -0.6227, -0.1162,  0.1550
-0.5922, -0.5045, -0.4757,  0.5003, -0.0860,  0.5863
-0.8861,  0.0170, -0.5761,  0.5972, -0.4053,  0.7301
 0.6877, -0.2380,  0.4997,  0.0223,  0.0819,  0.1404
 0.9189,  0.6079, -0.9354,  0.4188, -0.0700,  0.1907
-0.1428, -0.7820,  0.2676,  0.6059,  0.3936,  0.2790
 0.5324, -0.3151,  0.6917, -0.1425,  0.6480,  0.1071
-0.8432, -0.9633, -0.8666, -0.0828, -0.7733,  0.7784
-0.9444,  0.5097, -0.2103,  0.4939, -0.0952,  0.6787
-0.0520,  0.6063, -0.1952,  0.8094, -0.9259,  0.4836
 0.5477, -0.7487,  0.2370, -0.9793,  0.0773,  0.1241
 0.2450,  0.8116,  0.9799,  0.4222,  0.4636,  0.2355
 0.8186, -0.1983, -0.5003, -0.6531, -0.7611,  0.1511
-0.4714,  0.6382, -0.3788,  0.9648, -0.4667,  0.5950
 0.0673, -0.3711,  0.8215, -0.2669, -0.1328,  0.2677
-0.9381,  0.4338,  0.7820, -0.9454,  0.0441,  0.5518
-0.3480,  0.7190,  0.1170,  0.3805, -0.0943,  0.4724
-0.9813,  0.1535, -0.3771,  0.0345,  0.8328,  0.5438
-0.1471, -0.5052, -0.2574,  0.8637,  0.8737,  0.3042
-0.5454, -0.3712, -0.6505,  0.2142, -0.1728,  0.5783
 0.6327, -0.6297,  0.4038, -0.5193,  0.1484,  0.1153
-0.5424,  0.3282, -0.0055,  0.0380, -0.6506,  0.6613
 0.1414,  0.9935,  0.6337,  0.1887,  0.9520,  0.2540
-0.9351, -0.8128, -0.8693, -0.0965, -0.2491,  0.7353
 0.9507, -0.6640,  0.9456,  0.5349,  0.6485,  0.1059
-0.0462, -0.9737, -0.2940, -0.0159,  0.4602,  0.2606
-0.0627, -0.0852, -0.7247, -0.9782,  0.5166,  0.2977
 0.0478,  0.5098, -0.0723, -0.7504, -0.3750,  0.3335
 0.0090,  0.3477,  0.5403, -0.7393, -0.9542,  0.4415
-0.9748,  0.3449,  0.3736, -0.1015,  0.8296,  0.4358
 0.2887, -0.9895, -0.0311,  0.7186,  0.6608,  0.2057
 0.1570, -0.4518,  0.1211,  0.3435, -0.2951,  0.3244
 0.7117, -0.6099,  0.4946, -0.4208,  0.5476,  0.1096
-0.2929, -0.5726,  0.5346, -0.3827,  0.4665,  0.2465
 0.4889, -0.5572, -0.5718, -0.6021, -0.7150,  0.2163
-0.7782,  0.3491,  0.5996, -0.8389, -0.5366,  0.6516
-0.5847,  0.8347,  0.4226,  0.1078, -0.3910,  0.6134
 0.8469,  0.4121, -0.0439, -0.7476,  0.9521,  0.1571
-0.6803, -0.5948, -0.1376, -0.1916, -0.7065,  0.7156
 0.2878,  0.5086, -0.5785,  0.2019,  0.4979,  0.2980
 0.2764,  0.1943, -0.4090,  0.4632,  0.8906,  0.2960
-0.8877,  0.6705, -0.6155, -0.2098, -0.3998,  0.7107
-0.8398,  0.8093, -0.2597,  0.0614, -0.0118,  0.6502
-0.8476,  0.0158, -0.4769, -0.2859, -0.7839,  0.7715
 0.5751, -0.7868,  0.9714, -0.6457,  0.1448,  0.1175
 0.4802, -0.7001,  0.1022, -0.5668,  0.5184,  0.1090
 0.4458, -0.6469,  0.7239, -0.9604,  0.7205,  0.0779
 0.5175,  0.4339,  0.9747, -0.4438, -0.9924,  0.2879
 0.8678,  0.7158,  0.4577,  0.0334,  0.4139,  0.1678
 0.5406,  0.5012,  0.2264, -0.1963,  0.3946,  0.2088
-0.9938,  0.5498,  0.7928, -0.5214, -0.7585,  0.7687
 0.7661,  0.0863, -0.4266, -0.7233, -0.4197,  0.1466
 0.2277, -0.3517, -0.0853, -0.1118,  0.6563,  0.1767
 0.3499, -0.5570, -0.0655, -0.3705,  0.2537,  0.1632
 0.7547, -0.1046,  0.5689, -0.0861,  0.3125,  0.1257
 0.8186,  0.2110,  0.5335,  0.0094, -0.0039,  0.1391
 0.6858, -0.8644,  0.1465,  0.8855,  0.0357,  0.1845
-0.4967,  0.4015,  0.0805,  0.8977,  0.2487,  0.4663
 0.6760, -0.9841,  0.9787, -0.8446, -0.3557,  0.1509
-0.1203, -0.4885,  0.6054, -0.0443, -0.7313,  0.4854
 0.8557,  0.7919, -0.0169,  0.7134, -0.1628,  0.2002
 0.0115, -0.6209,  0.9300, -0.4116, -0.7931,  0.4052
-0.7114, -0.9718,  0.4319,  0.1290,  0.5892,  0.3661
 0.3915,  0.5557, -0.1870,  0.2955, -0.6404,  0.2954
-0.3564, -0.6548, -0.1827, -0.5172, -0.1862,  0.4622
 0.2392, -0.4959,  0.5857, -0.1341, -0.2850,  0.2470
-0.3394,  0.3947, -0.4627,  0.6166, -0.4094,  0.5325
 0.7107,  0.7768, -0.6312,  0.1707,  0.7964,  0.2757
-0.1078,  0.8437, -0.4420,  0.2177,  0.3649,  0.4028
-0.3139,  0.5595, -0.6505, -0.3161, -0.7108,  0.5546
 0.4335,  0.3986,  0.3770, -0.4932,  0.3847,  0.1810
-0.2562, -0.2894, -0.8847,  0.2633,  0.4146,  0.4036
 0.2272,  0.2966, -0.6601, -0.7011,  0.0284,  0.2778
-0.0743, -0.1421, -0.0054, -0.6770, -0.3151,  0.3597
-0.4762,  0.6891,  0.6007, -0.1467,  0.2140,  0.4266
-0.4061,  0.7193,  0.3432,  0.2669, -0.7505,  0.6147
-0.0588,  0.9731,  0.8966,  0.2902, -0.6966,  0.4955
-0.0627, -0.1439,  0.1985,  0.6999,  0.5022,  0.3077
 0.1587,  0.8494, -0.8705,  0.9827, -0.8940,  0.4263
-0.7850,  0.2473, -0.9040, -0.4308, -0.8779,  0.7199
 0.4070,  0.3369, -0.2428, -0.6236,  0.4940,  0.2215
-0.0242,  0.0513, -0.9430,  0.2885, -0.2987,  0.3947
-0.5416, -0.1322, -0.2351, -0.0604,  0.9590,  0.3683
 0.1055,  0.7783, -0.2901, -0.5090,  0.8220,  0.2984
-0.9129,  0.9015,  0.1128, -0.2473,  0.9901,  0.4776
-0.9378,  0.1424, -0.6391,  0.2619,  0.9618,  0.5368
 0.7498, -0.0963,  0.4169,  0.5549, -0.0103,  0.1614
-0.2612, -0.7156,  0.4538, -0.0460, -0.1022,  0.3717
 0.7720,  0.0552, -0.1818, -0.4622, -0.8560,  0.1685
-0.4177,  0.0070,  0.9319, -0.7812,  0.3461,  0.3052
-0.0001,  0.5542, -0.7128, -0.8336, -0.2016,  0.3803
 0.5356, -0.4194, -0.5662, -0.9666, -0.2027,  0.1776
-0.2378,  0.3187, -0.8582, -0.6948, -0.9668,  0.5474
-0.1947, -0.3579,  0.1158,  0.9869,  0.6690,  0.2992
 0.3992,  0.8365, -0.9205, -0.8593, -0.0520,  0.3154
-0.0209,  0.0793,  0.7905, -0.1067,  0.7541,  0.1864
-0.4928, -0.4524, -0.3433,  0.0951, -0.5597,  0.6261
-0.8118,  0.7404, -0.5263, -0.2280,  0.1431,  0.6349
 0.0516, -0.8480,  0.7483,  0.9023,  0.6250,  0.1959
-0.3212,  0.1093,  0.9488, -0.3766,  0.3376,  0.2735
-0.3481,  0.5490, -0.3484,  0.7797,  0.5034,  0.4379
-0.5785, -0.9170, -0.3563, -0.9258,  0.3877,  0.4121
 0.3407, -0.1391,  0.5356,  0.0720, -0.9203,  0.3458
-0.3287, -0.8954,  0.2102,  0.0241,  0.2349,  0.3247
-0.1353,  0.6954, -0.0919, -0.9692,  0.7461,  0.3338
 0.9036, -0.8982, -0.5299, -0.8733, -0.1567,  0.1187
 0.7277, -0.8368, -0.0538, -0.7489,  0.5458,  0.0830
 0.9049,  0.8878,  0.2279,  0.9470, -0.3103,  0.2194
 0.7957, -0.1308, -0.5284,  0.8817,  0.3684,  0.2172
 0.4647, -0.4931,  0.2010,  0.6292, -0.8918,  0.3371
-0.7390,  0.6849,  0.2367,  0.0626, -0.5034,  0.7039
-0.1567, -0.8711,  0.7940, -0.5932,  0.6525,  0.1710
 0.7635, -0.0265,  0.1969,  0.0545,  0.2496,  0.1445
 0.7675,  0.1354, -0.7698, -0.5460,  0.1920,  0.1728
-0.5211, -0.7372, -0.6763,  0.6897,  0.2044,  0.5217
 0.1913,  0.1980,  0.2314, -0.8816,  0.5006,  0.1998
 0.8964,  0.0694, -0.6149,  0.5059, -0.9854,  0.1825
 0.1767,  0.7104,  0.2093,  0.6452,  0.7590,  0.2832
-0.3580, -0.7541,  0.4426, -0.1193, -0.7465,  0.5657
-0.5996,  0.5766, -0.9758, -0.3933, -0.9572,  0.6800
 0.9950,  0.1641, -0.4132,  0.8579,  0.0142,  0.2003
-0.4717, -0.3894, -0.2567, -0.5111,  0.1691,  0.4266
 0.3917, -0.8561,  0.9422,  0.5061,  0.6123,  0.1212
-0.0366, -0.1087,  0.3449, -0.1025,  0.4086,  0.2475
 0.3633,  0.3943,  0.2372, -0.6980,  0.5216,  0.1925
-0.5325, -0.6466, -0.2178, -0.3589,  0.6310,  0.3568
 0.2271,  0.5200, -0.1447, -0.8011, -0.7699,  0.3128
 0.6415,  0.1993,  0.3777, -0.0178, -0.8237,  0.2181
-0.5298, -0.0768, -0.6028, -0.9490,  0.4588,  0.4356
 0.6870, -0.1431,  0.7294,  0.3141,  0.1621,  0.1632
-0.5985,  0.0591,  0.7889, -0.3900,  0.7419,  0.2945
 0.3661,  0.7984, -0.8486,  0.7572, -0.6183,  0.3449
 0.6995,  0.3342, -0.3113, -0.6972,  0.2707,  0.1712
 0.2565,  0.9126,  0.1798, -0.6043, -0.1413,  0.2893
-0.3265,  0.9839, -0.2395,  0.9854,  0.0376,  0.4770
 0.2690, -0.1722,  0.9818,  0.8599, -0.7015,  0.3954
-0.2102, -0.0768,  0.1219,  0.5607, -0.0256,  0.3949
 0.8216, -0.9555,  0.6422, -0.6231,  0.3715,  0.0801
-0.2896,  0.9484, -0.7545, -0.6249,  0.7789,  0.4370
-0.9985, -0.5448, -0.7092, -0.5931,  0.7926,  0.5402

Test data:

# synthetic_test_40.txt
#
 0.7462,  0.4006, -0.0590,  0.6543, -0.0083,  0.1935
 0.8495, -0.2260, -0.0142, -0.4911,  0.7699,  0.1078
-0.2335, -0.4049,  0.4352, -0.6183, -0.7636,  0.5088
 0.1810, -0.5142,  0.2465,  0.2767, -0.3449,  0.3136
-0.8650,  0.7611, -0.0801,  0.5277, -0.4922,  0.7140
-0.2358, -0.7466, -0.5115, -0.8413, -0.3943,  0.4533
 0.4834,  0.2300,  0.3448, -0.9832,  0.3568,  0.1360
-0.6502, -0.6300,  0.6885,  0.9652,  0.8275,  0.3046
-0.3053,  0.5604,  0.0929,  0.6329, -0.0325,  0.4756
-0.7995,  0.0740, -0.2680,  0.2086,  0.9176,  0.4565
-0.2144, -0.2141,  0.5813,  0.2902, -0.2122,  0.4119
-0.7278, -0.0987, -0.3312, -0.5641,  0.8515,  0.4438
 0.3793,  0.1976,  0.4933,  0.0839,  0.4011,  0.1905
-0.8568,  0.9573, -0.5272,  0.3212, -0.8207,  0.7415
-0.5785,  0.0056, -0.7901, -0.2223,  0.0760,  0.5551
 0.0735, -0.2188,  0.3925,  0.3570,  0.3746,  0.2191
 0.1230, -0.2838,  0.2262,  0.8715,  0.1938,  0.2878
 0.4792, -0.9248,  0.5295,  0.0366, -0.9894,  0.3149
-0.4456,  0.0697,  0.5359, -0.8938,  0.0981,  0.3879
 0.8629, -0.8505, -0.4464,  0.8385,  0.5300,  0.1769
 0.1995,  0.6659,  0.7921,  0.9454,  0.9970,  0.2330
-0.0249, -0.3066, -0.2927, -0.4923,  0.8220,  0.2437
 0.4513, -0.9481, -0.0770, -0.4374, -0.9421,  0.2879
-0.3405,  0.5931, -0.3507, -0.3842,  0.8562,  0.3987
 0.9538,  0.0471,  0.9039,  0.7760,  0.0361,  0.1706
-0.0887,  0.2104,  0.9808,  0.5478, -0.3314,  0.4128
-0.8220, -0.6302,  0.0537, -0.1658,  0.6013,  0.4306
-0.4123, -0.2880,  0.9074, -0.0461, -0.4435,  0.5144
 0.0060,  0.2867, -0.7775,  0.5161,  0.7039,  0.3599
-0.7968, -0.5484,  0.9426, -0.4308,  0.8148,  0.2979
 0.7811,  0.8450, -0.6877,  0.7594,  0.2640,  0.2362
-0.6802, -0.1113, -0.8325, -0.6694, -0.6056,  0.6544
 0.3821,  0.1476,  0.7466, -0.5107,  0.2592,  0.1648
 0.7265,  0.9683, -0.9803, -0.4943, -0.5523,  0.2454
-0.9049, -0.9797, -0.0196, -0.9090, -0.4433,  0.6447
-0.4607,  0.1811, -0.2389,  0.4050, -0.0078,  0.5229
 0.2664, -0.2932, -0.4259, -0.7336,  0.8742,  0.1834
-0.4507,  0.1029, -0.6294, -0.1158, -0.6294,  0.6081
 0.8948, -0.0124,  0.9278,  0.2899, -0.0314,  0.1534
-0.1323, -0.8813, -0.0146, -0.0697,  0.6135,  0.2386
Posted in JavaScript, Machine Learning | Leave a comment