If you have a neural network binary classifier, the output prediction is a number between 0 and 1 where a value less than 0.5 means the prediction is class 0 (whatever that might be) and a value greater than 0.5 means the prediction is class 1.
Suppose you are trying to predict if a person is male (class 0) or female (class 1). If the output pseudo-probability is 0.51, the prediction is (just barely) class 1 = female. If the output pseudo-probability is 0.94, the prediction is still just class 1 = female. But in the pp = 0.94 case, you hope that the likelihood of being correct is much greater than the likelihood of being correct when pp = 0.51.
Calibration error is a metric that measures how well pseudo-probability outputs match the actual frequency of being correct. Briefly, you construct 10 bins of output pseudo-probabilities. For each bin, you track how many data items fall in the bin, the average of the pseudo-probabilities in each bin, and the accuracy for each bin. Calibration error is the weighted sum of the absolute value differences between pseudo-probabilities and accuracy.
I was sitting on a two-hour flight from Las Vegas to Seattle and figured I’d make use of my time by implementing a calibration error function for my standard C# neural network binary classifier code.
For my demo, I used synthetic data that looks like:
1, 0.24, 1, 0, 0, 0.2950, 0, 0, 1 0, 0.39, 0, 0, 1, 0.5120, 0, 1, 0 1, 0.63, 0, 1, 0, 0.7580, 1, 0, 0 0, 0.36, 1, 0, 0, 0.4450, 0, 1, 0 . . .
Each line represents a person. The fields are sex (male = 0, female = 1), age (divided by 100), State (Michigan = 100, Nebraska = 010, Oklahoma = 001), income (divided by 100,000), political leaning (conservative = 100, moderate = 010, liberal = 001). The goal is to predict sex from age, State, income, and political leaning.
There are just 200 training items and 40 test items, which is barely enough for a useable neural network.
For my experiment, I decided to implement two different calibration error functions — one that considers only correct predictions, the second that considers all predictions (correct and wrong).
The output of my demo is:
Neural network binary classification using C# Predict sex from age, State, income, political leaning First three X data: 0.2400 1.0000 0.0000 0.0000 0.2950 0.0000 0.0000 1.0000 0.3900 0.0000 0.0000 1.0000 0.5120 0.0000 1.0000 0.0000 0.6300 0.0000 1.0000 0.0000 0.7580 1.0000 0.0000 0.0000 First three target Y: 1.0 0.0 1.0 Creating 8-10-1 tanh() sigmoid() neural network Setting maxEpochs = 400 Setting lrnRate = 0.010 Setting batSize = 10 Starting training epoch: 0 MBCCE = 0.6922 acc = 0.5050 epoch: 80 MBCCE = 0.6732 acc = 0.6000 epoch: 160 MBCCE = 0.6006 acc = 0.7000 epoch: 240 MBCCE = 0.4977 acc = 0.7900 epoch: 320 MBCCE = 0.4058 acc = 0.8250 Done Accuracy on train data = 0.8900 Accuracy on test data = 0.7500 Calibration error training data, correct predictions Calibration bin counts: 43 17 8 6 6 7 6 28 14 43 0.1360 Calibration error test data, correct predictions Calibration bin counts: 10 3 0 2 2 0 0 3 0 10 0.0874 Calibration error training data, all predictions Calibration bin counts: 43 17 10 8 10 12 10 33 14 43 0.3221 Calibration error test data, all predictions Calibration bin counts: 10 3 0 2 3 4 0 4 2 12 0.5388 Predicting sex (0 = M, 1 = F) for 30 Oklahoma $40,000 moderate Predicted gender = 0.1626 End demo
As a very general rule of thumb, calibration error less than 0.20 is good, between 0.20 and 0.40 is so-so, and greater than 0.40 is weak. In the demo, the calibration error values are quite good, except when taking a predictions into account, for the 40-tem test data, the calibration error is 0.5388. The poor calibration is almost certainly because the test dataset is so small and a few incorrect predictions can have a big impact o the calibration error.
An interesting way to kill some time on an Alaska Airlines flight.
I’ve always been fascinated by electro-mechanical arcade games. On summer when I was a teenager, I worked at a miniature golf course and arcade in Anaheim, California, on La Palma Ave (I think — that was a long time ago). One of my jobs was to maintain the various game machines in the arcade. The machines needed daily calibration, but doing that each morning before the arcade opened was a relatively fun part of my job.
Left: “Southland Little Pro” (1964) is a golf game that was at the arcade I worked at. The inner mechanisms were super interesting. It was a fairly reliable machine and it didn’t give me too many problems.
Center: This is “Williams Hollywood Driving Range” (1965).
Right: “Bromley Little Pro” (1991) was an updated Southland Little Pro with added miniature golf like obstacles. Notice all three machines used the same little man. It was common for arcade game manufacturers to sell parts to each other.
Demo code. Replace “lt” (less than), “gt”, “lte”, “gte”, “and” with Boolean operator symbols. (My lame blog editor consistently chokes on symbols).
using System;
using System.IO;
using System.Collections.Generic;
namespace NeuralNetworkBinaryCalibration
{
internal class NeuralNetworkBinaryCalibrationProgram
{
static void Main(string[] args)
{
Console.WriteLine("\nNeural network " +
"binary classification using C# ");
Console.WriteLine("Predict sex from age," +
" State, income, political leaning ");
string trainFile =
"..\\..\\..\\Data\\people_train.txt";
// sex, age, State, income, politics
// 1 0.24 1 0 0 0.2950 0 0 1
// 0 0.39 0 0 1 0.5120 0 1 0
double[][] trainX = Utils.MatLoad(trainFile,
new int[] { 1, 2, 3, 4, 5, 6, 7, 8 }, ',', "#");
double[] trainY =
Utils.MatToVec(Utils.MatLoad(trainFile,
new int[] { 0 }, ',', "#"));
string testFile =
"..\\..\\..\\Data\\people_test.txt";
double[][] testX = Utils.MatLoad(testFile,
new int[] { 1, 2, 3, 4, 5, 6, 7, 8 }, ',', "#");
double[] testY =
Utils.MatToVec(Utils.MatLoad(testFile,
new int[] { 0 }, ',', "#"));
Console.WriteLine("\nFirst three X data: ");
for (int i = 0; i "lt" 3; ++i)
Utils.VecShow(trainX[i], 4, 9, true);
Console.WriteLine("\nFirst three target Y: ");
for (int i = 0; i "lt" 3; ++i)
Console.WriteLine(trainY[i].ToString("F1"));
Console.WriteLine("\nCreating 8-10-1 tanh()" +
" sigmoid() neural network ");
NeuralNetwork nn =
new NeuralNetwork(8, 100, 1, seed: 0);
int maxEpochs = 400;
double lrnRate = 0.01; // if divide grads by batSize
int batSize = 10;
Console.WriteLine("\nSetting maxEpochs = " +
maxEpochs);
Console.WriteLine("Setting lrnRate = " +
lrnRate.ToString("F3"));
Console.WriteLine("Setting batSize = " + batSize);
// ------------------------------------------------------
Console.WriteLine("\nStarting training ");
nn.Train(trainX, trainY, lrnRate, batSize, maxEpochs);
Console.WriteLine("Done ");
double trainAcc = nn.Accuracy(trainX, trainY);
Console.WriteLine("\nAccuracy on train data = " +
trainAcc.ToString("F4"));
double testAcc = nn.Accuracy(testX, testY);
Console.WriteLine("Accuracy on test data = " +
testAcc.ToString("F4"));
//Console.WriteLine("\nConfusion matrix: ");
//int[][] cm = nn.ConfusionMatrix(testX, testY);
//nn.ShowConfusion(cm);
Console.WriteLine("\nCalibration error training data," +
" correct predictions ");
double calErrTrainCorrects =
nn.CalibrationErrorCorrects(trainX, trainY);
Console.WriteLine(calErrTrainCorrects.ToString("F4"));
Console.WriteLine("\nCalibration error test data," +
" correct predictions ");
double calErrTestCorrects =
nn.CalibrationErrorCorrects(testX, testY);
Console.WriteLine(calErrTestCorrects.ToString("F4"));
Console.WriteLine("\nCalibration error training data," +
" all predictions ");
double calErrTrainAll =
nn.CalibrationErrorAll(trainX, trainY);
Console.WriteLine(calErrTrainAll.ToString("F4"));
Console.WriteLine("\nCalibration error test data," +
" all predictions ");
double calErrTestAll =
nn.CalibrationErrorAll(testX, testY);
Console.WriteLine(calErrTestAll.ToString("F4"));
Console.WriteLine("\nPredicting sex (0 = male, " +
"1 = female) for 30 Oklahoma $40,000 moderate ");
double[] X =
new double[] { 0.30, 0, 0, 1, 0.40000, 0, 1, 0 };
double y = nn.ComputeOutput(X);
Console.WriteLine("Predicted gender = " +
y.ToString("F4"));
Console.WriteLine("\nEnd demo ");
Console.ReadLine();
} // Main
} // Program
public class NeuralNetwork
{
private int ni; // number input nodes
private int nh;
private int no;
private double[] iNodes;
private double[][] ihWeights; // input-hidden
private double[] hBiases;
private double[] hNodes;
private double[][] hoWeights; // hidden-output
private double[] oBiases;
private double[] oNodes; // single val as array
// gradients
private double[][] ihGrads;
private double[] hbGrads;
private double[][] hoGrads;
private double[] obGrads;
private Random rnd;
// --------------------------------------------------------
public NeuralNetwork(int numIn, int numHid,
int numOut, int seed)
{
this.ni = numIn;
this.nh = numHid;
this.no = numOut; // 1 for binary classification
this.iNodes = new double[numIn];
this.ihWeights = Utils.MatCreate(numIn, numHid);
this.hBiases = new double[numHid];
this.hNodes = new double[numHid];
this.hoWeights = Utils.MatCreate(numHid, numOut);
this.oBiases = new double[numOut]; // [1]
this.oNodes = new double[numOut]; // [1]
this.ihGrads = Utils.MatCreate(numIn, numHid);
this.hbGrads = new double[numHid];
this.hoGrads = Utils.MatCreate(numHid, numOut);
this.obGrads = new double[numOut];
this.rnd = new Random(seed);
this.InitWeights(); // all weights and biases
} // ctor
// --------------------------------------------------------
private void InitWeights() // helper for ctor
{
// weights and biases to small random values
double lo = -0.10; double hi = +0.10;
int numWts = (this.ni * this.nh) +
(this.nh * this.no) + this.nh + this.no;
double[] initialWeights = new double[numWts];
for (int i = 0; i "lt" initialWeights.Length; ++i)
initialWeights[i] =
(hi - lo) * rnd.NextDouble() + lo;
this.SetWeights(initialWeights);
}
// --------------------------------------------------------
public void SetWeights(double[] wts)
{
// copy serialized weights and biases in wts[]
// to ih weights, ih biases, ho weights, ho biases
int numWts = (this.ni * this.nh) +
(this.nh * this.no) + this.nh + this.no;
if (wts.Length != numWts)
throw new Exception("Bad array in SetWeights");
int k = 0; // points into wts param
for (int i = 0; i "lt" this.ni; ++i)
for (int j = 0; j "lt" this.nh; ++j)
this.ihWeights[i][j] = wts[k++];
for (int i = 0; i "lt" this.nh; ++i)
this.hBiases[i] = wts[k++];
for (int i = 0; i "lt" this.nh; ++i)
for (int j = 0; j "lt" this.no; ++j)
this.hoWeights[i][j] = wts[k++];
for (int i = 0; i "lt" this.no; ++i)
this.oBiases[i] = wts[k++];
}
// --------------------------------------------------------
public double[] GetWeights()
{
int numWts = (this.ni * this.nh) +
(this.nh * this.no) + this.nh + this.no;
double[] result = new double[numWts];
int k = 0;
for (int i = 0; i "lt" ihWeights.Length; ++i)
for (int j = 0; j "lt" this.ihWeights[0].Length; ++j)
result[k++] = this.ihWeights[i][j];
for (int i = 0; i "lt" this.hBiases.Length; ++i)
result[k++] = this.hBiases[i];
for (int i = 0; i "lt" this.hoWeights.Length; ++i)
for (int j = 0; j "lt" this.hoWeights[0].Length; ++j)
result[k++] = this.hoWeights[i][j];
for (int i = 0; i "lt" this.oBiases.Length; ++i)
result[k++] = this.oBiases[i];
return result;
}
// --------------------------------------------------------
public double ComputeOutput(double[] x)
{
// aka Predict()
double[] hSums = new double[this.nh]; // scratch
double[] oSums = new double[this.no]; // out sums
for (int i = 0; i "lt" x.Length; ++i)
this.iNodes[i] = x[i];
// note: no need to copy x-values unless
// you implement a ToString.
// more efficient to simply use the x[] directly
// or copy by reference.
// 1. compute i-h sum of weights * inputs
for (int j = 0; j "lt" this.nh; ++j)
for (int i = 0; i "lt" this.ni; ++i)
hSums[j] += this.iNodes[i] *
this.ihWeights[i][j]; // note +=
// 2. add biases to hidden sums
for (int i = 0; i "lt" this.nh; ++i)
hSums[i] += this.hBiases[i];
// 3. apply hidden activation
for (int i = 0; i "lt" this.nh; ++i)
this.hNodes[i] = HyperTan(hSums[i]);
// 4. compute h-o sum of wts * hOutputs
for (int j = 0; j "lt" this.no; ++j)
for (int i = 0; i "lt" this.nh; ++i)
oSums[j] += this.hNodes[i] *
this.hoWeights[i][j]; // [1]
// 5. add biases to output sums
for (int i = 0; i "lt" this.no; ++i)
oSums[i] += this.oBiases[i];
// 6. apply output activation
for (int i = 0; i "lt" this.no; ++i)
this.oNodes[i] = LogSigmoid(oSums[i]);
return this.oNodes[0]; // single value
}
// --------------------------------------------------------
private static double HyperTan(double x)
{
if (x "lt" -10.0) return -1.0;
else if (x "gt" 10.0) return 1.0;
else return Math.Tanh(x);
}
// --------------------------------------------------------
private static double LogSigmoid(double x)
{
if (x "lt" -10.0) return 0.0;
else if (x "gt" 10.0) return 1.0;
else return 1.0 / (1.0 + Math.Exp(-x));
}
// --------------------------------------------------------
private void ZeroOutGrads()
{
for (int i = 0; i "lt" this.ni; ++i)
for (int j = 0; j "lt" this.nh; ++j)
this.ihGrads[i][j] = 0.0;
for (int j = 0; j "lt" this.nh; ++j)
this.hbGrads[j] = 0.0;
for (int j = 0; j "lt" this.nh; ++j)
for (int k = 0; k "lt" this.no; ++k)
this.hoGrads[j][k] = 0.0;
for (int k = 0; k "lt" this.no; ++k)
this.obGrads[k] = 0.0;
} // ZeroOutGrads()
// ------------------------------------------------------
private void AccumGrads(double y)
{
double[] oSignals = new double[this.no];
double[] hSignals = new double[this.nh];
// 1. compute output node signals
for (int k = 0; k "lt" this.no; ++k)
{
double derivative = 1.0; // for LogSig with BCEE
//double derivative = (1 - oNodes[k]) * oNodes[k];
// MSE
oSignals[k] = derivative * (this.oNodes[k] - y);
}
// 2. accum hidden-to-output gradients
for (int j = 0; j "lt" this.nh; ++j)
for (int k = 0; k "lt" this.no; ++k)
hoGrads[j][k] +=
oSignals[k] * this.hNodes[j];
// 3. accum output node bias gradients
for (int k = 0; k "lt" this.no; ++k)
obGrads[k] +=
oSignals[k] * 1.0; // 1.0 dummy
// 4. compute hidden node signals
for (int j = 0; j "lt" this.nh; ++j)
{
double sum = 0.0;
for (int k = 0; k "lt" this.no; ++k)
sum += oSignals[k] * this.hoWeights[j][k];
double derivative =
(1 - this.hNodes[j]) *
(1 + this.hNodes[j]); // assumes tanh
hSignals[j] = derivative * sum;
}
// 5. accum input-to-hidden gradients
for (int i = 0; i "lt" this.ni; ++i)
for (int j = 0; j "lt" this.nh; ++j)
this.ihGrads[i][j] +=
hSignals[j] * this.iNodes[i];
// 6. accum hidden node bias gradients
for (int j = 0; j "lt" this.nh; ++j)
this.hbGrads[j] +=
hSignals[j] * 1.0; // 1.0 dummy
} // AccumGrads
// ------------------------------------------------------
private void UpdateWeights(double lrnRate)
{
// assumes all gradients computed
// 1. update input-to-hidden weights
for (int i = 0; i "lt" this.ni; ++i)
{
for (int j = 0; j "lt" this.nh; ++j)
{
double delta = -1.0 * lrnRate *
this.ihGrads[i][j];
this.ihWeights[i][j] += delta;
}
}
// 2. update hidden node biases
for (int j = 0; j "lt" this.nh; ++j)
{
double delta = -1.0 * lrnRate *
this.hbGrads[j];
this.hBiases[j] += delta;
}
// 3. update hidden-to-output weights
for (int j = 0; j "lt" this.nh; ++j)
{
for (int k = 0; k "lt" this.no; ++k)
{
double delta = -1.0 * lrnRate *
this.hoGrads[j][k];
this.hoWeights[j][k] += delta;
}
}
// 4. update output node biases
for (int k = 0; k "lt" this.no; ++k)
{
double delta = -1.0 * lrnRate *
this.obGrads[k];
this.oBiases[k] += delta;
}
} // UpdateWeights()
public void Train(double[][] trainX, double[] trainY,
double lrnRate, int batSize, int maxEpochs)
{
// modular, several helpers version
int n = trainX.Length; // 200
int batchesPerEpoch = n / batSize; // 20
int freq = maxEpochs / 5; // to show progress
int[] indices = new int[n];
for (int i = 0; i "lt" n; ++i)
indices[i] = i;
for (int epoch = 0; epoch "lt" maxEpochs; ++epoch)
{
Shuffle(indices);
int ptr = 0; // points into indices
for (int batIdx = 0; batIdx "lt" batchesPerEpoch;
++batIdx) // 0, 1, . . 19
{
for (int i = 0; i "lt" batSize; ++i) // 0 . . 9
{
int ii = indices[ptr++]; // compute output
double[] x = trainX[ii];
double y = trainY[ii];
this.ComputeOutput(x); // into this.oNoodes
this.AccumGrads(y);
}
this.UpdateWeights(lrnRate);
this.ZeroOutGrads(); // prep for next batch
} // batches
if (epoch % freq == 0) // progress every few epochs
{
double mbcee = this.MeanBCE(trainX, trainY);
double acc = this.Accuracy(trainX, trainY);
string s1 = "epoch: " + epoch.ToString().PadLeft(4);
string s2 = " MBCCE = " + mbcee.ToString("F4");
string s3 = " acc = " + acc.ToString("F4");
Console.WriteLine(s1 + s2 + s3);
}
} // epoch
} // Train()
// --------------------------------------------------------
private void Shuffle(int[] sequence)
{
for (int i = 0; i "lt" sequence.Length; ++i)
{
int r = this.rnd.Next(i, sequence.Length);
int tmp = sequence[r];
sequence[r] = sequence[i];
sequence[i] = tmp;
//sequence[i] = i; // for testing
}
} // Shuffle
// --------------------------------------------------------
public double Error(double[][] trainX, double[] trainY)
{
// MSE -- useful for progress
int n = trainX.Length;
double sumSquaredError = 0.0;
for (int i = 0; i "lt" n; ++i)
{
double predY = this.ComputeOutput(trainX[i]);
double actualY = trainY[i];
sumSquaredError += (predY - actualY) *
(predY - actualY);
}
return sumSquaredError / n; // consider RMSE
} // Error
// ------------------------------------------------------
public double MeanBCE(double[][] trainX, double[] trainY)
{
// mean binary cross entropy error
// for arbitary target yi and predicted pi:
// bcee = -1 * [ (yi * log(pi)) + ((1-yi) * log(1-pi)) ]
//
// if yi target == 0 or 1 only:
// when yi == 1, bcee = -1 * [log(pi)]
// when yi == 0, bcee = -1 * [log(1-pi)]
int n = trainX.Length;
double err = 0.0;
for (int i = 0; i "lt" n; ++i)
{
double predY = this.ComputeOutput(trainX[i]);
double actualY = trainY[i]; // 0.0 or 1.0
if ((int)actualY == 1) // target == 1
err += -Math.Log(predY);
else // target == 0
err += -Math.Log(1.0 - predY);
}
return err / n;
} // meanBCE
// --------------------------------------------------------
public double Accuracy(double[][] dataX, double[] dataY)
{
int n = dataX.Length;
int nCorrect = 0;
int nWrong = 0;
for (int i = 0; i "lt" n; ++i)
{
double predY = this.ComputeOutput(dataX[i]);
double actualY = dataY[i]; // 0.0 or 1.0
if ((int)actualY == 0 "and" predY "lt" 0.5)
++nCorrect;
else if ((int)actualY == 1 "and" predY "gte" 0.5)
++nCorrect;
else
++nWrong;
}
return (nCorrect * 1.0) / (nCorrect + nWrong);
}
// ------------------------------------------------------
public double CalibrationErrorCorrects(double[][] dataX,
double[] dataY)
{
// just for correct predictions
int[] counts = new int[10];
double[] sums = new double[10];
int[] numCorrects = new int[10];
int[] numWrongs = new int[10];
double[] accuracies = new double[10];
double[] avgPseudoProbs = new double[10];
double[] absDiffs = new double[10];
for (int i = 0; i "lt" dataX.Length; ++i)
{
double[] x = dataX[i];
int target = (int)dataY[i]; // 0 or 1
double predY = this.ComputeOutput(x); // like 0.67
bool correct = false;
if (target == 1 "and" predY "gte" 0.5)
correct = true;
else if (target == 0 "and" predY "lt" 0.5)
correct = true;
if (correct == true)
{
int bin = -1;
if (predY "gte" 0.0 "and" predY "lt" 0.1) bin = 0;
else if (predY "gte" 0.1 "and" predY
"lt" 0.2) bin = 1;
else if (predY "gte" 0.2 "and" predY
"lt" 0.3) bin = 2;
else if (predY "gte" 0.3 "and" predY
"lt" 0.4) bin = 3;
else if (predY "gte" 0.4 "and" predY
"lt" 0.5) bin = 4;
else if (predY "gte" 0.5 "and" predY
"lt" 0.6) bin = 5;
else if (predY "gte" 0.6 "and" predY
"lt" 0.7) bin = 6;
else if (predY "gte" 0.7 "and" predY
"lt" 0.8) bin = 7;
else if (predY "gte" 0.8 "and" predY
"lt" 0.9) bin = 8;
else if (predY "gte" 0.9 "and" predY
"lte" 1.0) bin = 9;
counts[bin] += 1;
sums[bin] += predY;
numCorrects[bin] += 1;
}
} // i
for (int bin = 0; bin "lt" 10; ++bin)
if (counts[bin] == 0) accuracies[bin] = 0.0;
else accuracies[bin] = numCorrects[bin] / counts[bin];
for (int bin = 0; bin "lt" 10; ++bin)
if (counts[bin] == 0) avgPseudoProbs[bin] = 0.0;
else avgPseudoProbs[bin] = sums[bin] / counts[bin];
for (int bin = 0; bin "lt" 10; ++bin)
{
if (bin "lte" 4) // class 0
{
absDiffs[bin] =
Math.Abs((1.0 - avgPseudoProbs[bin]) -
accuracies[bin]);
}
else if (bin "gte" 5)
{
absDiffs[bin] = Math.Abs(avgPseudoProbs[bin] -
accuracies[bin]);
}
} // bin
double calibErr = 0.0;
for (int bin = 0; bin "lt" 10; ++bin)
calibErr += counts[bin] * absDiffs[bin]; // weights
calibErr /= dataX.Length;
Console.WriteLine("Calibration bin counts: ");
for (int bin = 0; bin "lt" 10; ++bin)
Console.Write(counts[bin].ToString().PadLeft(4));
Console.WriteLine("");
return calibErr;
} // CalibrationErrorCorrects()
// ------------------------------------------------------
public double CalibrationErrorAll(double[][] dataX,
double[] dataY)
{
// for all (correct and incorrect) predictions
int[] counts = new int[10];
double[] sums = new double[10];
int[] numCorrects = new int[10];
int[] numWrongs = new int[10]; // not used
double[] accuracies = new double[10];
double[] avgPseudoProbs = new double[10];
double[] absDiffs = new double[10];
for (int i = 0; i "lt" dataX.Length; ++i)
{
double[] x = dataX[i];
int target = (int)dataY[i]; // 0 or 1
double predY = this.ComputeOutput(x); // like 0.67
bool correct = false;
if (target == 1 "and" predY "gte" 0.5)
correct = true;
else if (target == 0 "and" predY "lt" 0.5)
correct = true;
int bin = -1;
if (predY "gte" 0.0 "and" predY "lt" 0.1) bin = 0;
else if (predY "gte" 0.1 "and" predY
"lt" 0.2) bin = 1;
else if (predY "gte" 0.2 "and" predY
"lt" 0.3) bin = 2;
else if (predY "gte" 0.3 "and" predY
"lt" 0.4) bin = 3;
else if (predY "gte" 0.4 "and" predY
"lt" 0.5) bin = 4;
else if (predY "gte" 0.5 "and" predY
"lt" 0.6) bin = 5;
else if (predY "gte" 0.6 "and" predY
"lt" 0.7) bin = 6;
else if (predY "gte" 0.7 "and" predY
"lt" 0.8) bin = 7;
else if (predY "gte" 0.8 "and" predY
"lt" 0.9) bin = 8;
else if (predY "gte" 0.9 "and" predY
"lte" 1.0) bin = 9;
counts[bin] += 1; // just for fun style
sums[bin] += predY;
if (correct == true)
numCorrects[bin] += 1;
} // i
for (int bin = 0; bin "lt" 10; ++bin)
if (counts[bin] == 0) accuracies[bin] = 0.0;
else accuracies[bin] = numCorrects[bin] / counts[bin];
for (int bin = 0; bin "lt" 10; ++bin)
if (counts[bin] == 0) avgPseudoProbs[bin] = 0.0;
else avgPseudoProbs[bin] = sums[bin] / counts[bin];
for (int bin = 0; bin "lt" 10; ++bin)
{
if (bin "lte" 4) // class 0
{
absDiffs[bin] =
Math.Abs((1.0 - avgPseudoProbs[bin]) -
accuracies[bin]);
}
else if (bin "gte" 5)
{
absDiffs[bin] = Math.Abs(avgPseudoProbs[bin] -
accuracies[bin]);
}
} // bin
double calibErr = 0.0;
for (int bin = 0; bin "lt" 10; ++bin)
calibErr += counts[bin] * absDiffs[bin]; // weighted
calibErr /= dataX.Length;
Console.WriteLine("Calibration bin counts: ");
for (int bin = 0; bin "lt" 10; ++bin)
Console.Write(counts[bin].ToString().PadLeft(4));
Console.WriteLine("");
return calibErr;
} // CalibrationErrorAll()
// ------------------------------------------------------
} // NeuralNetwork class
public class Utils
{
public static double[][] VecToMat(double[] vec,
int rows, int cols)
{
// vector to row vec/matrix
double[][] result = MatCreate(rows, cols);
int k = 0;
for (int i = 0; i "lt" rows; ++i)
for (int j = 0; j "lt" cols; ++j)
result[i][j] = vec[k++];
return result;
}
// ------------------------------------------------------
public static double[][] MatCreate(int rows,
int cols)
{
double[][] result = new double[rows][];
for (int i = 0; i "lt" rows; ++i)
result[i] = new double[cols];
return result;
}
// ------------------------------------------------------
public static double[][] MatLoad(string fn, int[] usecols,
char sep, string comment)
{
List"lt"double[]"gt" result = new List"lt"double[]"gt"();
string line = "";
FileStream ifs = new FileStream(fn, FileMode.Open);
StreamReader sr = new StreamReader(ifs);
while ((line = sr.ReadLine()) != null)
{
if (line.StartsWith(comment) == true)
continue;
string[] tokens = line.Split(sep);
List"lt"double"gt" lst = new List"lt"double"gt"();
for (int j = 0; j "lt" usecols.Length; ++j)
lst.Add(double.Parse(tokens[usecols[j]]));
double[] row = lst.ToArray();
result.Add(row);
}
sr.Close(); ifs.Close();
return result.ToArray();
}
// ------------------------------------------------------
public static double[] MatToVec(double[][] m)
{
int rows = m.Length;
int cols = m[0].Length;
double[] result = new double[rows * cols];
int k = 0;
for (int i = 0; i "lt" rows; ++i)
for (int j = 0; j "lt" cols; ++j)
result[k++] = m[i][j];
return result;
}
// ------------------------------------------------------
public static double[][] MatToOneHot(double[][] m,
int n)
{
// convert ordinal (0,1,2 . .) to one-hot
int rows = m.Length;
int cols = m[0].Length; // assumed 1
double[][] result = MatCreate(rows, n);
for (int i = 0; i "lt" rows; ++i)
{
int k = (int)m[i][0]; // 0,1,2 . .
result[i] = new double[n]; // [0.0 0.0 0.0]
result[i][k] = 1.0; // [ 0.0 1.0 0.0]
}
return result;
}
// ------------------------------------------------------
public static void MatShow(double[][] m,
int dec, int wid)
{
for (int i = 0; i "lt" m.Length; ++i)
{
for (int j = 0; j "lt" m[0].Length; ++j)
{
double v = m[i][j];
if (Math.Abs(v) "lt" 1.0e-8) v = 0.0; // hack
Console.Write(v.ToString("F" +
dec).PadLeft(wid));
}
Console.WriteLine("");
}
}
// ------------------------------------------------------
public static void VecShow(int[] vec, int wid)
{
for (int i = 0; i "lt" vec.Length; ++i)
Console.Write(vec[i].ToString().PadLeft(wid));
Console.WriteLine("");
}
// ------------------------------------------------------
public static void VecShow(double[] vec,
int dec, int wid, bool newLine)
{
for (int i = 0; i "lt" vec.Length; ++i)
{
double x = vec[i];
if (Math.Abs(x) "lt" 1.0e-8) x = 0.0;
Console.Write(x.ToString("F" +
dec).PadLeft(wid));
}
if (newLine == true)
Console.WriteLine("");
}
} // Utils class
} // ns
Training data:
# people_train.txt # sex (0 = male, 1 = female) - dependent variable # age, state (michigan, nebraska, oklahoma), income, # politics type (conservative, moderate, liberal) # 1, 0.24, 1, 0, 0, 0.2950, 0, 0, 1 0, 0.39, 0, 0, 1, 0.5120, 0, 1, 0 1, 0.63, 0, 1, 0, 0.7580, 1, 0, 0 0, 0.36, 1, 0, 0, 0.4450, 0, 1, 0 1, 0.27, 0, 1, 0, 0.2860, 0, 0, 1 1, 0.50, 0, 1, 0, 0.5650, 0, 1, 0 1, 0.50, 0, 0, 1, 0.5500, 0, 1, 0 0, 0.19, 0, 0, 1, 0.3270, 1, 0, 0 1, 0.22, 0, 1, 0, 0.2770, 0, 1, 0 0, 0.39, 0, 0, 1, 0.4710, 0, 0, 1 1, 0.34, 1, 0, 0, 0.3940, 0, 1, 0 0, 0.22, 1, 0, 0, 0.3350, 1, 0, 0 1, 0.35, 0, 0, 1, 0.3520, 0, 0, 1 0, 0.33, 0, 1, 0, 0.4640, 0, 1, 0 1, 0.45, 0, 1, 0, 0.5410, 0, 1, 0 1, 0.42, 0, 1, 0, 0.5070, 0, 1, 0 0, 0.33, 0, 1, 0, 0.4680, 0, 1, 0 1, 0.25, 0, 0, 1, 0.3000, 0, 1, 0 0, 0.31, 0, 1, 0, 0.4640, 1, 0, 0 1, 0.27, 1, 0, 0, 0.3250, 0, 0, 1 1, 0.48, 1, 0, 0, 0.5400, 0, 1, 0 0, 0.64, 0, 1, 0, 0.7130, 0, 0, 1 1, 0.61, 0, 1, 0, 0.7240, 1, 0, 0 1, 0.54, 0, 0, 1, 0.6100, 1, 0, 0 1, 0.29, 1, 0, 0, 0.3630, 1, 0, 0 1, 0.50, 0, 0, 1, 0.5500, 0, 1, 0 1, 0.55, 0, 0, 1, 0.6250, 1, 0, 0 1, 0.40, 1, 0, 0, 0.5240, 1, 0, 0 1, 0.22, 1, 0, 0, 0.2360, 0, 0, 1 1, 0.68, 0, 1, 0, 0.7840, 1, 0, 0 0, 0.60, 1, 0, 0, 0.7170, 0, 0, 1 0, 0.34, 0, 0, 1, 0.4650, 0, 1, 0 0, 0.25, 0, 0, 1, 0.3710, 1, 0, 0 0, 0.31, 0, 1, 0, 0.4890, 0, 1, 0 1, 0.43, 0, 0, 1, 0.4800, 0, 1, 0 1, 0.58, 0, 1, 0, 0.6540, 0, 0, 1 0, 0.55, 0, 1, 0, 0.6070, 0, 0, 1 0, 0.43, 0, 1, 0, 0.5110, 0, 1, 0 0, 0.43, 0, 0, 1, 0.5320, 0, 1, 0 0, 0.21, 1, 0, 0, 0.3720, 1, 0, 0 1, 0.55, 0, 0, 1, 0.6460, 1, 0, 0 1, 0.64, 0, 1, 0, 0.7480, 1, 0, 0 0, 0.41, 1, 0, 0, 0.5880, 0, 1, 0 1, 0.64, 0, 0, 1, 0.7270, 1, 0, 0 0, 0.56, 0, 0, 1, 0.6660, 0, 0, 1 1, 0.31, 0, 0, 1, 0.3600, 0, 1, 0 0, 0.65, 0, 0, 1, 0.7010, 0, 0, 1 1, 0.55, 0, 0, 1, 0.6430, 1, 0, 0 0, 0.25, 1, 0, 0, 0.4030, 1, 0, 0 1, 0.46, 0, 0, 1, 0.5100, 0, 1, 0 0, 0.36, 1, 0, 0, 0.5350, 1, 0, 0 1, 0.52, 0, 1, 0, 0.5810, 0, 1, 0 1, 0.61, 0, 0, 1, 0.6790, 1, 0, 0 1, 0.57, 0, 0, 1, 0.6570, 1, 0, 0 0, 0.46, 0, 1, 0, 0.5260, 0, 1, 0 0, 0.62, 1, 0, 0, 0.6680, 0, 0, 1 1, 0.55, 0, 0, 1, 0.6270, 1, 0, 0 0, 0.22, 0, 0, 1, 0.2770, 0, 1, 0 0, 0.50, 1, 0, 0, 0.6290, 1, 0, 0 0, 0.32, 0, 1, 0, 0.4180, 0, 1, 0 0, 0.21, 0, 0, 1, 0.3560, 1, 0, 0 1, 0.44, 0, 1, 0, 0.5200, 0, 1, 0 1, 0.46, 0, 1, 0, 0.5170, 0, 1, 0 1, 0.62, 0, 1, 0, 0.6970, 1, 0, 0 1, 0.57, 0, 1, 0, 0.6640, 1, 0, 0 0, 0.67, 0, 0, 1, 0.7580, 0, 0, 1 1, 0.29, 1, 0, 0, 0.3430, 0, 0, 1 1, 0.53, 1, 0, 0, 0.6010, 1, 0, 0 0, 0.44, 1, 0, 0, 0.5480, 0, 1, 0 1, 0.46, 0, 1, 0, 0.5230, 0, 1, 0 0, 0.20, 0, 1, 0, 0.3010, 0, 1, 0 0, 0.38, 1, 0, 0, 0.5350, 0, 1, 0 1, 0.50, 0, 1, 0, 0.5860, 0, 1, 0 1, 0.33, 0, 1, 0, 0.4250, 0, 1, 0 0, 0.33, 0, 1, 0, 0.3930, 0, 1, 0 1, 0.26, 0, 1, 0, 0.4040, 1, 0, 0 1, 0.58, 1, 0, 0, 0.7070, 1, 0, 0 1, 0.43, 0, 0, 1, 0.4800, 0, 1, 0 0, 0.46, 1, 0, 0, 0.6440, 1, 0, 0 1, 0.60, 1, 0, 0, 0.7170, 1, 0, 0 0, 0.42, 1, 0, 0, 0.4890, 0, 1, 0 0, 0.56, 0, 0, 1, 0.5640, 0, 0, 1 0, 0.62, 0, 1, 0, 0.6630, 0, 0, 1 0, 0.50, 1, 0, 0, 0.6480, 0, 1, 0 1, 0.47, 0, 0, 1, 0.5200, 0, 1, 0 0, 0.67, 0, 1, 0, 0.8040, 0, 0, 1 0, 0.40, 0, 0, 1, 0.5040, 0, 1, 0 1, 0.42, 0, 1, 0, 0.4840, 0, 1, 0 1, 0.64, 1, 0, 0, 0.7200, 1, 0, 0 0, 0.47, 1, 0, 0, 0.5870, 0, 0, 1 1, 0.45, 0, 1, 0, 0.5280, 0, 1, 0 0, 0.25, 0, 0, 1, 0.4090, 1, 0, 0 1, 0.38, 1, 0, 0, 0.4840, 1, 0, 0 1, 0.55, 0, 0, 1, 0.6000, 0, 1, 0 0, 0.44, 1, 0, 0, 0.6060, 0, 1, 0 1, 0.33, 1, 0, 0, 0.4100, 0, 1, 0 1, 0.34, 0, 0, 1, 0.3900, 0, 1, 0 1, 0.27, 0, 1, 0, 0.3370, 0, 0, 1 1, 0.32, 0, 1, 0, 0.4070, 0, 1, 0 1, 0.42, 0, 0, 1, 0.4700, 0, 1, 0 0, 0.24, 0, 0, 1, 0.4030, 1, 0, 0 1, 0.42, 0, 1, 0, 0.5030, 0, 1, 0 1, 0.25, 0, 0, 1, 0.2800, 0, 0, 1 1, 0.51, 0, 1, 0, 0.5800, 0, 1, 0 0, 0.55, 0, 1, 0, 0.6350, 0, 0, 1 1, 0.44, 1, 0, 0, 0.4780, 0, 0, 1 0, 0.18, 1, 0, 0, 0.3980, 1, 0, 0 0, 0.67, 0, 1, 0, 0.7160, 0, 0, 1 1, 0.45, 0, 0, 1, 0.5000, 0, 1, 0 1, 0.48, 1, 0, 0, 0.5580, 0, 1, 0 0, 0.25, 0, 1, 0, 0.3900, 0, 1, 0 0, 0.67, 1, 0, 0, 0.7830, 0, 1, 0 1, 0.37, 0, 0, 1, 0.4200, 0, 1, 0 0, 0.32, 1, 0, 0, 0.4270, 0, 1, 0 1, 0.48, 1, 0, 0, 0.5700, 0, 1, 0 0, 0.66, 0, 0, 1, 0.7500, 0, 0, 1 1, 0.61, 1, 0, 0, 0.7000, 1, 0, 0 0, 0.58, 0, 0, 1, 0.6890, 0, 1, 0 1, 0.19, 1, 0, 0, 0.2400, 0, 0, 1 1, 0.38, 0, 0, 1, 0.4300, 0, 1, 0 0, 0.27, 1, 0, 0, 0.3640, 0, 1, 0 1, 0.42, 1, 0, 0, 0.4800, 0, 1, 0 1, 0.60, 1, 0, 0, 0.7130, 1, 0, 0 0, 0.27, 0, 0, 1, 0.3480, 1, 0, 0 1, 0.29, 0, 1, 0, 0.3710, 1, 0, 0 0, 0.43, 1, 0, 0, 0.5670, 0, 1, 0 1, 0.48, 1, 0, 0, 0.5670, 0, 1, 0 1, 0.27, 0, 0, 1, 0.2940, 0, 0, 1 0, 0.44, 1, 0, 0, 0.5520, 1, 0, 0 1, 0.23, 0, 1, 0, 0.2630, 0, 0, 1 0, 0.36, 0, 1, 0, 0.5300, 0, 0, 1 1, 0.64, 0, 0, 1, 0.7250, 1, 0, 0 1, 0.29, 0, 0, 1, 0.3000, 0, 0, 1 0, 0.33, 1, 0, 0, 0.4930, 0, 1, 0 0, 0.66, 0, 1, 0, 0.7500, 0, 0, 1 0, 0.21, 0, 0, 1, 0.3430, 1, 0, 0 1, 0.27, 1, 0, 0, 0.3270, 0, 0, 1 1, 0.29, 1, 0, 0, 0.3180, 0, 0, 1 0, 0.31, 1, 0, 0, 0.4860, 0, 1, 0 1, 0.36, 0, 0, 1, 0.4100, 0, 1, 0 1, 0.49, 0, 1, 0, 0.5570, 0, 1, 0 0, 0.28, 1, 0, 0, 0.3840, 1, 0, 0 0, 0.43, 0, 0, 1, 0.5660, 0, 1, 0 0, 0.46, 0, 1, 0, 0.5880, 0, 1, 0 1, 0.57, 1, 0, 0, 0.6980, 1, 0, 0 0, 0.52, 0, 0, 1, 0.5940, 0, 1, 0 0, 0.31, 0, 0, 1, 0.4350, 0, 1, 0 0, 0.55, 1, 0, 0, 0.6200, 0, 0, 1 1, 0.50, 1, 0, 0, 0.5640, 0, 1, 0 1, 0.48, 0, 1, 0, 0.5590, 0, 1, 0 0, 0.22, 0, 0, 1, 0.3450, 1, 0, 0 1, 0.59, 0, 0, 1, 0.6670, 1, 0, 0 1, 0.34, 1, 0, 0, 0.4280, 0, 0, 1 0, 0.64, 1, 0, 0, 0.7720, 0, 0, 1 1, 0.29, 0, 0, 1, 0.3350, 0, 0, 1 0, 0.34, 0, 1, 0, 0.4320, 0, 1, 0 0, 0.61, 1, 0, 0, 0.7500, 0, 0, 1 1, 0.64, 0, 0, 1, 0.7110, 1, 0, 0 0, 0.29, 1, 0, 0, 0.4130, 1, 0, 0 1, 0.63, 0, 1, 0, 0.7060, 1, 0, 0 0, 0.29, 0, 1, 0, 0.4000, 1, 0, 0 0, 0.51, 1, 0, 0, 0.6270, 0, 1, 0 0, 0.24, 0, 0, 1, 0.3770, 1, 0, 0 1, 0.48, 0, 1, 0, 0.5750, 0, 1, 0 1, 0.18, 1, 0, 0, 0.2740, 1, 0, 0 1, 0.18, 1, 0, 0, 0.2030, 0, 0, 1 1, 0.33, 0, 1, 0, 0.3820, 0, 0, 1 0, 0.20, 0, 0, 1, 0.3480, 1, 0, 0 1, 0.29, 0, 0, 1, 0.3300, 0, 0, 1 0, 0.44, 0, 0, 1, 0.6300, 1, 0, 0 0, 0.65, 0, 0, 1, 0.8180, 1, 0, 0 0, 0.56, 1, 0, 0, 0.6370, 0, 0, 1 0, 0.52, 0, 0, 1, 0.5840, 0, 1, 0 0, 0.29, 0, 1, 0, 0.4860, 1, 0, 0 0, 0.47, 0, 1, 0, 0.5890, 0, 1, 0 1, 0.68, 1, 0, 0, 0.7260, 0, 0, 1 1, 0.31, 0, 0, 1, 0.3600, 0, 1, 0 1, 0.61, 0, 1, 0, 0.6250, 0, 0, 1 1, 0.19, 0, 1, 0, 0.2150, 0, 0, 1 1, 0.38, 0, 0, 1, 0.4300, 0, 1, 0 0, 0.26, 1, 0, 0, 0.4230, 1, 0, 0 1, 0.61, 0, 1, 0, 0.6740, 1, 0, 0 1, 0.40, 1, 0, 0, 0.4650, 0, 1, 0 0, 0.49, 1, 0, 0, 0.6520, 0, 1, 0 1, 0.56, 1, 0, 0, 0.6750, 1, 0, 0 0, 0.48, 0, 1, 0, 0.6600, 0, 1, 0 1, 0.52, 1, 0, 0, 0.5630, 0, 0, 1 0, 0.18, 1, 0, 0, 0.2980, 1, 0, 0 0, 0.56, 0, 0, 1, 0.5930, 0, 0, 1 0, 0.52, 0, 1, 0, 0.6440, 0, 1, 0 0, 0.18, 0, 1, 0, 0.2860, 0, 1, 0 0, 0.58, 1, 0, 0, 0.6620, 0, 0, 1 0, 0.39, 0, 1, 0, 0.5510, 0, 1, 0 0, 0.46, 1, 0, 0, 0.6290, 0, 1, 0 0, 0.40, 0, 1, 0, 0.4620, 0, 1, 0 0, 0.60, 1, 0, 0, 0.7270, 0, 0, 1 1, 0.36, 0, 1, 0, 0.4070, 0, 0, 1 1, 0.44, 1, 0, 0, 0.5230, 0, 1, 0 1, 0.28, 1, 0, 0, 0.3130, 0, 0, 1 1, 0.54, 0, 0, 1, 0.6260, 1, 0, 0
Test data:
# people_test.txt # 0, 0.51, 1, 0, 0, 0.6120, 0, 1, 0 0, 0.32, 0, 1, 0, 0.4610, 0, 1, 0 1, 0.55, 1, 0, 0, 0.6270, 1, 0, 0 1, 0.25, 0, 0, 1, 0.2620, 0, 0, 1 1, 0.33, 0, 0, 1, 0.3730, 0, 0, 1 0, 0.29, 0, 1, 0, 0.4620, 1, 0, 0 1, 0.65, 1, 0, 0, 0.7270, 1, 0, 0 0, 0.43, 0, 1, 0, 0.5140, 0, 1, 0 0, 0.54, 0, 1, 0, 0.6480, 0, 0, 1 1, 0.61, 0, 1, 0, 0.7270, 1, 0, 0 1, 0.52, 0, 1, 0, 0.6360, 1, 0, 0 1, 0.30, 0, 1, 0, 0.3350, 0, 0, 1 1, 0.29, 1, 0, 0, 0.3140, 0, 0, 1 0, 0.47, 0, 0, 1, 0.5940, 0, 1, 0 1, 0.39, 0, 1, 0, 0.4780, 0, 1, 0 1, 0.47, 0, 0, 1, 0.5200, 0, 1, 0 0, 0.49, 1, 0, 0, 0.5860, 0, 1, 0 0, 0.63, 0, 0, 1, 0.6740, 0, 0, 1 0, 0.30, 1, 0, 0, 0.3920, 1, 0, 0 0, 0.61, 0, 0, 1, 0.6960, 0, 0, 1 0, 0.47, 0, 0, 1, 0.5870, 0, 1, 0 1, 0.30, 0, 0, 1, 0.3450, 0, 0, 1 0, 0.51, 0, 0, 1, 0.5800, 0, 1, 0 0, 0.24, 1, 0, 0, 0.3880, 0, 1, 0 0, 0.49, 1, 0, 0, 0.6450, 0, 1, 0 1, 0.66, 0, 0, 1, 0.7450, 1, 0, 0 0, 0.65, 1, 0, 0, 0.7690, 1, 0, 0 0, 0.46, 0, 1, 0, 0.5800, 1, 0, 0 0, 0.45, 0, 0, 1, 0.5180, 0, 1, 0 0, 0.47, 1, 0, 0, 0.6360, 1, 0, 0 0, 0.29, 1, 0, 0, 0.4480, 1, 0, 0 0, 0.57, 0, 0, 1, 0.6930, 0, 0, 1 0, 0.20, 1, 0, 0, 0.2870, 0, 0, 1 0, 0.35, 1, 0, 0, 0.4340, 0, 1, 0 0, 0.61, 0, 0, 1, 0.6700, 0, 0, 1 0, 0.31, 0, 0, 1, 0.3730, 0, 1, 0 1, 0.18, 1, 0, 0, 0.2080, 0, 0, 1 1, 0.26, 0, 0, 1, 0.2920, 0, 0, 1 0, 0.28, 1, 0, 0, 0.3640, 0, 0, 1 0, 0.59, 0, 0, 1, 0.6940, 0, 0, 1
.NET Test Automation Recipes
Software Testing
SciPy Programming Succinctly
Keras Succinctly
R Programming
2026 Visual Studio Live
2025 Summer MLADS Conference
2026 DevIntersection Conference
2025 Machine Learning Week
2025 Ai4 Conference
2026 G2E Conference
2026 iSC West Conference
You must be logged in to post a comment.