A few days ago, I implemented a calibration error function for my standard C# neural network binary classifier code. But then, as usual, one idea leads to another and I wondered about tuning a trained network model to improve the calibration error. Could I do that and would it help? Answer: I used particle swarm optimization to tune a trained binary classifier to improve the calibration, and the calibrated network had better prediction accuracy than the untuned model — in other words, my experiment worked very well and exceeded my expectations.
Bear with me, there are a lot of ideas involved here.
If you have a neural network binary classifier, the output prediction is a number between 0 and 1 where a value less than 0.5 means the prediction is class 0 (whatever that might be) and a value greater than 0.5 means the prediction is class 1 (the “other” class).
Suppose you are trying to predict if a person is male (class 0) or female (class 1). If the output pseudo-probability is 0.51, the prediction is (just barely) class 1 = female. If the output pseudo-probability (pp) is 0.94, the prediction is still just class 1 = female. But in the pp = 0.94 case, you hope that the likelihood of being correct is much greater than the likelihood of being correct when pp = 0.51.
Calibration error is a metric that measures how well pseudo-probability outputs match the actual frequency of being correct. Higher calibration error is worse than lower calibration error. Briefly, you construct 10 bins of output pseudo-probabilities. For each bin, you track how many data items fall in the bin, the average of the pseudo-probabilities in each bin, and the accuracy for each bin. Calibration error is the weighted sum of the absolute value differences between pseudo-probabilities and accuracy.
For my demo, I used synthetic data that looks like:
1, 0.24, 1, 0, 0, 0.2950, 0, 0, 1 0, 0.39, 0, 0, 1, 0.5120, 0, 1, 0 1, 0.63, 0, 1, 0, 0.7580, 1, 0, 0 0, 0.36, 1, 0, 0, 0.4450, 0, 1, 0 . . .
Each line represents a person. The fields are sex (male = 0, female = 1), age (divided by 100), State (Michigan = 100, Nebraska = 010, Oklahoma = 001), income (divided by 100,000), political leaning (conservative = 100, moderate = 010, liberal = 001). The goal is to predict sex from age, State, income, and political leaning. There are just 200 training items and 40 test items, which is barely enough for a useable neural network.
The key calling statements in the demo are:
. . . NeuralNetwork nn = new NeuralNetwork(8, 10, 1, seed: 0); // create classifier nn.Train(trainX, trainY, lrnRate, batSize, maxEpochs); // train double[] modelWeights = nn.GetWeights(); // trained weights double[] wtsBias = nn.Calibrate(50, 5000, modelWeights, trainX, trainY); // improved weights nn.SetWeights(wtsBias); // modify the classifier . . .
The demo creates an 8-10-1 binary classifier and trains it using standard SGD back-propagation. The resulting weights and biases are fetched and used as a starting point for swarm optimization tuning to improve calibration error — as opposed to starting the calibration from scratch. The particle swarm optimization calibration tuning function uses 50 Particles and 5,000 tuning iterations. The resulting tuned weights and biases are then placed back into the neural network
The output of my demo is:
Neural network binary classification using C# Predict sex from age, State, income, political leaning First three X data: 0.2400 1.0000 0.0000 0.0000 0.2950 0.0000 0.0000 1.0000 0.3900 0.0000 0.0000 1.0000 0.5120 0.0000 1.0000 0.0000 0.6300 0.0000 1.0000 0.0000 0.7580 1.0000 0.0000 0.0000 First three target Y: 1.0 0.0 1.0 Creating 8-10-1 tanh() sigmoid() neural network Setting maxEpochs = 400 Setting lrnRate = 0.010 Setting batSize = 10 Starting training epoch: 0 MBCCE = 0.6918 acc = 0.5300 epoch: 80 MBCCE = 0.6839 acc = 0.5400 epoch: 160 MBCCE = 0.6553 acc = 0.6300 epoch: 240 MBCCE = 0.5785 acc = 0.7100 epoch: 320 MBCCE = 0.3240 acc = 0.9000 Done Accuracy on train data = 0.8650 Accuracy on test data = 0.7000 Calibration error training data, all predictions 0.6788 Calibration error test data, all predictions 0.6457 Starting calibration using swarm optimization iter = 0 calibration err = 0.3376 acc = 0.8450 iter = 1000 calibration err = 0.2478 acc = 0.8850 iter = 2000 calibration err = 0.2461 acc = 0.8850 iter = 3000 calibration err = 0.2451 acc = 0.8850 iter = 4000 calibration err = 0.2450 acc = 0.8850 New accuracy on train data = 0.8900 New accuracy on test data = 0.7500 New calibration error training data 0.2401 New calibration error test data 0.5603 End demo
To summarize the demo output: standard training gave a prediction model with reasonably good accuracy (86.50% = 173 out of 200 correct) but very poor calibration (0.6788 error). After tuning the network using particle swarm optimization, accuracy improved significantly to 89.00% and the calibration error improved dramatically to 0.2401 error.
The key component of the calibration tuning system is particle swarm optimization (PSO). Calibration error is not calculus-differentiable so ordinary math techniques such as gradient descent won’t work.
The demo calibration system uses particle swarm optimization (PSO).
For a neural network, a solution is a set of weights and biases. The demo 8-10-1 neural network has 8*10 input-to-hidden weights, 10 hidden biases, 10*1 hidden-to-output weights, 1 hidden bias, for a total of 101 values.
Each solution is encapsulated Particle. In very high-level pseudo-code:
create a collection (swarm) of possible solutions (particles)
loop max_iterations times
for-each particle/solution in swarm
compute a velocity
use velocity to update particle/solution
end-for
keep track of best solution/particle found
end-loop
return best particle/solution found by any particle
Each particle “moves” to a new solution based on a combination of its current “direction”, the best solution/position found by the particle so far, and the best solution/position found by any particle in the swarm so far. The particles tend to quickly spiral towards a common result/position/solution.
My demo program is long and complex because there are many ideas involved — neural networks, calibration error, particle swarm optimization. But all the ideas can be understood with effort. I was able to put together my demo in only a couple of hours, but understanding the all underlying ideas took me many years of (enjoyable) study effort.
One way of thinking about calibration is that it’s a way to match expectation and reality. Here are three examples of children’s clowns with high calibration error. Many years ago, for my son’s first birthday party extravaganza, I rented “Steve-O the Clown”. The less said about that entertainment episode, the better.
Demo program. Replace “lt” (less than), “gt”, “lte”, “gte”, “and” with Boolean operator symbols. My blog editor chokes on symbols.
using System;
using System.IO;
using System.Collections.Generic;
namespace NeuralNetworkBinaryCalibration
{
internal class NeuralNetworkBinaryCalibrationProgram
{
static void Main(string[] args)
{
Console.WriteLine("\nNeural network " +
"binary classification using C# ");
Console.WriteLine("Predict sex from age," +
" State, income, political leaning ");
string trainFile =
"..\\..\\..\\Data\\people_train.txt";
// sex, age, State, income, politics
// 1 0.24 1 0 0 0.2950 0 0 1
// 0 0.39 0 0 1 0.5120 0 1 0
double[][] trainX = Utils.MatLoad(trainFile,
new int[] { 1, 2, 3, 4, 5, 6, 7, 8 }, ',', "#");
double[] trainY =
Utils.MatToVec(Utils.MatLoad(trainFile,
new int[] { 0 }, ',', "#"));
string testFile =
"..\\..\\..\\Data\\people_test.txt";
double[][] testX = Utils.MatLoad(testFile,
new int[] { 1, 2, 3, 4, 5, 6, 7, 8 }, ',', "#");
double[] testY =
Utils.MatToVec(Utils.MatLoad(testFile,
new int[] { 0 }, ',', "#"));
Console.WriteLine("\nFirst three X data: ");
for (int i = 0; i "lt" 3; ++i)
Utils.VecShow(trainX[i], 4, 8, true);
Console.WriteLine("\nFirst three target Y: ");
for (int i = 0; i "lt" 3; ++i)
Console.WriteLine(trainY[i].ToString("F1"));
Console.WriteLine("\nCreating 8-10-1 tanh()" +
" sigmoid() neural network ");
NeuralNetwork nn =
new NeuralNetwork(8, 10, 1, seed: 0);
int maxEpochs = 400;
double lrnRate = 0.01; // if divide grads by batSize
int batSize = 10;
Console.WriteLine("\nSetting maxEpochs = " +
maxEpochs);
Console.WriteLine("Setting lrnRate = " +
lrnRate.ToString("F3"));
Console.WriteLine("Setting batSize = " + batSize);
// ------------------------------------------------------
Console.WriteLine("\nStarting training ");
nn.Train(trainX, trainY, lrnRate, batSize, maxEpochs);
Console.WriteLine("Done ");
double trainAcc = nn.Accuracy(trainX, trainY);
Console.WriteLine("\nAccuracy on train data = " +
trainAcc.ToString("F4"));
double testAcc = nn.Accuracy(testX, testY);
Console.WriteLine("Accuracy on test data = " +
testAcc.ToString("F4"));
Console.WriteLine("\nCalibration error training data," +
" all predictions ");
double calErrTrainAll =
nn.CalibrationErrorAll(trainX, trainY);
Console.WriteLine(calErrTrainAll.ToString("F4"));
Console.WriteLine("\nCalibration error test data," +
" all predictions ");
double calErrTestAll =
nn.CalibrationErrorAll(testX, testY);
Console.WriteLine(calErrTestAll.ToString("F4"));
Console.WriteLine("\nStarting calibration using" +
" swarm optimization ");
double[] modelWeights = nn.GetWeights();
double[] wtsBias = nn.Calibrate(50, 5000,
modelWeights, trainX, trainY);
nn.SetWeights(wtsBias);
trainAcc = nn.Accuracy(trainX, trainY);
Console.WriteLine("\nNew accuracy on train data = " +
trainAcc.ToString("F4"));
testAcc = nn.Accuracy(testX, testY);
Console.WriteLine("New accuracy on test data = " +
testAcc.ToString("F4"));
Console.WriteLine("\nNew calibration error" +
" training data ");
calErrTrainAll =
nn.CalibrationErrorAll(trainX, trainY);
Console.WriteLine(calErrTrainAll.ToString("F4"));
Console.WriteLine("\nNew calibration error" +
" test data ");
calErrTestAll =
nn.CalibrationErrorAll(testX, testY);
Console.WriteLine(calErrTestAll.ToString("F4"));
//Console.WriteLine("\nPredicting sex (0 = male, " +
// "1 = female) for 30 Oklahoma $40,000 moderate ");
//double[] X =
// new double[] { 0.30, 0, 0, 1, 0.40000, 0, 1, 0 };
//double y = nn.ComputeOutput(X);
//Console.WriteLine("Predicted gender = " +
// y.ToString("F4"));
Console.WriteLine("\nEnd demo ");
Console.ReadLine();
} // Main
} // Program
public class NeuralNetwork
{
private int ni; // number input nodes
private int nh;
private int no;
private double[] iNodes;
private double[][] ihWeights; // input-hidden
private double[] hBiases;
private double[] hNodes;
private double[][] hoWeights; // hidden-output
private double[] oBiases;
private double[] oNodes; // single val as array
// gradients
private double[][] ihGrads;
private double[] hbGrads;
private double[][] hoGrads;
private double[] obGrads;
private Random rnd;
// ======================================================
public class Particle
{
public double[] position; // soln = wts + bias
public double error;
public double[] velocity; // to determine next position
public double[] bestPosition; // best seen
public double bestError;
public Particle(int solnLen)
{
this.position = new double[solnLen];
this.velocity = new double[solnLen];
this.bestPosition = new double[solnLen];
}
} // Particle
// ======================================================
// ------------------------------------------------------
public NeuralNetwork(int numIn, int numHid,
int numOut, int seed)
{
this.ni = numIn;
this.nh = numHid;
this.no = numOut; // 1 for binary classification
this.iNodes = new double[numIn];
this.ihWeights = Utils.MatCreate(numIn, numHid);
this.hBiases = new double[numHid];
this.hNodes = new double[numHid];
this.hoWeights = Utils.MatCreate(numHid, numOut);
this.oBiases = new double[numOut]; // [1]
this.oNodes = new double[numOut]; // [1]
this.ihGrads = Utils.MatCreate(numIn, numHid);
this.hbGrads = new double[numHid];
this.hoGrads = Utils.MatCreate(numHid, numOut);
this.obGrads = new double[numOut];
this.rnd = new Random(seed);
this.InitWeights(); // all weights and biases
} // ctor
// ------------------------------------------------------
private void InitWeights() // helper for ctor
{
// weights and biases to small random values
double lo = -0.10; double hi = +0.10;
int numWts = (this.ni * this.nh) +
(this.nh * this.no) + this.nh + this.no;
double[] initialWeights = new double[numWts];
for (int i = 0; i "lt" initialWeights.Length; ++i)
initialWeights[i] =
(hi - lo) * rnd.NextDouble() + lo;
this.SetWeights(initialWeights);
}
// ------------------------------------------------------
public void SetWeights(double[] wts)
{
// copy serialized weights and biases in wts[]
// to ih weights, ih biases, ho weights, ho biases
int numWts = (this.ni * this.nh) +
(this.nh * this.no) + this.nh + this.no;
if (wts.Length != numWts)
throw new Exception("Bad array in SetWeights");
int p = 0; // points into wts param
for (int i = 0; i "lt" this.ni; ++i)
for (int j = 0; j "lt" this.nh; ++j)
this.ihWeights[i][j] = wts[p++];
for (int j = 0; j "lt" this.nh; ++j)
this.hBiases[j] = wts[p++];
for (int j = 0; j "lt" this.nh; ++j)
for (int k = 0; k "lt" this.no; ++k)
this.hoWeights[j][k] = wts[p++];
for (int k = 0; k "lt" this.no; ++k)
this.oBiases[k] = wts[p++];
}
// ------------------------------------------------------
public double[] GetWeights()
{
int numWts = (this.ni * this.nh) +
(this.nh * this.no) + this.nh + this.no;
double[] result = new double[numWts];
int p = 0;
// ih-weights
for (int i = 0; i "lt" this.ni; ++i)
for (int j = 0; j "lt" this.nh; ++j)
result[p++] = this.ihWeights[i][j];
// h-biases
for (int j = 0; j "lt" this.nh; ++j)
result[p++] = this.hBiases[j];
// ho-weights
for (int j = 0; j "lt" this.nh; ++j)
for (int k = 0; k "lt" this.no; ++k)
result[p++] = this.hoWeights[j][k];
// o-biases
for (int k = 0; k "lt" this.no; ++k)
result[p++] = this.oBiases[k];
return result;
}
// ------------------------------------------------------
public double ComputeOutput(double[] x)
{
// aka Predict()
double[] hSums = new double[this.nh]; // scratch
double[] oSums = new double[this.no]; // out sums
for (int i = 0; i "lt" x.Length; ++i)
this.iNodes[i] = x[i];
// note: no need to copy x-values unless
// you implement a ToString.
// more efficient to simply use the x[] directly
// or copy by reference.
// 1. compute i-h sum of weights * inputs
for (int j = 0; j "lt" this.nh; ++j)
for (int i = 0; i "lt" this.ni; ++i)
hSums[j] += this.iNodes[i] *
this.ihWeights[i][j]; // note +=
// 2. add biases to hidden sums
for (int j = 0; j "lt" this.nh; ++j)
hSums[j] += this.hBiases[j];
// 3. apply hidden activation
for (int j = 0; j "lt" this.nh; ++j)
this.hNodes[j] = HyperTan(hSums[j]);
// 4. compute h-o sum of wts * hOutputs
for (int k = 0; k "lt" this.no; ++k)
for (int j = 0; j "lt" this.nh; ++j)
oSums[k] += this.hNodes[j] *
this.hoWeights[j][k]; // [1]
// 5. add biases to output sums
for (int k = 0; k "lt" this.no; ++k)
oSums[k] += this.oBiases[k];
// 6. apply output activation
for (int k = 0; k "lt" this.no; ++k)
this.oNodes[k] = LogSigmoid(oSums[k]);
return this.oNodes[0]; // single value
}
// ------------------------------------------------------
private static double HyperTan(double x)
{
if (x "lt" -10.0) return -1.0;
else if (x "gt" 10.0) return 1.0;
else return Math.Tanh(x);
}
// ------------------------------------------------------
private static double LogSigmoid(double x)
{
if (x "lt" -10.0) return 0.0;
else if (x "gt" 10.0) return 1.0;
else return 1.0 / (1.0 + Math.Exp(-x));
}
// ------------------------------------------------------
private void ZeroOutGrads()
{
for (int i = 0; i "lt" this.ni; ++i)
for (int j = 0; j "lt" this.nh; ++j)
this.ihGrads[i][j] = 0.0;
for (int j = 0; j "lt" this.nh; ++j)
this.hbGrads[j] = 0.0;
for (int j = 0; j "lt" this.nh; ++j)
for (int k = 0; k "lt" this.no; ++k)
this.hoGrads[j][k] = 0.0;
for (int k = 0; k "lt" this.no; ++k)
this.obGrads[k] = 0.0;
} // ZeroOutGrads()
// ------------------------------------------------------
private void AccumGrads(double y)
{
double[] oSignals = new double[this.no];
double[] hSignals = new double[this.nh];
// 1. compute output node signals
for (int k = 0; k "lt" this.no; ++k)
{
double derivative = 1.0; // for LogSig with BCEE
//double derivative = (1 - oNodes[k]) * oNodes[k];
// MSE
oSignals[k] = derivative * (this.oNodes[k] - y);
}
// 2. accum hidden-to-output gradients
for (int j = 0; j "lt" this.nh; ++j)
for (int k = 0; k "lt" this.no; ++k)
hoGrads[j][k] +=
oSignals[k] * this.hNodes[j];
// 3. accum output node bias gradients
for (int k = 0; k "lt" this.no; ++k)
obGrads[k] +=
oSignals[k] * 1.0; // 1.0 dummy
// 4. compute hidden node signals
for (int j = 0; j "lt" this.nh; ++j)
{
double sum = 0.0;
for (int k = 0; k "lt" this.no; ++k)
sum += oSignals[k] * this.hoWeights[j][k];
double derivative =
(1 - this.hNodes[j]) *
(1 + this.hNodes[j]); // assumes tanh
hSignals[j] = derivative * sum;
}
// 5. accum input-to-hidden gradients
for (int i = 0; i "lt" this.ni; ++i)
for (int j = 0; j "lt" this.nh; ++j)
this.ihGrads[i][j] +=
hSignals[j] * this.iNodes[i];
// 6. accum hidden node bias gradients
for (int j = 0; j "lt" this.nh; ++j)
this.hbGrads[j] +=
hSignals[j] * 1.0; // 1.0 dummy
} // AccumGrads
// ------------------------------------------------------
private void UpdateWeights(double lrnRate)
{
// assumes all gradients computed
// 1. update input-to-hidden weights
for (int i = 0; i "lt" this.ni; ++i)
{
for (int j = 0; j "lt" this.nh; ++j)
{
double delta = -1.0 * lrnRate *
this.ihGrads[i][j];
this.ihWeights[i][j] += delta;
}
}
// 2. update hidden node biases
for (int j = 0; j "lt" this.nh; ++j)
{
double delta = -1.0 * lrnRate *
this.hbGrads[j];
this.hBiases[j] += delta;
}
// 3. update hidden-to-output weights
for (int j = 0; j "lt" this.nh; ++j)
{
for (int k = 0; k "lt" this.no; ++k)
{
double delta = -1.0 * lrnRate *
this.hoGrads[j][k];
this.hoWeights[j][k] += delta;
}
}
// 4. update output node biases
for (int k = 0; k "lt" this.no; ++k)
{
double delta = -1.0 * lrnRate *
this.obGrads[k];
this.oBiases[k] += delta;
}
} // UpdateWeights()
// ------------------------------------------------------
public void Train(double[][] trainX, double[] trainY,
double lrnRate, int batSize, int maxEpochs)
{
// modular, several helpers version
int n = trainX.Length; // 200
int batchesPerEpoch = n / batSize; // 20
int freq = maxEpochs / 5; // to show progress
int[] indices = new int[n];
for (int i = 0; i "lt" n; ++i)
indices[i] = i;
for (int epoch = 0; epoch "lt" maxEpochs; ++epoch)
{
Shuffle(indices);
int ptr = 0; // points into indices
for (int batIdx = 0; batIdx "lt" batchesPerEpoch;
++batIdx) // 0, 1, . . 19
{
for (int i = 0; i "lt" batSize; ++i) // 0 . . 9
{
int ii = indices[ptr++]; // compute output
double[] x = trainX[ii];
double y = trainY[ii];
this.ComputeOutput(x); // into this.oNoodes
this.AccumGrads(y);
}
this.UpdateWeights(lrnRate);
this.ZeroOutGrads(); // prep for next batch
} // batches
if (epoch % freq == 0) // progress every few epochs
{
double mbcee = this.MeanBCE(trainX, trainY);
double acc = this.Accuracy(trainX, trainY);
string s1 = "epoch: " + epoch.ToString().PadLeft(4);
string s2 = " MBCCE = " + mbcee.ToString("F4");
string s3 = " acc = " + acc.ToString("F4");
Console.WriteLine(s1 + s2 + s3);
}
} // epoch
} // Train()
// ------------------------------------------------------
private void Shuffle(int[] sequence)
{
for (int i = 0; i "lt" sequence.Length; ++i)
{
int r = this.rnd.Next(i, sequence.Length);
int tmp = sequence[r];
sequence[r] = sequence[i];
sequence[i] = tmp;
//sequence[i] = i; // for testing
}
} // Shuffle
// ------------------------------------------------------
public double Error(double[][] trainX, double[] trainY)
{
// MSE -- useful for progress
int n = trainX.Length;
double sumSquaredError = 0.0;
for (int i = 0; i "lt" n; ++i)
{
double predY = this.ComputeOutput(trainX[i]);
double actualY = trainY[i];
sumSquaredError += (predY - actualY) *
(predY - actualY);
}
return sumSquaredError / n; // consider RMSE
} // Error
// ------------------------------------------------------
public double MeanBCE(double[][] trainX, double[] trainY)
{
// mean binary cross entropy error
// for arbitary target yi and predicted pi:
// bcee = -1 * [ (yi * log(pi)) + ((1-yi) * log(1-pi)) ]
//
// if yi target == 0 or 1 only:
// when yi == 1, bcee = -1 * [log(pi)]
// when yi == 0, bcee = -1 * [log(1-pi)]
int n = trainX.Length;
double err = 0.0;
for (int i = 0; i "lt" n; ++i)
{
double predY = this.ComputeOutput(trainX[i]);
double actualY = trainY[i]; // 0.0 or 1.0
if ((int)actualY == 1) // target == 1
err += -Math.Log(predY);
else // target == 0
err += -Math.Log(1.0 - predY);
}
return err / n;
} // meanBCE
// --------------------------------------------------------
public double Accuracy(double[][] dataX, double[] dataY)
{
int n = dataX.Length;
int nCorrect = 0;
int nWrong = 0;
for (int i = 0; i "lt" n; ++i)
{
double predY = this.ComputeOutput(dataX[i]);
double actualY = dataY[i]; // 0.0 or 1.0
if ((int)actualY == 0 "and" predY "lt" 0.5)
++nCorrect;
else if ((int)actualY == 1 "and" predY "gte" 0.5)
++nCorrect;
else
++nWrong;
}
return (nCorrect * 1.0) / (nCorrect + nWrong);
}
// ------------------------------------------------------
public double CalibrationErrorAll(double[][] dataX,
double[] dataY)
{
// for all (correct and incorrect) predictions
int[] counts = new int[10];
double[] sums = new double[10];
int[] numCorrects = new int[10];
int[] numWrongs = new int[10]; // not used
double[] accuracies = new double[10];
double[] avgPseudoProbs = new double[10];
double[] absDiffs = new double[10];
for (int i = 0; i "lt" dataX.Length; ++i)
{
double[] x = dataX[i];
int target = (int)dataY[i]; // 0 or 1
double predY = this.ComputeOutput(x); // like 0.67
bool correct = false;
if (target == 1 "and" predY "gte" 0.5)
correct = true;
else if (target == 0 "and" predY "lt" 0.5)
correct = true;
int bin = -1;
if (predY "gte" 0.0 "and" predY "lt" 0.1) bin = 0;
else if (predY "gte" 0.1 "and" predY
"lt" 0.2) bin = 1;
else if (predY "gte" 0.2 "and" predY
"lt" 0.3) bin = 2;
else if (predY "gte" 0.3 "and" predY
"lt" 0.4) bin = 3;
else if (predY "gte" 0.4 "and" predY
"lt" 0.5) bin = 4;
else if (predY "gte" 0.5 "and" predY
"lt" 0.6) bin = 5;
else if (predY "gte" 0.6 "and" predY
"lt" 0.7) bin = 6;
else if (predY "gte" 0.7 "and" predY
"lt" 0.8) bin = 7;
else if (predY "gte" 0.8 "and" predY
"lt" 0.9) bin = 8;
else if (predY "gte" 0.9 "and" predY
"lte" 1.0) bin = 9;
counts[bin] += 1; // just for fun style
sums[bin] += predY;
if (correct == true)
numCorrects[bin] += 1;
} // i
for (int bin = 0; bin "lt" 10; ++bin)
if (counts[bin] == 0) accuracies[bin] = 0.0;
else accuracies[bin] = numCorrects[bin] / counts[bin];
for (int bin = 0; bin "lt" 10; ++bin)
if (counts[bin] == 0) avgPseudoProbs[bin] = 0.0;
else avgPseudoProbs[bin] = sums[bin] / counts[bin];
for (int bin = 0; bin "lt" 10; ++bin)
{
if (bin "lte" 4) // class 0
{
absDiffs[bin] =
Math.Abs((1.0 - avgPseudoProbs[bin]) -
accuracies[bin]);
}
else if (bin "gte" 5)
{
absDiffs[bin] = Math.Abs(avgPseudoProbs[bin] -
accuracies[bin]);
}
} // bin
double calibErr = 0.0;
for (int bin = 0; bin "lt" 10; ++bin)
calibErr += counts[bin] * absDiffs[bin]; // weighted
calibErr /= dataX.Length;
return calibErr;
} // CalibrationErrorAll()
// ------------------------------------------------------
public double CalibrationErrorUsing(double[] wtsBias,
double[][] dataX, double[] dataY)
{
// for all (correct and incorrect) predictions
int[] counts = new int[10];
double[] sums = new double[10];
int[] numCorrects = new int[10];
int[] numWrongs = new int[10]; // not used
double[] accuracies = new double[10];
double[] avgPseudoProbs = new double[10];
double[] absDiffs = new double[10];
for (int i = 0; i "lt" dataX.Length; ++i)
{
double[] x = dataX[i];
int target = (int)dataY[i]; // 0 or 1
//double predY = this.ComputeOutput(x); // like 0.67
double predY = this.PredictUsing(wtsBias, x)[0];
bool correct = false;
if (target == 1 "and" predY "gte" 0.5)
correct = true;
else if (target == 0 "and" predY "lt" 0.5)
correct = true;
int bin = -1;
if (predY "gte" 0.0 "and" predY "lt" 0.1) bin = 0;
else if (predY "gte" 0.1 "and" predY
"lt" 0.2) bin = 1;
else if (predY "gte" 0.2 "and" predY
"lt" 0.3) bin = 2;
else if (predY "gte" 0.3 "and" predY
"lt" 0.4) bin = 3;
else if (predY "gte" 0.4 "and" predY
"lt" 0.5) bin = 4;
else if (predY "gte" 0.5 "and" predY
"lt" 0.6) bin = 5;
else if (predY "gte" 0.6 "and" predY
"lt" 0.7) bin = 6;
else if (predY "gte" 0.7 "and" predY
"lt" 0.8) bin = 7;
else if (predY "gte" 0.8 "and" predY
"lt" 0.9) bin = 8;
else if (predY "gte" 0.9 "and" predY
"lte" 1.0) bin = 9;
counts[bin] += 1; // just for fun style
sums[bin] += predY;
if (correct == true)
numCorrects[bin] += 1;
} // i
for (int bin = 0; bin "lt" 10; ++bin)
if (counts[bin] == 0) accuracies[bin] = 0.0;
else accuracies[bin] = numCorrects[bin] / counts[bin];
for (int bin = 0; bin "lt" 10; ++bin)
if (counts[bin] == 0) avgPseudoProbs[bin] = 0.0;
else avgPseudoProbs[bin] = sums[bin] / counts[bin];
for (int bin = 0; bin "lt" 10; ++bin)
{
if (bin "lte" 4) // class 0
{
absDiffs[bin] =
Math.Abs((1.0 - avgPseudoProbs[bin]) -
accuracies[bin]);
}
else if (bin "gte" 5)
{
absDiffs[bin] = Math.Abs(avgPseudoProbs[bin] -
accuracies[bin]);
}
} // bin
double calibErr = 0.0;
for (int bin = 0; bin "lt" 10; ++bin)
calibErr += counts[bin] * absDiffs[bin];
calibErr /= dataX.Length;
return calibErr;
} // CalibrationErrorUsing()
// ------------------------------------------------------
public double[] PredictUsing(double[] wtsBias,
double[] x)
{
// assumes ih-wts, h-biases, ho-wts, o-biases
double[] hNodes = new double[nh];
double[] oNodes = new double[no];
int ptr = 0; // ptr into wtsBias vector
// the indexing is very tricky
for (int i = 0; i "lt" this.ni; ++i) // hNodes
for (int j = 0; j "lt" this.nh; ++j)
hNodes[j] += x[i] * wtsBias[ptr++];
for (int j = 0; j "lt" this.nh; ++j)
{
hNodes[j] += wtsBias[ptr++];
hNodes[j] = HyperTan(hNodes[j]);
}
for (int j = 0; j "lt" this.nh; ++j) // oNodes
for (int k = 0; k "lt" this.no; ++k)
oNodes[k] += hNodes[j] * wtsBias[ptr++];
for (int k = 0; k "lt" this.no; ++k)
{
oNodes[k] += wtsBias[ptr++];
oNodes[k] = LogSigmoid(oNodes[k]);
}
double[] result = new double[no];
for (int k = 0; k "lt" no; ++k)
result[k] = oNodes[k];
return result;
} // PredictUsing()
// ------------------------------------------------------
public double AccuracyUsing(double[] wtsBias,
double[][] dataX, double[] dataY)
{
int n = dataX.Length;
int nCorrect = 0;
int nWrong = 0;
for (int i = 0; i "lt" n; ++i)
{
//double predY = this.ComputeOutput(dataX[i]);
double[] x = dataX[i];
double predY = this.PredictUsing(wtsBias, x)[0];
double actualY = dataY[i]; // 0.0 or 1.0
if ((int)actualY == 0 "and" predY "lt" 0.5)
++nCorrect;
else if ((int)actualY == 1 "and" predY "gte" 0.5)
++nCorrect;
else
++nWrong;
}
return (nCorrect * 1.0) / (nCorrect + nWrong);
} // AccuracyUsing()
// ------------------------------------------------------
public double[] Calibrate(int numParticles, int maxIter,
double[] startWeights, double[][] trainX,
double[] trainY)
{
// 1. create a swarm of particles based on startWeights
int solnLen = startWeights.Length;
double lo = -0.10; double hi = +0.10;
double[] globalBestPosition = new double[solnLen];
double globalBestError = double.MaxValue;
double w = 0.729; // inertia weight
double c1 = 1.49445; // cognitive weight
double c2 = 1.49445; // social weight
Particle[] swarm = new Particle[numParticles];
for (int i = 0; i "lt" numParticles; ++i)
{
swarm[i] = new Particle(solnLen);
for (int j = 0; j "lt" solnLen; ++j)
swarm[i].position[j] = startWeights[j] +
((hi - lo) * this.rnd.NextDouble() + lo);
swarm[i].error =
this.CalibrationErrorUsing(swarm[i].position,
trainX, trainY);
for (int j = 0; j "lt" solnLen; ++j)
swarm[i].velocity[j] = (hi - lo) *
this.rnd.NextDouble() + lo;
for (int j = 0; j "lt" solnLen; ++j)
swarm[i].bestPosition[j] = swarm[i].position[j];
swarm[i].bestError = swarm[i].error;
}
// 2. set global bests
for (int i = 0; i "lt" numParticles; ++i)
{
if (swarm[i].error "lt" globalBestError)
{
globalBestError = swarm[i].error;
for (int j = 0; j "lt" solnLen; ++j)
globalBestPosition[j] = swarm[i].position[j];
}
}
// main processing loop
for (int iter = 0; iter "lt" maxIter; ++iter)
{
for (int i = 0; i "lt" numParticles; ++i)
{
Particle currP = swarm[i]; // ref for clarity
for (int j = 0; j "lt" solnLen; ++j) // 1. velocity
{
double r1 = this.rnd.NextDouble();
double r2 = this.rnd.NextDouble();
currP.velocity[j] = (w * currP.velocity[j]) +
(c1 * r1 * (currP.bestPosition[j] -
currP.position[j])) +
(c2 * r2 * (globalBestPosition[j] -
currP.position[j]));
}
for (int j = 0; j "lt" solnLen; ++j) // 2. position
{
currP.position[j] = currP.position[j] +
currP.velocity[j];
}
// 3. update particle's error
currP.error =
this.CalibrationErrorUsing(currP.position,
trainX, trainY);
// 4. check if particle new best
if (currP.error "lt" currP.bestError)
{
currP.bestError = currP.error;
for (int j = 0; j "lt" solnLen; ++j)
currP.bestPosition[j] = currP.position[j];
}
// 5. check if new global best found
if (currP.error "lt" globalBestError)
{
globalBestError = currP.error;
for (int j = 0; j "lt" solnLen; ++j)
globalBestPosition[j] = currP.position[j];
}
} // each particle
if (iter % (maxIter / 5) == 0) // display progress
{
double bestAcc =
this.AccuracyUsing(globalBestPosition,
trainX, trainY);
string s1 = "iter = " +
iter.ToString().PadLeft(8);
string s2 = " calibration err = " +
globalBestError.ToString("F4").PadLeft(8);
string s3 = " acc = " +
bestAcc.ToString("F4");
Console.WriteLine(s1 + s2 + s3);
}
} // main loop
// return best position/solution found
double[] result = new double[solnLen];
for (int j = 0; j "lt" solnLen; ++j)
result[j] = globalBestPosition[j];
return result;
} // Calibrate
// ******************************************************
} // NeuralNetwork class
public class Utils
{
public static double[][] VecToMat(double[] vec,
int rows, int cols)
{
// vector to row vec/matrix
double[][] result = MatCreate(rows, cols);
int k = 0;
for (int i = 0; i "lt" rows; ++i)
for (int j = 0; j "lt" cols; ++j)
result[i][j] = vec[k++];
return result;
}
// ------------------------------------------------------
public static double[][] MatCreate(int rows,
int cols)
{
double[][] result = new double[rows][];
for (int i = 0; i "lt" rows; ++i)
result[i] = new double[cols];
return result;
}
// ------------------------------------------------------
public static double[][] MatLoad(string fn, int[] usecols,
char sep, string comment)
{
List"lt"double[]"gt" result = new List"lt"double[]"gt"();
string line = "";
FileStream ifs = new FileStream(fn, FileMode.Open);
StreamReader sr = new StreamReader(ifs);
while ((line = sr.ReadLine()) != null)
{
if (line.StartsWith(comment) == true)
continue;
string[] tokens = line.Split(sep);
List"lt"double"gt" lst = new List"lt"double"gt"();
for (int j = 0; j "lt" usecols.Length; ++j)
lst.Add(double.Parse(tokens[usecols[j]]));
double[] row = lst.ToArray();
result.Add(row);
}
sr.Close(); ifs.Close();
return result.ToArray();
}
// ------------------------------------------------------
public static double[] MatToVec(double[][] m)
{
int rows = m.Length;
int cols = m[0].Length;
double[] result = new double[rows * cols];
int k = 0;
for (int i = 0; i "lt" rows; ++i)
for (int j = 0; j "lt" cols; ++j)
result[k++] = m[i][j];
return result;
}
// ------------------------------------------------------
public static double[][] MatToOneHot(double[][] m,
int n)
{
// convert ordinal (0,1,2 . .) to one-hot
int rows = m.Length;
int cols = m[0].Length; // assumed 1
double[][] result = MatCreate(rows, n);
for (int i = 0; i "lt" rows; ++i)
{
int k = (int)m[i][0]; // 0,1,2 . .
result[i] = new double[n]; // [0.0 0.0 0.0]
result[i][k] = 1.0; // [ 0.0 1.0 0.0]
}
return result;
}
// ------------------------------------------------------
public static void MatShow(double[][] m,
int dec, int wid)
{
for (int i = 0; i "lt" m.Length; ++i)
{
for (int j = 0; j "lt" m[0].Length; ++j)
{
double v = m[i][j];
if (Math.Abs(v) "lt" 1.0e-8) v = 0.0; // hack
Console.Write(v.ToString("F" +
dec).PadLeft(wid));
}
Console.WriteLine("");
}
}
// ------------------------------------------------------
public static void VecShow(int[] vec, int wid)
{
for (int i = 0; i "lt" vec.Length; ++i)
Console.Write(vec[i].ToString().PadLeft(wid));
Console.WriteLine("");
}
// ------------------------------------------------------
public static void VecShow(double[] vec,
int dec, int wid, bool newLine)
{
for (int i = 0; i "lt" vec.Length; ++i)
{
double x = vec[i];
if (Math.Abs(x) "lt" 1.0e-8) x = 0.0;
Console.Write(x.ToString("F" +
dec).PadLeft(wid));
}
if (newLine == true)
Console.WriteLine("");
}
} // Utils class
} // ns
Training data:
# people_train.txt # sex (0 = male, 1 = female) - dependent variable # age, state (michigan, nebraska, oklahoma), income, # politics type (conservative, moderate, liberal) # 1, 0.24, 1, 0, 0, 0.2950, 0, 0, 1 0, 0.39, 0, 0, 1, 0.5120, 0, 1, 0 1, 0.63, 0, 1, 0, 0.7580, 1, 0, 0 0, 0.36, 1, 0, 0, 0.4450, 0, 1, 0 1, 0.27, 0, 1, 0, 0.2860, 0, 0, 1 1, 0.50, 0, 1, 0, 0.5650, 0, 1, 0 1, 0.50, 0, 0, 1, 0.5500, 0, 1, 0 0, 0.19, 0, 0, 1, 0.3270, 1, 0, 0 1, 0.22, 0, 1, 0, 0.2770, 0, 1, 0 0, 0.39, 0, 0, 1, 0.4710, 0, 0, 1 1, 0.34, 1, 0, 0, 0.3940, 0, 1, 0 0, 0.22, 1, 0, 0, 0.3350, 1, 0, 0 1, 0.35, 0, 0, 1, 0.3520, 0, 0, 1 0, 0.33, 0, 1, 0, 0.4640, 0, 1, 0 1, 0.45, 0, 1, 0, 0.5410, 0, 1, 0 1, 0.42, 0, 1, 0, 0.5070, 0, 1, 0 0, 0.33, 0, 1, 0, 0.4680, 0, 1, 0 1, 0.25, 0, 0, 1, 0.3000, 0, 1, 0 0, 0.31, 0, 1, 0, 0.4640, 1, 0, 0 1, 0.27, 1, 0, 0, 0.3250, 0, 0, 1 1, 0.48, 1, 0, 0, 0.5400, 0, 1, 0 0, 0.64, 0, 1, 0, 0.7130, 0, 0, 1 1, 0.61, 0, 1, 0, 0.7240, 1, 0, 0 1, 0.54, 0, 0, 1, 0.6100, 1, 0, 0 1, 0.29, 1, 0, 0, 0.3630, 1, 0, 0 1, 0.50, 0, 0, 1, 0.5500, 0, 1, 0 1, 0.55, 0, 0, 1, 0.6250, 1, 0, 0 1, 0.40, 1, 0, 0, 0.5240, 1, 0, 0 1, 0.22, 1, 0, 0, 0.2360, 0, 0, 1 1, 0.68, 0, 1, 0, 0.7840, 1, 0, 0 0, 0.60, 1, 0, 0, 0.7170, 0, 0, 1 0, 0.34, 0, 0, 1, 0.4650, 0, 1, 0 0, 0.25, 0, 0, 1, 0.3710, 1, 0, 0 0, 0.31, 0, 1, 0, 0.4890, 0, 1, 0 1, 0.43, 0, 0, 1, 0.4800, 0, 1, 0 1, 0.58, 0, 1, 0, 0.6540, 0, 0, 1 0, 0.55, 0, 1, 0, 0.6070, 0, 0, 1 0, 0.43, 0, 1, 0, 0.5110, 0, 1, 0 0, 0.43, 0, 0, 1, 0.5320, 0, 1, 0 0, 0.21, 1, 0, 0, 0.3720, 1, 0, 0 1, 0.55, 0, 0, 1, 0.6460, 1, 0, 0 1, 0.64, 0, 1, 0, 0.7480, 1, 0, 0 0, 0.41, 1, 0, 0, 0.5880, 0, 1, 0 1, 0.64, 0, 0, 1, 0.7270, 1, 0, 0 0, 0.56, 0, 0, 1, 0.6660, 0, 0, 1 1, 0.31, 0, 0, 1, 0.3600, 0, 1, 0 0, 0.65, 0, 0, 1, 0.7010, 0, 0, 1 1, 0.55, 0, 0, 1, 0.6430, 1, 0, 0 0, 0.25, 1, 0, 0, 0.4030, 1, 0, 0 1, 0.46, 0, 0, 1, 0.5100, 0, 1, 0 0, 0.36, 1, 0, 0, 0.5350, 1, 0, 0 1, 0.52, 0, 1, 0, 0.5810, 0, 1, 0 1, 0.61, 0, 0, 1, 0.6790, 1, 0, 0 1, 0.57, 0, 0, 1, 0.6570, 1, 0, 0 0, 0.46, 0, 1, 0, 0.5260, 0, 1, 0 0, 0.62, 1, 0, 0, 0.6680, 0, 0, 1 1, 0.55, 0, 0, 1, 0.6270, 1, 0, 0 0, 0.22, 0, 0, 1, 0.2770, 0, 1, 0 0, 0.50, 1, 0, 0, 0.6290, 1, 0, 0 0, 0.32, 0, 1, 0, 0.4180, 0, 1, 0 0, 0.21, 0, 0, 1, 0.3560, 1, 0, 0 1, 0.44, 0, 1, 0, 0.5200, 0, 1, 0 1, 0.46, 0, 1, 0, 0.5170, 0, 1, 0 1, 0.62, 0, 1, 0, 0.6970, 1, 0, 0 1, 0.57, 0, 1, 0, 0.6640, 1, 0, 0 0, 0.67, 0, 0, 1, 0.7580, 0, 0, 1 1, 0.29, 1, 0, 0, 0.3430, 0, 0, 1 1, 0.53, 1, 0, 0, 0.6010, 1, 0, 0 0, 0.44, 1, 0, 0, 0.5480, 0, 1, 0 1, 0.46, 0, 1, 0, 0.5230, 0, 1, 0 0, 0.20, 0, 1, 0, 0.3010, 0, 1, 0 0, 0.38, 1, 0, 0, 0.5350, 0, 1, 0 1, 0.50, 0, 1, 0, 0.5860, 0, 1, 0 1, 0.33, 0, 1, 0, 0.4250, 0, 1, 0 0, 0.33, 0, 1, 0, 0.3930, 0, 1, 0 1, 0.26, 0, 1, 0, 0.4040, 1, 0, 0 1, 0.58, 1, 0, 0, 0.7070, 1, 0, 0 1, 0.43, 0, 0, 1, 0.4800, 0, 1, 0 0, 0.46, 1, 0, 0, 0.6440, 1, 0, 0 1, 0.60, 1, 0, 0, 0.7170, 1, 0, 0 0, 0.42, 1, 0, 0, 0.4890, 0, 1, 0 0, 0.56, 0, 0, 1, 0.5640, 0, 0, 1 0, 0.62, 0, 1, 0, 0.6630, 0, 0, 1 0, 0.50, 1, 0, 0, 0.6480, 0, 1, 0 1, 0.47, 0, 0, 1, 0.5200, 0, 1, 0 0, 0.67, 0, 1, 0, 0.8040, 0, 0, 1 0, 0.40, 0, 0, 1, 0.5040, 0, 1, 0 1, 0.42, 0, 1, 0, 0.4840, 0, 1, 0 1, 0.64, 1, 0, 0, 0.7200, 1, 0, 0 0, 0.47, 1, 0, 0, 0.5870, 0, 0, 1 1, 0.45, 0, 1, 0, 0.5280, 0, 1, 0 0, 0.25, 0, 0, 1, 0.4090, 1, 0, 0 1, 0.38, 1, 0, 0, 0.4840, 1, 0, 0 1, 0.55, 0, 0, 1, 0.6000, 0, 1, 0 0, 0.44, 1, 0, 0, 0.6060, 0, 1, 0 1, 0.33, 1, 0, 0, 0.4100, 0, 1, 0 1, 0.34, 0, 0, 1, 0.3900, 0, 1, 0 1, 0.27, 0, 1, 0, 0.3370, 0, 0, 1 1, 0.32, 0, 1, 0, 0.4070, 0, 1, 0 1, 0.42, 0, 0, 1, 0.4700, 0, 1, 0 0, 0.24, 0, 0, 1, 0.4030, 1, 0, 0 1, 0.42, 0, 1, 0, 0.5030, 0, 1, 0 1, 0.25, 0, 0, 1, 0.2800, 0, 0, 1 1, 0.51, 0, 1, 0, 0.5800, 0, 1, 0 0, 0.55, 0, 1, 0, 0.6350, 0, 0, 1 1, 0.44, 1, 0, 0, 0.4780, 0, 0, 1 0, 0.18, 1, 0, 0, 0.3980, 1, 0, 0 0, 0.67, 0, 1, 0, 0.7160, 0, 0, 1 1, 0.45, 0, 0, 1, 0.5000, 0, 1, 0 1, 0.48, 1, 0, 0, 0.5580, 0, 1, 0 0, 0.25, 0, 1, 0, 0.3900, 0, 1, 0 0, 0.67, 1, 0, 0, 0.7830, 0, 1, 0 1, 0.37, 0, 0, 1, 0.4200, 0, 1, 0 0, 0.32, 1, 0, 0, 0.4270, 0, 1, 0 1, 0.48, 1, 0, 0, 0.5700, 0, 1, 0 0, 0.66, 0, 0, 1, 0.7500, 0, 0, 1 1, 0.61, 1, 0, 0, 0.7000, 1, 0, 0 0, 0.58, 0, 0, 1, 0.6890, 0, 1, 0 1, 0.19, 1, 0, 0, 0.2400, 0, 0, 1 1, 0.38, 0, 0, 1, 0.4300, 0, 1, 0 0, 0.27, 1, 0, 0, 0.3640, 0, 1, 0 1, 0.42, 1, 0, 0, 0.4800, 0, 1, 0 1, 0.60, 1, 0, 0, 0.7130, 1, 0, 0 0, 0.27, 0, 0, 1, 0.3480, 1, 0, 0 1, 0.29, 0, 1, 0, 0.3710, 1, 0, 0 0, 0.43, 1, 0, 0, 0.5670, 0, 1, 0 1, 0.48, 1, 0, 0, 0.5670, 0, 1, 0 1, 0.27, 0, 0, 1, 0.2940, 0, 0, 1 0, 0.44, 1, 0, 0, 0.5520, 1, 0, 0 1, 0.23, 0, 1, 0, 0.2630, 0, 0, 1 0, 0.36, 0, 1, 0, 0.5300, 0, 0, 1 1, 0.64, 0, 0, 1, 0.7250, 1, 0, 0 1, 0.29, 0, 0, 1, 0.3000, 0, 0, 1 0, 0.33, 1, 0, 0, 0.4930, 0, 1, 0 0, 0.66, 0, 1, 0, 0.7500, 0, 0, 1 0, 0.21, 0, 0, 1, 0.3430, 1, 0, 0 1, 0.27, 1, 0, 0, 0.3270, 0, 0, 1 1, 0.29, 1, 0, 0, 0.3180, 0, 0, 1 0, 0.31, 1, 0, 0, 0.4860, 0, 1, 0 1, 0.36, 0, 0, 1, 0.4100, 0, 1, 0 1, 0.49, 0, 1, 0, 0.5570, 0, 1, 0 0, 0.28, 1, 0, 0, 0.3840, 1, 0, 0 0, 0.43, 0, 0, 1, 0.5660, 0, 1, 0 0, 0.46, 0, 1, 0, 0.5880, 0, 1, 0 1, 0.57, 1, 0, 0, 0.6980, 1, 0, 0 0, 0.52, 0, 0, 1, 0.5940, 0, 1, 0 0, 0.31, 0, 0, 1, 0.4350, 0, 1, 0 0, 0.55, 1, 0, 0, 0.6200, 0, 0, 1 1, 0.50, 1, 0, 0, 0.5640, 0, 1, 0 1, 0.48, 0, 1, 0, 0.5590, 0, 1, 0 0, 0.22, 0, 0, 1, 0.3450, 1, 0, 0 1, 0.59, 0, 0, 1, 0.6670, 1, 0, 0 1, 0.34, 1, 0, 0, 0.4280, 0, 0, 1 0, 0.64, 1, 0, 0, 0.7720, 0, 0, 1 1, 0.29, 0, 0, 1, 0.3350, 0, 0, 1 0, 0.34, 0, 1, 0, 0.4320, 0, 1, 0 0, 0.61, 1, 0, 0, 0.7500, 0, 0, 1 1, 0.64, 0, 0, 1, 0.7110, 1, 0, 0 0, 0.29, 1, 0, 0, 0.4130, 1, 0, 0 1, 0.63, 0, 1, 0, 0.7060, 1, 0, 0 0, 0.29, 0, 1, 0, 0.4000, 1, 0, 0 0, 0.51, 1, 0, 0, 0.6270, 0, 1, 0 0, 0.24, 0, 0, 1, 0.3770, 1, 0, 0 1, 0.48, 0, 1, 0, 0.5750, 0, 1, 0 1, 0.18, 1, 0, 0, 0.2740, 1, 0, 0 1, 0.18, 1, 0, 0, 0.2030, 0, 0, 1 1, 0.33, 0, 1, 0, 0.3820, 0, 0, 1 0, 0.20, 0, 0, 1, 0.3480, 1, 0, 0 1, 0.29, 0, 0, 1, 0.3300, 0, 0, 1 0, 0.44, 0, 0, 1, 0.6300, 1, 0, 0 0, 0.65, 0, 0, 1, 0.8180, 1, 0, 0 0, 0.56, 1, 0, 0, 0.6370, 0, 0, 1 0, 0.52, 0, 0, 1, 0.5840, 0, 1, 0 0, 0.29, 0, 1, 0, 0.4860, 1, 0, 0 0, 0.47, 0, 1, 0, 0.5890, 0, 1, 0 1, 0.68, 1, 0, 0, 0.7260, 0, 0, 1 1, 0.31, 0, 0, 1, 0.3600, 0, 1, 0 1, 0.61, 0, 1, 0, 0.6250, 0, 0, 1 1, 0.19, 0, 1, 0, 0.2150, 0, 0, 1 1, 0.38, 0, 0, 1, 0.4300, 0, 1, 0 0, 0.26, 1, 0, 0, 0.4230, 1, 0, 0 1, 0.61, 0, 1, 0, 0.6740, 1, 0, 0 1, 0.40, 1, 0, 0, 0.4650, 0, 1, 0 0, 0.49, 1, 0, 0, 0.6520, 0, 1, 0 1, 0.56, 1, 0, 0, 0.6750, 1, 0, 0 0, 0.48, 0, 1, 0, 0.6600, 0, 1, 0 1, 0.52, 1, 0, 0, 0.5630, 0, 0, 1 0, 0.18, 1, 0, 0, 0.2980, 1, 0, 0 0, 0.56, 0, 0, 1, 0.5930, 0, 0, 1 0, 0.52, 0, 1, 0, 0.6440, 0, 1, 0 0, 0.18, 0, 1, 0, 0.2860, 0, 1, 0 0, 0.58, 1, 0, 0, 0.6620, 0, 0, 1 0, 0.39, 0, 1, 0, 0.5510, 0, 1, 0 0, 0.46, 1, 0, 0, 0.6290, 0, 1, 0 0, 0.40, 0, 1, 0, 0.4620, 0, 1, 0 0, 0.60, 1, 0, 0, 0.7270, 0, 0, 1 1, 0.36, 0, 1, 0, 0.4070, 0, 0, 1 1, 0.44, 1, 0, 0, 0.5230, 0, 1, 0 1, 0.28, 1, 0, 0, 0.3130, 0, 0, 1 1, 0.54, 0, 0, 1, 0.6260, 1, 0, 0
Test data:
# people_test.txt # 0, 0.51, 1, 0, 0, 0.6120, 0, 1, 0 0, 0.32, 0, 1, 0, 0.4610, 0, 1, 0 1, 0.55, 1, 0, 0, 0.6270, 1, 0, 0 1, 0.25, 0, 0, 1, 0.2620, 0, 0, 1 1, 0.33, 0, 0, 1, 0.3730, 0, 0, 1 0, 0.29, 0, 1, 0, 0.4620, 1, 0, 0 1, 0.65, 1, 0, 0, 0.7270, 1, 0, 0 0, 0.43, 0, 1, 0, 0.5140, 0, 1, 0 0, 0.54, 0, 1, 0, 0.6480, 0, 0, 1 1, 0.61, 0, 1, 0, 0.7270, 1, 0, 0 1, 0.52, 0, 1, 0, 0.6360, 1, 0, 0 1, 0.30, 0, 1, 0, 0.3350, 0, 0, 1 1, 0.29, 1, 0, 0, 0.3140, 0, 0, 1 0, 0.47, 0, 0, 1, 0.5940, 0, 1, 0 1, 0.39, 0, 1, 0, 0.4780, 0, 1, 0 1, 0.47, 0, 0, 1, 0.5200, 0, 1, 0 0, 0.49, 1, 0, 0, 0.5860, 0, 1, 0 0, 0.63, 0, 0, 1, 0.6740, 0, 0, 1 0, 0.30, 1, 0, 0, 0.3920, 1, 0, 0 0, 0.61, 0, 0, 1, 0.6960, 0, 0, 1 0, 0.47, 0, 0, 1, 0.5870, 0, 1, 0 1, 0.30, 0, 0, 1, 0.3450, 0, 0, 1 0, 0.51, 0, 0, 1, 0.5800, 0, 1, 0 0, 0.24, 1, 0, 0, 0.3880, 0, 1, 0 0, 0.49, 1, 0, 0, 0.6450, 0, 1, 0 1, 0.66, 0, 0, 1, 0.7450, 1, 0, 0 0, 0.65, 1, 0, 0, 0.7690, 1, 0, 0 0, 0.46, 0, 1, 0, 0.5800, 1, 0, 0 0, 0.45, 0, 0, 1, 0.5180, 0, 1, 0 0, 0.47, 1, 0, 0, 0.6360, 1, 0, 0 0, 0.29, 1, 0, 0, 0.4480, 1, 0, 0 0, 0.57, 0, 0, 1, 0.6930, 0, 0, 1 0, 0.20, 1, 0, 0, 0.2870, 0, 0, 1 0, 0.35, 1, 0, 0, 0.4340, 0, 1, 0 0, 0.61, 0, 0, 1, 0.6700, 0, 0, 1 0, 0.31, 0, 0, 1, 0.3730, 0, 1, 0 1, 0.18, 1, 0, 0, 0.2080, 0, 0, 1 1, 0.26, 0, 0, 1, 0.2920, 0, 0, 1 0, 0.28, 1, 0, 0, 0.3640, 0, 0, 1 0, 0.59, 0, 0, 1, 0.6940, 0, 0, 1
.NET Test Automation Recipes
Software Testing
SciPy Programming Succinctly
Keras Succinctly
R Programming
2026 Visual Studio Live
2025 Summer MLADS Conference
2026 DevIntersection Conference
2025 Machine Learning Week
2025 Ai4 Conference
2026 G2E Conference
2026 iSC West Conference
You must be logged in to post a comment.