This is my third stab at a neural network quantile regression system. My first exploration used PyTorch combined with a custom quantile loss function. It worked quite well. My second exploration used a C# neural network, but with a modified Predict() method, only because I couldn’t get a version with a quantile loss function to calibrate correctly. It worked moderately well, but the modified predict technique has theoretical problems (that would take too long to explain). This third version uses a C# neural network, but I was able to make a quantile loss function work. It works well and feels right.
Explaining what quantile regression is, is not simple. Briefly, quantile regression is a machine learning technique to predict a single numeric value in situations where you care about under-prediction (or sometimes over-prediction). I’ll phrase the rest of this blog post in terms of scenarios where you mostly care about under-prediction.
For example, suppose you want to predict how much liquor to order for an important event. There’s a high negative consequence if your prediction is too low (angry people who don’t get a drink) but minor consequence if your prediction is too high (you just hold the excess liquor until the next party). In such a scenario, you could make a 90th percentile regression prediction, which means 90% of your predictions will be correct or sightly high (and therefore only 10% of your predictions would be too low). You can also loosely phrase this as, “There’s a 90% chance my prediction will meet the demand.”
Please note that I am dramatically over-simplifying the ideas of quantile regression, because otherwise I’d never get to my demo code.
Most machine learning regression techniques over-predict about 50% of the time, and under-predict about 50% of the time. Put another way, most regression techniques are implicitly 50th percentile quantile techniques.
Classical quantile regression techniques exist but usually do not work very well. They are variations of linear regression (and therefore not very powerful) and are extremely difficult to train because there require linear programming (and therefore very complex). However, a relatively new form of quantile regression is neural network quantile regression — a variation of neural network regression. With PyTorch, by using a custom loss function that penalizes low predictions more than high predictions, you can coerce the network to make high predictions to a specified quantile value, such as 90 percentile.
I spent many hours experimenting with a C# neural network with a custom quantile loss function, and its math calculus derivative. But, originally, that approach only worked partially. For a 90th percentile quantile neural network, with C# and a custom quantile loss function, I just couldn’t calibrate the network well. For example, if I specified a 90th percentile quantile network, the final trained network over-predicted only about 75% of the time rather than the required 90% of the time.
This blog post describes how I solved the calibration issue.
For my demo experiment, I used a set of synthetic data that looks like:
-0.1660, 0.4406, -0.9998, -0.3953, -0.7065, -0.8153, 0.7022 -0.2065, 0.0776, -0.1616, 0.3704, -0.5911, 0.7562, 0.5666 -0.9452, 0.3409, -0.1654, 0.1174, -0.7192, -0.6038, 0.8186 . . .
Each line has 6 predictor variables (all values between -1 and +1), followed by the target variable to predict. There are 200 training items and 40 test items. The data was generated by a 6-10-1 neural network with random weights and bias values, and therefore the data has an underlying structure that is, in theory, predictable.
The output of my demo is:
Begin neural network quantile regression with quantile loss Loading train (200) and test (40) data Done First three train X: -0.1660 0.4406 -0.9998 -0.3953 -0.7065 -0.8153 -0.2065 0.0776 -0.1616 0.3704 -0.5911 0.7562 -0.9452 0.3409 -0.1654 0.1174 -0.7192 -0.6038 First three train y: 0.7022 0.5666 0.8186 Creating 6-100-1 0.90 quantile neural network Done Setting train parameters maxEpochs = 180 lrnRate = 0.020 qRate = 0.010 batSize = 10 Starting training epoch: 0 MSE = 0.0293 acc = 0.3200 curr quantile = 0.6100 epoch: 18 MSE = 0.0296 acc = 0.4100 curr quantile = 0.7850 epoch: 36 MSE = 0.0040 acc = 0.7000 curr quantile = 0.8200 epoch: 54 MSE = 0.0031 acc = 0.7550 curr quantile = 0.8300 epoch: 72 MSE = 0.0034 acc = 0.7450 curr quantile = 0.8550 epoch: 90 MSE = 0.0034 acc = 0.7500 curr quantile = 0.8500 epoch: 108 MSE = 0.0039 acc = 0.7200 curr quantile = 0.9050 epoch: 126 MSE = 0.0034 acc = 0.7450 curr quantile = 0.8500 epoch: 144 MSE = 0.0038 acc = 0.7250 curr quantile = 0.8900 epoch: 162 MSE = 0.0037 acc = 0.7350 curr quantile = 0.8850 Done Evaluating final quantile regression model Accuracy (15%) on train data = 0.7100 Accuracy (15%) on test data = 0.7000 Computed quantile training = 0.90 Computed quantile test = 0.75 Predicting for x = -0.1660 0.4406 -0.9998 -0.3953 -0.7065 -0.8153 Predicted y = 0.7351 End demo
The output should be fairly self-explanatory. The trained neural network quantile regression system predicts the training data at 90th percentile as desired. In other words, 180 out of the 200 predicted values for the training data are at or above the true target values — in other words there’s little under-prediction. The test data should also be at about 90th percentile quantile, but because there’s only 40 test items, the 75th percentile result is close enough.
Notice that there’s a significant price to pay for quantile regression. The prediction accuracy of the demo system is only 71.00% (142 out of 200) on the training data, where a prediction is scored as correct if it’s within 15% of the true target value.
My first attempt at training a neural network quantile regression system using a quantile loss function was straightforward. I’m familiar with the theory of back-propagation. Each weight has a gradient, which is value that indicates how much, and in what direction, to update its associated weight so that computed output is closer to target output. Interestingly, the error function (most usual are cross entropy error and mean squared error) isn’t used explicitly — instead you use the calculus derivative of the error function, combined with the calculus derivative of the output activation function.
The key equations for back-propagation for multi-class classification (not regression) using mean squared error or cross entropy error. Very tricky stuff.
The calculus derivative of the output activation is easy; it’s just the constant 1.0 for Identity() activation used for regression problems. The calculus derivative of the quantile loss is, “the quantile value t if actual target value y is greater than the computed predicted value y, else 1 – t”. Technically, the derivative doesn’t exist when the computed y equals the target y, but that’s not a problem in practice.
So, the key code I first tried was:
// First attempt -- training stalls out
// 1. compute output node scratch signals
for (int k = 0; k "lt" this.no; ++k)
{
// with quantile loss
double outActDerivative = 1.0; // Identity()
double lossDerivative; // for quantile loss
if (y "gte" this.oNodes[k]) // tech no exist at ==
lossDerivative = this.quantile;
else
lossDerivative = 1.0 - this.quantile;
oSignals[k] = outActDerivative *
lossDerivative * (this.oNodes[k] - y);
}
// use signals to compute gradients
// use gradients to update weights
And . . . it didn’t quite work. For example, if the specified quantile is 90th percentile (meaning 90% of the predictions should be greater than the true target values), the system could only reach about 78th percentile and then it stalled during training. When neural network training stalls, it’s usually due to the vanishing gradient problem.
After many hours, I finally tried adding a dynamic scaling factor that bumps the quantile value up or down during training, once each epoch, in order to keep gradients from becoming too small, and therefore achieve the desired percentile:
. . . // compute current quantile // this.quantile is desired quantile, like 0.90 . . . if (currQuantile "lt" this.quantile) this.quantScale += 0.01; // 0.01 is the qRate parameter else if (currQuantile "gt" this.quantile) this.quantScale -= 0.01; } // each epoch . . . // 1. compute output node scratch signals . . . if (y "gte" this.oNodes[k]) // tech no exist at == lossDerivative = this.quantScale * this.quantile; else lossDerivative = 1.0 - this.quantile;
And. . . it works quite well. This was a truly interesting exploration!
The vanishing gradient problem occurs quite often in neural networks. The vanishing/shrinking character theme occurred quite often in the old comics I read as a young man.
Left: “Tales of the Unexpected” #38, June 1959, cover artist Nick Cardy.
Center: “Forbidden Worlds” #66, May 1958, cover artist Ogden Whitney.
Right: “House of Mystery” #68, November 1957, cover artist Ruben Moreira.
Demo program. Long and complex. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols (my blog editor often chokes on symbols).
using System;
using System.IO;
using System.Collections.Generic;
namespace NeuralNetworkQuantileRegressionQuantileLoss
{
internal class NeuralQuantileRegressionProgram
{
static void Main(string[] args)
{
Console.WriteLine("\nBegin neural network" +
" quantile regression with quantile loss ");
// 1. load data
Console.WriteLine("\nLoading train (200) " +
"and test (40) data");
string trainFile =
"..\\..\\..\\Data\\synthetic_train_200.txt";
int[] colsX = new int[] { 0, 1, 2, 3, 4, 5, };
double[][] trainX =
MatLoad(trainFile, colsX, ',', "#");
double[] trainY =
MatToVec(MatLoad(trainFile,
new int[] { 6 }, ',', "#"));
string testFile =
"..\\..\\..\\Data\\synthetic_test_40.txt";
double[][] testX =
MatLoad(testFile, colsX, ',', "#");
double[] testY =
MatToVec(MatLoad(testFile,
new int[] { 6 }, ',', "#"));
Console.WriteLine("Done ");
Console.WriteLine("\nFirst three train X: ");
for (int i = 0; i "lt" 3; ++i)
VecShow(trainX[i], 4, 8);
Console.WriteLine("\nFirst three train y: ");
for (int i = 0; i "lt" 3; ++i)
Console.WriteLine(trainY[i].ToString("F4").
PadLeft(8));
// 2. create model
Console.WriteLine("\nCreating 6-100-1 " +
"0.90 quantile neural network ");
double quantile = 0.90;
NeuralNetwork nn =
new NeuralNetwork(6, 100, 1, quantile, seed: 0);
Console.WriteLine("Done ");
// 3. train model
Console.WriteLine("\nSetting train parameters ");
int maxEpochs = 180;
double lrnRate = 0.02;
double qRate = 0.01;
int batSize = 10;
Console.WriteLine("\nmaxEpochs = " +
maxEpochs);
Console.WriteLine("lrnRate = " +
lrnRate.ToString("F3"));
Console.WriteLine("qRate = " +
qRate.ToString("F3"));
Console.WriteLine("batSize = " + batSize);
Console.WriteLine("\nStarting training ");
nn.Train(trainX, trainY, lrnRate, qRate,
batSize, maxEpochs);
Console.WriteLine("Done ");
// 4. evaluate model
Console.WriteLine("\nEvaluating final quantile" +
" regression model ");
double trainAcc = nn.Accuracy(trainX, trainY, 0.15);
Console.WriteLine("\nAccuracy (15%) on train data = " +
trainAcc.ToString("F4"));
double testAcc = nn.Accuracy(testX, testY, 0.15);
Console.WriteLine("Accuracy (15%) on test data = " +
testAcc.ToString("F4"));
double qTrain = nn.ComputedQuantile(trainX, trainY);
Console.WriteLine("\nComputed quantile training = " +
qTrain.ToString("F2"));
double qTest = nn.ComputedQuantile(testX, testY);
Console.WriteLine("Computed quantile test = " +
qTest.ToString("F2"));
// 5. use model
double[] x = trainX[0];
Console.WriteLine("\nPredicting for x = ");
VecShow(x, 4, 8);
double predY = nn.Predict(x);
Console.WriteLine("\nPredicted y = " +
predY.ToString("F4"));
Console.WriteLine("\nEnd demo ");
Console.ReadLine();
// 6. show under-predictions on training data
// true target. For 90th percentile, should be 10%
// of training items (200 * 0.10 = 20 items)
//Console.WriteLine("\nModel under-predictions: ");
//int ct = 0;
//Console.WriteLine("\n item actual predicted ");
//Console.WriteLine("---------------------------");
//for (int i = 0; i "lt" trainX.Length; ++i)
//{
// double[] inptX = trainX[i];
// double actualY = trainY[i];
// double predictedY = nn.Predict(inptX);
// if (predictedY "lt" actualY)
// {
// Console.WriteLine(ct.ToString().PadLeft(5) +
// " " + actualY.ToString("F4") +
// " " + predictedY.ToString("F4"));
// ++ct;
// }
//}
//Console.ReadLine();
} // Main()
// ------------------------------------------------------
// helpers for Main()
// ------------------------------------------------------
static double[][] MatLoad(string fn, int[] usecols,
char sep, string comment)
{
List"lt"double[]"gt" result = new List"lt"double[]"gt"();
string line = "";
FileStream ifs = new FileStream(fn, FileMode.Open);
StreamReader sr = new StreamReader(ifs);
while ((line = sr.ReadLine()) != null)
{
if (line.StartsWith(comment) == true)
continue;
string[] tokens = line.Split(sep);
List"lt"double"gt" lst = new List"lt"double"gt"();
for (int j = 0; j "lt" usecols.Length; ++j)
lst.Add(double.Parse(tokens[usecols[j]]));
double[] row = lst.ToArray();
result.Add(row);
}
sr.Close(); ifs.Close();
return result.ToArray();
}
static double[] MatToVec(double[][] mat)
{
int nRows = mat.Length;
int nCols = mat[0].Length;
double[] result = new double[nRows * nCols];
int k = 0;
for (int i = 0; i "lt" nRows; ++i)
for (int j = 0; j "lt" nCols; ++j)
result[k++] = mat[i][j];
return result;
}
static void VecShow(double[] vec, int dec, int wid)
{
for (int i = 0; i "lt" vec.Length; ++i)
Console.Write(vec[i].ToString("F" + dec).
PadLeft(wid));
Console.WriteLine("");
}
} // Program
// --------------------------------------------------------
public class NeuralNetwork
{
private int ni; // number input nodes
private int nh;
private int no;
private double[] iNodes;
private double[][] ihWeights; // input-hidden
private double[] hBiases;
private double[] hNodes;
private double[][] hoWeights; // hidden-output
private double[] oBiases;
private double[] oNodes; // single val as array
// gradients
private double[][] ihGrads;
private double[] hbGrads;
private double[][] hoGrads;
private double[] obGrads;
// quantile
public double quantile; // desired model quantile
public double quantScale = 1.0; // initial grad scale
private Random rnd;
// ======================================================
public NeuralNetwork(int numIn, int numHid,
int numOut, double quantile, int seed)
{
this.ni = numIn;
this.nh = numHid;
this.no = numOut; // 1 for regression
this.quantile = quantile;
this.iNodes = new double[numIn];
this.ihWeights = MatCreate(numIn, numHid);
this.hBiases = new double[numHid];
this.hNodes = new double[numHid];
this.hoWeights = MatCreate(numHid, numOut);
this.oBiases = new double[numOut]; // [1]
this.oNodes = new double[numOut]; // [1]
this.ihGrads = MatCreate(numIn, numHid);
this.hbGrads = new double[numHid];
this.hoGrads = MatCreate(numHid, numOut);
this.obGrads = new double[numOut];
this.rnd = new Random(seed);
this.InitWeights(); // all weights and biases
} // ctor
// ------------------------------------------------------
private static double[][] MatCreate(int rows,
int cols)
{
double[][] result = new double[rows][];
for (int i = 0; i "lt" rows; ++i)
result[i] = new double[cols];
return result;
}
// ------------------------------------------------------
private void InitWeights() // helper for ctor
{
// weights and biases to small random values
double lo = -0.01; double hi = +0.01;
int numWts = (this.ni * this.nh) +
(this.nh * this.no) + this.nh + this.no;
double[] initialWeights = new double[numWts];
for (int i = 0; i "lt" initialWeights.Length; ++i)
initialWeights[i] =
(hi - lo) * rnd.NextDouble() + lo;
this.SetWeights(initialWeights);
}
// ------------------------------------------------------
public void SetWeights(double[] wts)
{
// copy serialized weights and biases in wts[]
// to ih weights, ih biases, ho weights, ho biases
int numWts = (this.ni * this.nh) +
(this.nh * this.no) + this.nh + this.no;
if (wts.Length != numWts)
throw new Exception("Bad array in SetWeights");
int k = 0; // points into wts param
for (int i = 0; i "lt" this.ni; ++i)
for (int j = 0; j "lt" this.nh; ++j)
this.ihWeights[i][j] = wts[k++];
for (int i = 0; i "lt" this.nh; ++i)
this.hBiases[i] = wts[k++];
for (int i = 0; i "lt" this.nh; ++i)
for (int j = 0; j "lt" this.no; ++j)
this.hoWeights[i][j] = wts[k++];
for (int i = 0; i "lt" this.no; ++i)
this.oBiases[i] = wts[k++];
}
// ------------------------------------------------------
public double[] GetWeights()
{
int numWts = (this.ni * this.nh) +
(this.nh * this.no) + this.nh + this.no;
double[] result = new double[numWts];
int k = 0;
for (int i = 0; i "lt" ihWeights.Length; ++i)
for (int j = 0; j "lt" this.ihWeights[0].Length; ++j)
result[k++] = this.ihWeights[i][j];
for (int i = 0; i "lt" this.hBiases.Length; ++i)
result[k++] = this.hBiases[i];
for (int i = 0; i "lt" this.hoWeights.Length; ++i)
for (int j = 0; j "lt" this.hoWeights[0].Length; ++j)
result[k++] = this.hoWeights[i][j];
for (int i = 0; i "lt" this.oBiases.Length; ++i)
result[k++] = this.oBiases[i];
return result;
}
// ------------------------------------------------------
public double Predict(double[] x)
{
double[] hSums = new double[this.nh]; // scratch
double[] oSums = new double[this.no]; // out sums
for (int i = 0; i "lt" x.Length; ++i)
this.iNodes[i] = x[i];
// note: no need to copy x-values unless
// you implement a ToString.
// more efficient to simply use the X[] directly.
// 1. compute i-h sum of weights * inputs
for (int j = 0; j "lt" this.nh; ++j)
for (int i = 0; i "lt" this.ni; ++i)
hSums[j] += this.iNodes[i] *
this.ihWeights[i][j]; // note +=
// 2. add biases to hidden sums
for (int i = 0; i "lt" this.nh; ++i)
hSums[i] += this.hBiases[i];
// 3. apply hidden activation
for (int i = 0; i "lt" this.nh; ++i)
this.hNodes[i] = HyperTan(hSums[i]);
//// 3. apply hidden activation
//for (int i = 0; i "lt" this.nh; ++i)
// this.hNodes[i] = Relu(hSums[i]);
// 4. compute h-o sum of wts * hOutputs
for (int j = 0; j "lt" this.no; ++j)
for (int i = 0; i "lt" this.nh; ++i)
oSums[j] += this.hNodes[i] *
this.hoWeights[i][j]; // [1]
// 5. add biases to output sums
for (int i = 0; i "lt" this.no; ++i)
oSums[i] += this.oBiases[i];
// 6. apply output activation
for (int i = 0; i "lt" this.no; ++i)
this.oNodes[i] = Identity(oSums[i]);
double result = this.oNodes[0];
return result; // single value
}
// ------------------------------------------------------
private static double HyperTan(double x)
{
if (x "lt" -10.0) return -1.0;
else if (x "gt" 10.0) return 1.0;
else return Math.Tanh(x);
}
// ------------------------------------------------------
private static double Identity(double x)
{
return x;
}
// ------------------------------------------------------
private static double Relu(double x)
{
if (x "lte" 0) return 0;
else return x;
}
// ------------------------------------------------------
private void ZeroOutGrads()
{
for (int i = 0; i "lt" this.ni; ++i)
for (int j = 0; j "lt" this.nh; ++j)
this.ihGrads[i][j] = 0.0;
for (int j = 0; j "lt" this.nh; ++j)
this.hbGrads[j] = 0.0;
for (int j = 0; j "lt" this.nh; ++j)
for (int k = 0; k "lt" this.no; ++k)
this.hoGrads[j][k] = 0.0;
for (int k = 0; k "lt" this.no; ++k)
this.obGrads[k] = 0.0;
} // ZeroOutGrads()
// ------------------------------------------------------
private void AccumGrads(double y)
{
double[] oSignals = new double[this.no];
double[] hSignals = new double[this.nh];
//def quantile_loss(preds, trgt, quantile):
// # quantile must be in (0.0, 1.0)
// errs = trgt - preds
// loss = T.max((quantile - 1.0) * errs,
// quantile * errs)
// return T.abs(loss).mean()
// 1. compute output node scratch signals
for (int k = 0; k "lt" this.no; ++k)
{
// std regression w/ CEE
// double derivative = 1.0; // of oupt act.
// oSignals[k] = derivative *
// (this.oNodes[k] - y); // of err
// std regression w/ MSE
// double derivative = 1.0; // of oupt act.
// oSignals[k] =
// derivative * (this.oNodes[k] - y) *
// (this.oNodes[k] * (1 - this.oNodes[k]));
// derivative is "the quantile value t if actual value
// is greater than the prediction, else 1 - t".
// with quantile loss
double outActDerivative = 1.0; // Identity()
double lossDerivative;
if (y "gte" this.oNodes[k]) // tech no exist at ==
lossDerivative = this.quantScale * this.quantile;
else
lossDerivative = 1.0 - this.quantile;
oSignals[k] = outActDerivative *
lossDerivative * (this.oNodes[k] - y);
}
// 2. accum hidden-to-output gradients
for (int j = 0; j "lt" this.nh; ++j)
for (int k = 0; k "lt" this.no; ++k)
hoGrads[j][k] +=
oSignals[k] * this.hNodes[j];
// 3. accum output node bias gradients
for (int k = 0; k "lt" this.no; ++k)
obGrads[k] +=
oSignals[k] * 1.0; // 1.0 dummy
// 4. compute hidden node signals
for (int j = 0; j "lt" this.nh; ++j)
{
double sum = 0.0;
for (int k = 0; k "lt" this.no; ++k)
sum += oSignals[k] * this.hoWeights[j][k];
double derivative =
(1 - this.hNodes[j]) *
(1 + this.hNodes[j]); // assumes tanh
hSignals[j] = derivative * sum;
}
// 5. accum input-to-hidden gradients
for (int i = 0; i "lt" this.ni; ++i)
for (int j = 0; j "lt" this.nh; ++j)
this.ihGrads[i][j] +=
hSignals[j] * this.iNodes[i];
// 6. accum hidden node bias gradients
for (int j = 0; j "lt" this.nh; ++j)
this.hbGrads[j] +=
hSignals[j] * 1.0; // 1.0 dummy
} // AccumGrads
// ------------------------------------------------------
private void UpdateWeights(double lrnRate)
{
// assumes all gradients computed
// 1. update input-to-hidden weights
for (int i = 0; i "lt" this.ni; ++i)
{
for (int j = 0; j "lt" this.nh; ++j)
{
double delta = -1.0 * lrnRate *
this.ihGrads[i][j];
this.ihWeights[i][j] += delta;
}
}
// 2. update hidden node biases
for (int j = 0; j "lt" this.nh; ++j)
{
double delta = -1.0 * lrnRate *
this.hbGrads[j];
this.hBiases[j] += delta;
}
// 3. update hidden-to-output weights
for (int j = 0; j "lt" this.nh; ++j)
{
for (int k = 0; k "lt" this.no; ++k)
{
double delta = -1.0 * lrnRate *
this.hoGrads[j][k];
this.hoWeights[j][k] += delta;
}
}
// 4. update output node biases
for (int k = 0; k "lt" this.no; ++k)
{
double delta = -1.0 * lrnRate *
this.obGrads[k];
this.oBiases[k] += delta;
}
} // UpdateWeights()
// ------------------------------------------------------
public void Train(double[][] trainX, double[] trainY,
double lrnRate, double qRate, int batSize,
int maxEpochs)
{
int n = trainX.Length; // like 200
int batchesPerEpoch = n / batSize; // like 20
int freq = maxEpochs / 10; // to show progress
int[] indices = new int[n];
for (int i = 0; i "lt" n; ++i)
indices[i] = i;
// ----------------------------------------------------
//
// n = 200; bs = 10
// batches per epoch = 200 / 10 = 20
// for epoch = 0; epoch "lt" maxEpochs; ++epoch
// shuffle indices
// for batch = 0; batch "lt" bpe; ++batch
// for item = 0; item "lt" bs; ++item
// compute output
// accum grads
// end-item
// update weights
// zero-out grads
// end-batches
// modify quantScale up or down
// end-epochs
//
// ----------------------------------------------------
for (int epoch = 0; epoch "lt" maxEpochs; ++epoch)
{
Shuffle(indices);
int ptr = 0; // points into indices
for (int batIdx = 0; batIdx "lt" batchesPerEpoch;
++batIdx) // 0, 1, . . 19
{
for (int i = 0; i "lt" batSize; ++i) // like 0..9
{
int ii = indices[ptr++]; // compute output
double[] x = trainX[ii];
double y = trainY[ii];
this.Predict(x); // into this.oNoodes
this.AccumGrads(y);
}
this.UpdateWeights(lrnRate);
this.ZeroOutGrads(); // prep for next batch
} // batches
double currQuantile =
this.ComputedQuantile(trainX, trainY);
if (epoch % freq == 0) // progress every few epochs
{
double mse =
this.Error(trainX, trainY);
double acc =
this.Accuracy(trainX, trainY, 0.15);
currQuantile =
this.ComputedQuantile(trainX, trainY);
string s1 = "epoch: " + epoch.ToString().PadLeft(4);
string s2 = " MSE = " + mse.ToString("F4");
string s3 = " acc = " + acc.ToString("F4");
string s4 = " curr quantile = " +
currQuantile.ToString("F4");
Console.WriteLine(s1 + s2 + s3 + s4);
}
if (currQuantile "lt" this.quantile)
this.quantScale += qRate;
else if (currQuantile "gt" this.quantile)
this.quantScale -= qRate;
} // each epoch
} // Train
// ------------------------------------------------------
private void Shuffle(int[] sequence)
{
for (int i = 0; i "lt" sequence.Length; ++i)
{
int r = this.rnd.Next(i, sequence.Length);
int tmp = sequence[r];
sequence[r] = sequence[i];
sequence[i] = tmp;
//sequence[i] = i; // for testing
}
} // Shuffle
// ------------------------------------------------------
public double Error(double[][] trainX, double[] trainY)
{
// MSE
int n = trainX.Length;
double sumSquaredError = 0.0;
for (int i = 0; i "lt" n; ++i)
{
double predY = this.Predict(trainX[i]);
double actualY = trainY[i];
sumSquaredError += (predY - actualY) *
(predY - actualY);
}
return sumSquaredError / n;
} // Error
// ------------------------------------------------------
public double Accuracy(double[][] dataX,
double[] dataY, double pctClose)
{
// percentage correct using winner-takes all
int n = dataX.Length;
int nCorrect = 0;
int nWrong = 0;
for (int i = 0; i "lt" n; ++i)
{
double predY = this.Predict(dataX[i]);
double actualY = dataY[i];
if (Math.Abs(predY - actualY) "lt"
Math.Abs(pctClose * actualY))
++nCorrect;
else
++nWrong;
}
return (nCorrect * 1.0) / (nCorrect + nWrong);
}
// ------------------------------------------------------
public double ComputedQuantile(double[][] dataX,
double[] dataY)
{
int n = dataX.Length;
int nHi = 0; // prediction higher than actual target
int nLo = 0;
for (int i = 0; i "lt" n; ++i)
{
double predY = this.Predict(dataX[i]);
double actualY = dataY[i];
if (predY "gte" actualY) ++nHi;
else ++nLo;
}
return (nHi * 1.0) / (nHi + nLo);
}
// ------------------------------------------------------
public void SaveWeights(string fn)
{
FileStream ofs = new FileStream(fn, FileMode.Create);
StreamWriter sw = new StreamWriter(ofs);
double[] wts = this.GetWeights();
for (int i = 0; i "lt" wts.Length; ++i)
sw.WriteLine(wts[i].ToString("F8")); // one per line
sw.Close();
ofs.Close();
}
public void LoadWeights(string fn)
{
FileStream ifs = new FileStream(fn, FileMode.Open);
StreamReader sr = new StreamReader(ifs);
List"lt"double"gt" listWts = new List"lt"double"gt"();
string line = ""; // one wt per line
while ((line = sr.ReadLine()) != null)
{
// if (line.StartsWith(comment) == true)
// continue;
listWts.Add(double.Parse(line));
}
sr.Close();
ifs.Close();
double[] wts = listWts.ToArray();
this.SetWeights(wts);
}
// ------------------------------------------------------
} // NeuralNetwork class
} // ns
Training data:
# synthetic_train.txt # -0.1660, 0.4406, -0.9998, -0.3953, -0.7065, -0.8153, 0.7022 -0.2065, 0.0776, -0.1616, 0.3704, -0.5911, 0.7562, 0.5666 -0.9452, 0.3409, -0.1654, 0.1174, -0.7192, -0.6038, 0.8186 0.7528, 0.7892, -0.8299, -0.9219, -0.6603, 0.7563, 0.3687 -0.8033, -0.1578, 0.9158, 0.0663, 0.3838, -0.3690, 0.7535 -0.9634, 0.5003, 0.9777, 0.4963, -0.4391, 0.5786, 0.7076 -0.7935, -0.1042, 0.8172, -0.4128, -0.4244, -0.7399, 0.8454 -0.0169, -0.8933, 0.1482, -0.7065, 0.1786, 0.3995, 0.7302 -0.7953, -0.1719, 0.3888, -0.1716, -0.9001, 0.0718, 0.8692 0.8892, 0.1731, 0.8068, -0.7251, -0.7214, 0.6148, 0.4740 -0.2046, -0.6693, 0.8550, -0.3045, 0.5016, 0.4520, 0.6714 0.5019, -0.3022, -0.4601, 0.7918, -0.1438, 0.9297, 0.4331 0.3269, 0.2434, -0.7705, 0.8990, -0.1002, 0.1568, 0.3716 0.8068, 0.1474, -0.9943, 0.2343, -0.3467, 0.0541, 0.3829 0.7719, -0.2855, 0.8171, 0.2467, -0.9684, 0.8589, 0.4700 0.8652, 0.3936, -0.8680, 0.5109, 0.5078, 0.8460, 0.2648 0.4230, -0.7515, -0.9602, -0.9476, -0.9434, -0.5076, 0.8059 0.1056, 0.6841, -0.7517, -0.4416, 0.1715, 0.9392, 0.3512 0.1221, -0.9627, 0.6013, -0.5341, 0.6142, -0.2243, 0.6840 0.1125, -0.7271, -0.8802, -0.7573, -0.9109, -0.7850, 0.8640 -0.5486, 0.4260, 0.1194, -0.9749, -0.8561, 0.9346, 0.6109 -0.4953, 0.4877, -0.6091, 0.1627, 0.9400, 0.6937, 0.3382 -0.5203, -0.0125, 0.2399, 0.6580, -0.6864, -0.9628, 0.7400 0.2127, 0.1377, -0.3653, 0.9772, 0.1595, -0.2397, 0.4081 0.1019, 0.4907, 0.3385, -0.4702, -0.8673, -0.2598, 0.6582 0.5055, -0.8669, -0.4794, 0.6095, -0.6131, 0.2789, 0.6644 0.0493, 0.8496, -0.4734, -0.8681, 0.4701, 0.5444, 0.3214 0.9004, 0.1133, 0.8312, 0.2831, -0.2200, -0.0280, 0.3149 0.2086, 0.0991, 0.8524, 0.8375, -0.2102, 0.9265, 0.3619 -0.7298, 0.0113, -0.9570, 0.8959, 0.6542, -0.9700, 0.6451 -0.6476, -0.3359, -0.7380, 0.6190, -0.3105, 0.8802, 0.6606 0.6895, 0.8108, -0.0802, 0.0927, 0.5972, -0.4286, 0.2427 -0.0195, 0.1982, -0.9689, 0.1870, -0.1326, 0.6147, 0.4773 0.1557, -0.6320, 0.5759, 0.2241, -0.8922, -0.1596, 0.7581 0.3581, 0.8372, -0.9992, 0.9535, -0.2468, 0.9476, 0.2962 0.1494, 0.2562, -0.4288, 0.1737, 0.5000, 0.7166, 0.3513 0.5102, 0.3961, 0.7290, -0.3546, 0.3416, -0.0983, 0.3153 -0.1970, -0.3652, 0.2438, -0.1395, 0.9476, 0.3556, 0.4719 -0.6029, -0.1466, -0.3133, 0.5953, 0.7600, 0.8077, 0.3875 -0.4953, 0.7098, 0.0554, 0.6043, 0.1450, 0.4663, 0.4739 0.0380, 0.5418, 0.1377, -0.0686, -0.3146, -0.8636, 0.6048 0.9656, -0.6368, 0.6237, 0.7499, 0.3768, 0.1390, 0.3705 -0.6781, -0.0662, -0.3097, -0.5499, 0.1850, -0.3755, 0.7668 -0.6141, -0.0008, 0.4572, -0.5836, -0.5039, 0.7033, 0.7301 -0.1683, 0.2334, -0.5327, -0.7961, 0.0317, -0.0457, 0.5777 0.0880, 0.3083, -0.7109, 0.5031, -0.5559, 0.0387, 0.5118 0.5706, -0.9553, -0.3513, 0.7458, 0.6894, 0.0769, 0.4329 -0.8025, 0.3026, 0.4070, 0.2205, 0.5992, -0.9309, 0.7098 0.5405, 0.4635, -0.4806, -0.4859, 0.2646, -0.3094, 0.3566 0.5655, 0.9809, -0.3995, -0.7140, 0.8026, 0.0831, 0.2551 0.9495, 0.2732, 0.9878, 0.0921, 0.0529, -0.7291, 0.3074 -0.6792, 0.4913, -0.9392, -0.2669, 0.7247, 0.3854, 0.4362 0.3819, -0.6227, -0.1162, 0.1632, 0.9795, -0.5922, 0.4435 0.5003, -0.0860, -0.8861, 0.0170, -0.5761, 0.5972, 0.5136 -0.4053, -0.9448, 0.1869, 0.6877, -0.2380, 0.4997, 0.7859 0.9189, 0.6079, -0.9354, 0.4188, -0.0700, 0.8951, 0.2696 -0.5571, -0.4659, -0.8371, -0.1428, -0.7820, 0.2676, 0.8566 0.5324, -0.3151, 0.6917, -0.1425, 0.6480, 0.2530, 0.4252 -0.7132, -0.8432, -0.9633, -0.8666, -0.0828, -0.7733, 0.9217 -0.0952, -0.0998, -0.0439, -0.0520, 0.6063, -0.1952, 0.5140 0.8094, -0.9259, 0.5477, -0.7487, 0.2370, -0.9793, 0.5562 0.9024, 0.8108, 0.5919, 0.8305, -0.7089, -0.6845, 0.2993 -0.6247, 0.2450, 0.8116, 0.9799, 0.4222, 0.4636, 0.4619 -0.5003, -0.6531, -0.7611, 0.6252, -0.7064, -0.4714, 0.8452 0.6382, -0.3788, 0.9648, -0.4667, 0.0673, -0.3711, 0.5070 -0.1328, 0.0246, 0.8778, -0.9381, 0.4338, 0.7820, 0.5680 -0.9454, 0.0441, -0.3480, 0.7190, 0.1170, 0.3805, 0.6562 -0.4198, -0.9813, 0.1535, -0.3771, 0.0345, 0.8328, 0.7707 -0.1471, -0.5052, -0.2574, 0.8637, 0.8737, 0.6887, 0.3436 -0.3712, -0.6505, 0.2142, -0.1728, 0.6327, -0.6297, 0.7430 0.4038, -0.5193, 0.1484, -0.3020, -0.8861, -0.5424, 0.7499 0.0380, -0.6506, 0.1414, 0.9935, 0.6337, 0.1887, 0.4509 0.9520, 0.8031, 0.1912, -0.9351, -0.8128, -0.8693, 0.5336 0.9507, -0.6640, 0.9456, 0.5349, 0.6485, 0.2652, 0.3616 0.3375, -0.0462, -0.9737, -0.2940, -0.0159, 0.4602, 0.4840 -0.7247, -0.9782, 0.5166, -0.3601, 0.9688, -0.5595, 0.7751 -0.3226, 0.0478, 0.5098, -0.0723, -0.7504, -0.3750, 0.8025 0.5403, -0.7393, -0.9542, 0.0382, 0.6200, -0.9748, 0.5359 0.3449, 0.3736, -0.1015, 0.8296, 0.2887, -0.9895, 0.4390 0.6608, 0.2983, 0.3474, 0.1570, -0.4518, 0.1211, 0.3624 0.3435, -0.2951, 0.7117, -0.6099, 0.4946, -0.4208, 0.5283 0.6154, -0.2929, -0.5726, 0.5346, -0.3827, 0.4665, 0.4907 0.4889, -0.5572, -0.5718, -0.6021, -0.7150, -0.2458, 0.7202 -0.8389, -0.5366, -0.5847, 0.8347, 0.4226, 0.1078, 0.6391 -0.3910, 0.6697, -0.1294, 0.8469, 0.4121, -0.0439, 0.4693 -0.1376, -0.1916, -0.7065, 0.4586, -0.6225, 0.2878, 0.6695 0.5086, -0.5785, 0.2019, 0.4979, 0.2764, 0.1943, 0.4666 0.8906, -0.1489, 0.5644, -0.8877, 0.6705, -0.6155, 0.3480 -0.2098, -0.3998, -0.8398, 0.8093, -0.2597, 0.0614, 0.6341 -0.5871, -0.8476, 0.0158, -0.4769, -0.2859, -0.7839, 0.9006 0.5751, -0.7868, 0.9714, -0.6457, 0.1448, -0.9103, 0.6049 0.0558, 0.4802, -0.7001, 0.1022, -0.5668, 0.5184, 0.4612 0.4458, -0.6469, 0.7239, -0.9604, 0.7205, 0.1178, 0.5941 0.4339, 0.9747, -0.4438, -0.9924, 0.8678, 0.7158, 0.2627 0.4577, 0.0334, 0.4139, 0.5611, -0.2502, 0.5406, 0.3847 -0.1963, 0.3946, -0.9938, 0.5498, 0.7928, -0.5214, 0.5025 -0.7585, -0.5594, -0.3958, 0.7661, 0.0863, -0.4266, 0.7481 0.2277, -0.3517, -0.0853, -0.1118, 0.6563, -0.1473, 0.4798 -0.3086, 0.3499, -0.5570, -0.0655, -0.3705, 0.2537, 0.5768 0.5689, -0.0861, 0.3125, -0.7363, -0.1340, 0.8186, 0.5035 0.2110, 0.5335, 0.0094, -0.0039, 0.6858, -0.8644, 0.4243 0.0357, -0.6111, 0.6959, -0.4967, 0.4015, 0.0805, 0.6611 0.8977, 0.2487, 0.6760, -0.9841, 0.9787, -0.8446, 0.2873 -0.9821, 0.6455, 0.7224, -0.1203, -0.4885, 0.6054, 0.6908 -0.0443, -0.7313, 0.8557, 0.7919, -0.0169, 0.7134, 0.6039 -0.2040, 0.0115, -0.6209, 0.9300, -0.4116, -0.7931, 0.6495 -0.7114, -0.9718, 0.4319, 0.1290, 0.5892, 0.0142, 0.7675 0.5557, -0.1870, 0.2955, -0.6404, -0.3564, -0.6548, 0.6295 -0.1827, -0.5172, -0.1862, 0.9504, -0.3594, 0.9650, 0.5685 0.7150, 0.2392, -0.4959, 0.5857, -0.1341, -0.2850, 0.3585 -0.3394, 0.3947, -0.4627, 0.6166, -0.4094, 0.0882, 0.5962 0.7768, -0.6312, 0.1707, 0.7964, -0.1078, 0.8437, 0.4243 -0.4420, 0.2177, 0.3649, -0.5436, -0.9725, -0.1666, 0.8086 0.5595, -0.6505, -0.3161, -0.7108, 0.4335, 0.3986, 0.5846 0.3770, -0.4932, 0.3847, -0.5454, -0.1507, -0.2562, 0.6335 0.2633, 0.4146, 0.2272, 0.2966, -0.6601, -0.7011, 0.5653 0.0284, 0.7507, -0.6321, -0.0743, -0.1421, -0.0054, 0.4219 -0.4762, 0.6891, 0.6007, -0.1467, 0.2140, -0.7091, 0.6098 0.0192, -0.4061, 0.7193, 0.3432, 0.2669, -0.7505, 0.6549 0.8966, 0.2902, -0.6966, 0.2783, 0.1313, -0.0627, 0.2876 -0.1439, 0.1985, 0.6999, 0.5022, 0.1587, 0.8494, 0.3872 0.2473, -0.9040, -0.4308, -0.8779, 0.4070, 0.3369, 0.6825 -0.2428, -0.6236, 0.4940, -0.3192, 0.5906, -0.0242, 0.6770 0.2885, -0.2987, -0.5416, -0.1322, -0.2351, -0.0604, 0.6106 0.9590, -0.2712, 0.5488, 0.1055, 0.7783, -0.2901, 0.2956 -0.9129, 0.9015, 0.1128, -0.2473, 0.9901, -0.8833, 0.6500 0.0334, -0.9378, 0.1424, -0.6391, 0.2619, 0.9618, 0.7033 0.4169, 0.5549, -0.0103, 0.0571, -0.6984, -0.2612, 0.4935 -0.7156, 0.4538, -0.0460, -0.1022, 0.7720, 0.0552, 0.4983 -0.8560, -0.1637, -0.9485, -0.4177, 0.0070, 0.9319, 0.6445 -0.7812, 0.3461, -0.0001, 0.5542, -0.7128, -0.8336, 0.7720 -0.6166, 0.5356, -0.4194, -0.5662, -0.9666, -0.2027, 0.7401 -0.2378, 0.3187, -0.8582, -0.6948, -0.9668, -0.7724, 0.7670 -0.3579, 0.1158, 0.9869, 0.6690, 0.3992, 0.8365, 0.4184 -0.9205, -0.8593, -0.0520, -0.3017, 0.8745, -0.0209, 0.7723 -0.1067, 0.7541, -0.4928, -0.4524, -0.3433, 0.0951, 0.4645 -0.5597, 0.3429, -0.7144, -0.8118, 0.7404, -0.5263, 0.6117 0.0516, -0.8480, 0.7483, 0.9023, 0.6250, -0.4324, 0.5987 0.0557, -0.3212, 0.1093, 0.9488, -0.3766, 0.3376, 0.5739 -0.3484, 0.7797, 0.5034, 0.5253, -0.0610, -0.5785, 0.5365 -0.9170, -0.3563, -0.9258, 0.3877, 0.3407, -0.1391, 0.7131 -0.9203, -0.7304, -0.6132, -0.3287, -0.8954, 0.2102, 0.9329 0.0241, 0.2349, -0.1353, 0.6954, -0.0919, -0.9692, 0.5744 0.6460, 0.9036, -0.8982, -0.5299, -0.8733, -0.1567, 0.4425 0.7277, -0.8368, -0.0538, -0.7489, 0.5458, 0.6828, 0.5848 -0.5212, 0.9049, 0.8878, 0.2279, 0.9470, -0.3103, 0.4255 0.7957, -0.1308, -0.5284, 0.8817, 0.3684, -0.8702, 0.3969 0.2099, 0.4647, -0.4931, 0.2010, 0.6292, -0.8918, 0.4620 -0.7390, 0.6849, 0.2367, 0.0626, -0.5034, -0.4098, 0.7137 -0.8711, 0.7940, -0.5932, 0.6525, 0.7635, -0.0265, 0.5705 0.1969, 0.0545, 0.2496, 0.7101, -0.4357, 0.7675, 0.4242 -0.5460, 0.1920, -0.5211, -0.7372, -0.6763, 0.6897, 0.6769 0.2044, 0.9271, -0.3086, 0.1913, 0.1980, 0.2314, 0.2998 -0.6149, 0.5059, -0.9854, -0.3435, 0.8352, 0.1767, 0.4497 0.7104, 0.2093, 0.6452, 0.7590, -0.3580, -0.7541, 0.4076 -0.7465, 0.1796, -0.9279, -0.5996, 0.5766, -0.9758, 0.7713 -0.3933, -0.9572, 0.9950, 0.1641, -0.4132, 0.8579, 0.7421 0.1757, -0.4717, -0.3894, -0.2567, -0.5111, 0.1691, 0.7088 0.3917, -0.8561, 0.9422, 0.5061, 0.6123, 0.5033, 0.4824 -0.1087, 0.3449, -0.1025, 0.4086, 0.3633, 0.3943, 0.3760 0.2372, -0.6980, 0.5216, 0.5621, 0.8082, -0.5325, 0.5297 -0.3589, 0.6310, 0.2271, 0.5200, -0.1447, -0.8011, 0.5903 -0.7699, -0.2532, -0.6123, 0.6415, 0.1993, 0.3777, 0.6039 -0.5298, -0.0768, -0.6028, -0.9490, 0.4588, 0.4498, 0.6159 -0.3392, 0.6870, -0.1431, 0.7294, 0.3141, 0.1621, 0.4501 0.7889, -0.3900, 0.7419, 0.8175, -0.3403, 0.3661, 0.4087 0.7984, -0.8486, 0.7572, -0.6183, 0.6995, 0.3342, 0.5025 0.2707, 0.6956, 0.6437, 0.2565, 0.9126, 0.1798, 0.2331 -0.6043, -0.1413, -0.3265, 0.9839, -0.2395, 0.9854, 0.5444 -0.8509, -0.2594, -0.7532, 0.2690, -0.1722, 0.9818, 0.6516 0.8599, -0.7015, -0.2102, -0.0768, 0.1219, 0.5607, 0.4747 -0.4760, 0.8216, -0.9555, 0.6422, -0.6231, 0.3715, 0.5485 -0.2896, 0.9484, -0.7545, -0.6249, 0.7789, 0.1668, 0.3415 -0.5931, 0.7926, 0.7462, 0.4006, -0.0590, 0.6543, 0.4781 -0.0083, -0.2730, -0.4488, 0.8495, -0.2260, -0.0142, 0.5854 -0.2335, -0.4049, 0.4352, -0.6183, -0.7636, 0.6740, 0.7596 0.4883, 0.1810, -0.5142, 0.2465, 0.2767, -0.3449, 0.3995 -0.4922, 0.1828, -0.1424, -0.2358, -0.7466, -0.5115, 0.7968 -0.8413, -0.3943, 0.4834, 0.2300, 0.3448, -0.9832, 0.7989 -0.5382, -0.6502, -0.6300, 0.6885, 0.9652, 0.8275, 0.4353 -0.3053, 0.5604, 0.0929, 0.6329, -0.0325, 0.1799, 0.4848 0.0740, -0.2680, 0.2086, 0.9176, -0.2144, -0.2141, 0.5856 0.5813, 0.2902, -0.2122, 0.3779, -0.1920, -0.7278, 0.4079 -0.5641, 0.8515, 0.3793, 0.1976, 0.4933, 0.0839, 0.4716 0.4011, 0.8611, 0.7252, -0.6651, -0.4737, -0.8568, 0.5708 -0.5785, 0.0056, -0.7901, -0.2223, 0.0760, -0.3216, 0.7252 0.1118, 0.0735, -0.2188, 0.3925, 0.3570, 0.3746, 0.3688 0.2262, 0.8715, 0.1938, 0.9592, -0.1180, 0.4792, 0.2952 -0.9248, 0.5295, 0.0366, -0.9894, -0.4456, 0.0697, 0.7335 0.2992, 0.8629, -0.8505, -0.4464, 0.8385, 0.5300, 0.2702 0.1995, 0.6659, 0.7921, 0.9454, 0.9970, -0.7207, 0.2996 -0.3066, -0.2927, -0.4923, 0.8220, 0.4513, -0.9481, 0.6617 -0.0770, -0.4374, -0.9421, 0.7694, 0.5420, -0.3405, 0.5131 -0.3842, 0.8562, 0.9538, 0.0471, 0.9039, 0.7760, 0.3215 0.0361, -0.2545, 0.4207, -0.0887, 0.2104, 0.9808, 0.5202 -0.8220, -0.6302, 0.0537, -0.1658, 0.6013, 0.8664, 0.6598 -0.6443, 0.7201, 0.9148, 0.9189, -0.9243, -0.8848, 0.6095 -0.2880, 0.9074, -0.0461, -0.4435, 0.0060, 0.2867, 0.4025 -0.7775, 0.5161, 0.7039, 0.6885, 0.7810, -0.2363, 0.5234 -0.5484, 0.9426, -0.4308, 0.8148, 0.7811, 0.8450, 0.3479
Test data:
# synthetic_test.txt # -0.6877, 0.7594, 0.2640, -0.5787, -0.3098, -0.6802, 0.7071 -0.6694, -0.6056, 0.3821, 0.1476, 0.7466, -0.5107, 0.7282 0.2592, -0.9311, 0.0324, 0.7265, 0.9683, -0.9803, 0.5832 -0.9049, -0.9797, -0.0196, -0.9090, -0.4433, 0.2799, 0.9018 -0.4106, -0.4607, 0.1811, -0.2389, 0.4050, -0.0078, 0.6916 -0.4259, -0.7336, 0.8742, 0.6097, 0.8761, -0.6292, 0.6728 0.8663, 0.8715, -0.4329, -0.4507, 0.1029, -0.6294, 0.2936 0.8948, -0.0124, 0.9278, 0.2899, -0.0314, 0.9354, 0.3160 -0.7136, 0.2647, 0.3238, -0.1323, -0.8813, -0.0146, 0.8133 -0.4867, -0.2171, -0.5197, 0.3729, 0.9798, -0.6451, 0.5820 0.6429, -0.5380, -0.8840, -0.7224, 0.8703, 0.7771, 0.5777 0.6999, -0.1307, -0.0639, 0.2597, -0.6839, -0.9704, 0.5796 -0.4690, -0.9691, 0.3490, 0.1029, -0.3567, 0.5604, 0.8151 -0.4154, -0.6081, -0.8241, 0.7400, -0.8236, 0.3674, 0.7881 -0.7592, -0.9786, 0.1145, 0.8142, 0.7209, -0.3231, 0.6968 0.3393, 0.6156, 0.7950, -0.0923, 0.1157, 0.0123, 0.3229 0.3840, 0.3658, 0.0406, 0.6569, 0.0116, 0.6497, 0.2879 0.9397, 0.4839, -0.4804, 0.1625, 0.9105, -0.8385, 0.2410 -0.8329, 0.2383, -0.5510, 0.5304, 0.1363, 0.3324, 0.5862 -0.8255, -0.2579, 0.3443, -0.6208, 0.7915, 0.8997, 0.6109 0.9231, 0.4602, -0.1874, 0.4875, -0.4240, -0.3712, 0.3165 0.7573, -0.4908, 0.5324, 0.8820, -0.9979, -0.0478, 0.6093 0.3141, 0.6866, -0.6325, 0.7123, -0.2713, 0.7845, 0.3050 -0.1647, -0.6616, 0.2998, -0.9260, -0.3768, -0.3530, 0.8315 0.2149, 0.3017, 0.6921, 0.8552, 0.3209, 0.1563, 0.3157 -0.6918, 0.7902, -0.3780, 0.0970, 0.3641, -0.5271, 0.6323 -0.6645, 0.0170, 0.5837, 0.3848, -0.7621, 0.8015, 0.7440 0.1069, -0.8304, -0.5951, 0.7085, 0.4119, 0.7899, 0.4998 -0.3417, 0.0560, 0.3008, 0.1886, -0.5371, -0.1464, 0.7339 0.9734, -0.8669, 0.4279, -0.3398, 0.2509, -0.4837, 0.4665 0.3020, -0.2577, -0.4104, 0.8235, 0.8850, 0.2271, 0.3066 -0.5766, 0.6603, -0.5198, 0.2632, 0.4215, 0.4848, 0.4478 -0.2195, 0.5197, 0.8059, 0.1748, -0.8192, -0.7420, 0.6740 -0.9212, -0.5169, 0.7581, 0.9470, 0.2108, 0.9525, 0.6180 -0.9131, 0.8971, -0.3774, 0.5979, 0.6213, 0.7200, 0.4642 -0.4842, 0.8689, 0.2382, 0.9709, -0.9347, 0.4503, 0.5662 0.1311, -0.0152, -0.4816, -0.3463, -0.5011, -0.5615, 0.6979 -0.8336, 0.5540, 0.0673, 0.4788, 0.0308, -0.2001, 0.6917 0.9725, -0.9435, 0.8655, 0.8617, -0.2182, -0.5711, 0.6021 0.6064, -0.4921, -0.4184, 0.8318, 0.8058, 0.0708, 0.3221
.NET Test Automation Recipes
Software Testing
SciPy Programming Succinctly
Keras Succinctly
R Programming
2026 Visual Studio Live
2025 Summer MLADS Conference
2026 DevIntersection Conference
2025 Machine Learning Week
2025 Ai4 Conference
2026 G2E Conference
2026 iSC West Conference
You must be logged in to post a comment.