A simple decision tree for regression will almost always overfit the training data where the model accuracy on the training data is near 100% but accuracy on new, previously unseen data is very poor.

The simple idea of bagging tree regression (“bootstrap aggregation”) is to create a collection/ensemble of many simple decision trees, where each tree is trained on a different subset of the rows of the training data.

For example, if the source set of training data has 200 rows, you could create 50 simple trees, where each tree is trained on 180 randomly selected rows. The selection is done “with replacement” so for a random subset of training data, some rows will be used more than once and some rows won’t be used at all.

After all the trees are trained, a prediction for an input vector x is just the average of the predictions of the trees in the collection. A very simple and crude idea that sometimes works well, and sometimes doesn’t work very well.

Bagging tree regression is a specific kind of random forest regression. In bagging tree regression, the subsets of the training data use all columns. In random forest regression, a random subset of columns is used.

Even though bagging tree regression is a type of random forest regression, the two techniques are usually considered distinct.

Bagging tree regression was first described in 1994 paper by a fellow named Leo Breiman, even though the idea had been used for at least a decade before. Then in 1995, a different researcher published a paper that described the math foundations for a slightly modified bagging tree regression (randomly select columns) and called the new technique random forest regression, (even though the idea had been used for years before.)

So, even though bagging tree regression is really just a special case of random forest regression, the term bagging tree regression had been in use for several years so there was no getting rid of that term.

Bagging tree regression evolved into random forest regression, which evolved into AdaBoost.R and AdaBoost.R2 regression, which evolved into gradient boosting regression — a fascinating (but very long) story. Maybe I’ll post that story some day.

I used AI to optimize my bagging tree regression system. AI was remarkably accurate. I made a demo. The demo data is synthetic. It was generated by a 5-10-1 neural network with random weight and bias values. The data looks like:

-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
. . .

The first five values on each line are predictors. The last value on each line is the target y value to predict. There are 200 training items and 40 test items. Here’s the output of a demo:

JavaScript bagging tree regression

Loading synthetic train (200), test (40)
Done

First three train x:
-0.1660   0.4406  -0.9998  -0.3953  -0.7065
 0.0776  -0.1616   0.3704  -0.5911   0.7562
-0.9452   0.3409  -0.1654   0.1174  -0.7192

First three train y:
0.484 0.1568 0.8054

Setting numTrees = 100

Setting maxDepth = 6
Setting minSamples = 2
Setting minLeaf = 1
Setting nRows = 150

Creating and training bagging tree regressor model
Done

Evaluating model

Train acc (within 0.10) = 0.8000
Test acc (within 0.10) = 0.6000

Train MSE = 0.0005
Test MSE = 0.0019

Predicting for x =
-0.1660   0.4406  -0.9998  -0.3953  -0.7065
y = 0.4847

End bagging tree regression demo

Notice that the model still overfits quite a bit. This is because the synthetic data is more or less spread evenly. Because of this weakness, the more powerful gradient boosting regression model is often used instead of bagging tree regression or random forest regression. That said however, for some problem scenarios, bagging tree and random forest regression work quite well.

Machine learning regression algorithms find hidden patterns in data.

Right: June 1983. The bunny logo is disguised as part of the model’s clothing near her sternum area. Circled in green.

Demo program. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols (my blog editor often chokes on symbols).

// bagging_tree_regression.js
// node.js v22.16.0
// verified by OpenAI GPT-x

const FS = require('fs');

// ==========================================================

class BaggingTreeRegressor
{
  constructor(nTrees, nRows, maxDepth, minSamples,
    minLeaf, numSplitCols, seed = 0)
  {
    this.nTrees = nTrees;
    this.nRows = nRows;  // num train rows, each tree
    this.maxDepth = maxDepth;
    this.minSamples = minSamples;
    this.minLeaf = minLeaf;
    this.numSplitCols = numSplitCols;

    this.trees = [];

    this.seed = seed + 0.5; // avoid 0
  }

  // --------------------------------------------------------
  // primary: ctor(), train(), predict(), accuracy(), mse()
  // helpers: next(), nextInt()
  // --------------------------------------------------------

  train(trainX, trainY)
  {
    let totalRows = trainX.length;
    let nCols = trainX[0].length;

    // reusable buffers for all trees
    let subsetX = new Array(this.nRows);
    for (let i = 0; i "lt" this.nRows; ++i)
      subsetX[i] = new Array(nCols);

    let subsetY = new Array(this.nRows);

    // train each tree using buffers
    for (let t = 0; t "lt" this.nTrees; ++t) {

      // populate the buffers
      for (let r = 0; r "lt" this.nRows; ++r) {
        let rndIndex = this.nextInt(0, totalRows);

        // copy the row into the reusable buffer
        // slice() creates a copy of the source row.
        for (let c = 0; c "lt" nCols; ++c)
          subsetX[r][c] = trainX[rndIndex][c];

        subsetY[r] = trainY[rndIndex];
      }

      let dtr = new DecisionTreeRegressor(
        this.maxDepth,
        this.minSamples,
        this.minLeaf,
        this.numSplitCols,
        false, // don't save rows info
        this.nextInt(0, 1000000)
      );

      dtr.train(subsetX, subsetY); // safe to reuse

      this.trees.push(dtr);
    }
  } // train()

  // --------------------------------------------------------

  predict(x)
  {
    let sum = 0.0;

    for (let t = 0; t "lt" this.nTrees; ++t)
      sum += this.trees[t].predict(x);

    return sum / this.nTrees;
  }

  // --------------------------------------------------------

  accuracy(dataX, dataY, pctClose)
  {
    let numCorrect = 0;
    let numWrong = 0;

    for (let i = 0; i "lt" dataX.length; ++i) {
      let actualY = dataY[i];
      let predY = this.predict(dataX[i]);

      if (Math.abs(predY - actualY) "lt"
        (pctClose * Math.abs(actualY)))
        ++numCorrect;
      else
        ++numWrong;
    }

    return numCorrect / (numWrong + numCorrect);
  }

  // --------------------------------------------------------

  mse(dataX, dataY)
  {
    let n = dataX.length;
    let sum = 0.0;

    for (let i = 0; i "lt" n; ++i) {
      let actualY = dataY[i];
      let predY = this.predict(dataX[i]);

      sum += (actualY - predY) * (actualY - predY);
    }

    return sum / n;
  }

  // --------------------------------------------------------
  // helpers
  // --------------------------------------------------------

  next()
  {
    let x = Math.sin(this.seed++) * 10000;
    return x - Math.floor(x);
  }

  // --------------------------------------------------------

  nextInt(lo, hi)
  {
    return Math.trunc((hi - lo) * this.next() + lo);
  }

} // class BaggingTreeRegressor

// ==========================================================
// ==========================================================

class DecisionTreeRegressor
{
  // internal Node factory
  createNode()
  {
    // return an object literal
    return {
      id: -1,
      colIdx: -1,
      thresh: 0.0,
      left: -1,
      right: -1,
      value: 0.0,
      isLeaf: false,
      rows: null,
    };
  }

  // --------------------------------------------------------
  // primary: ctor(), train(), predict()
  // helpers: bestSplit(), treeTargetMean(), next(),
  //  nextInt(), sortParallel()
  // -------------------------------------------------------

  constructor(maxDepth, minSamples, minLeaf, numSplitCols,
    saveRows, seed)
  {
    this.maxDepth = maxDepth;
    this.minSamples = minSamples;
    this.minLeaf = minLeaf;
    this.numSplitCols = numSplitCols;
    this.saveRows = saveRows;

    let numNodes = Math.pow(2, maxDepth + 1) - 1;
    this.tree = new Array(numNodes).fill(null);

    this.seed = seed + 0.5; // avoid 0
    this.trainX = null;
    this.trainY = null;
  }

  // --------------------------------------------------------

  train(trainX, trainY)
  {
    this.trainX = trainX;
    this.trainY = trainY;

    let maxParentID = Math.pow(2, this.maxDepth) - 2;
    
    let allRows = new Array(this.trainX.length);
    for (let i = 0; i "lt" this.trainX.length; ++i)
      allRows[i] = i;

    let root = this.createNode();
    root.id = 0;
    root.value = this.treeTargetMean(allRows);
    root.rows = allRows;
    this.tree[0] = root;

    for (let i = 0; i "lt" this.tree.length; ++i) {
      let currNode = this.tree[i];
      if (currNode == null) continue;

      let leftID = currNode.id * 2 + 1;
      let rightID = currNode.id * 2 + 2;

      if (currNode.id "gt" maxParentID || 
        currNode.rows.length "lt" this.minSamples ||
        rightID "gte" this.tree.length) {
        currNode.isLeaf = true;
        continue;
      }

      let splitInfo = this.bestSplit(currNode.rows);
      let colIdx = splitInfo[0];
      let splitVal = splitInfo[1];

      // split failure
      if (colIdx == -1) {
        currNode.isLeaf = true;
        continue;
      }

      // split success
      currNode.colIdx = colIdx;
      currNode.thresh = splitVal;

      let leftIdxs = [];
      let rightIdxs = [];

      for (let k = 0; k "lt" currNode.rows.length; ++k) {
        let r = currNode.rows[k];
        if (this.trainX[r][colIdx] "lte" splitVal)
          leftIdxs.push(r);
        else
          rightIdxs.push(r);
      }

      // create and attach children
      currNode.left = leftID;
      let n1 = this.createNode();
      n1.id = leftID;
      n1.rows = leftIdxs;
      n1.value = this.treeTargetMean(leftIdxs);
      this.tree[leftID] = n1;

      currNode.right = rightID;
      let n2 = this.createNode();
      n2.id = rightID;
      n2.rows = rightIdxs;
      n2.value = this.treeTargetMean(rightIdxs);
      this.tree[rightID] = n2;
    } // main loop

    // delete row info to save memory
    if (this.saveRows == false) {
      for (let i = 0; i "lt" this.tree.length; ++i) {
        if (this.tree[i] != null)
          this.tree[i].rows = null;
      }
    }
  } // train()

  // --------------------------------------------------------

  predict(x)
  {
    let p = 0;
    let lastValidValue = 0.0;

    while (p != -1 && p "lt" this.tree.length) {
      let currNode = this.tree[p];
      if (currNode == null) break;

      lastValidValue = currNode.value;

      if (
        currNode.isLeaf ||
        currNode.colIdx == -1 ||
        currNode.left "gte" this.tree.length ||
        currNode.right "gte" this.tree.length
      )
        break;

      if (x[currNode.colIdx] "lte" currNode.thresh)
        p = currNode.left;
      else
        p = currNode.right;
    }
    return lastValidValue;
  }

  // --------------------------------------------------------
  // helpers
  // --------------------------------------------------------

  bestSplit(rows)
  {
    let bestColIdx = -1;
    let bestThresh = 0.0;
    let bestGain = -1.0;
    let nRows = rows.length;
    let nCols = this.trainX[0].length;

    if (nRows === 0)
      throw new Error("Empty data in bestSplit()");

    let colIndices = new Array(nCols);
    for (let k = 0; k "lt" nCols; ++k)
      colIndices[k] = k;

    // Fisher-Yates column selection
    for (let i = 0; i "lt" nCols; ++i) {
      let ri = this.nextInt(i, nCols);
      let tmp = colIndices[i];
      colIndices[i] = colIndices[ri];
      colIndices[ri] = tmp;
    }

    let numColsToUse;
    if (this.numSplitCols === -1)
      numColsToUse = nCols;
    else
      numColsToUse = Math.min(this.numSplitCols, nCols);

    let sortedRows = new Array(nRows);
    let featureKeys = new Array(nRows);

    let totalSum = 0.0;
    for (let i = 0; i "lt" nRows; ++i)
      totalSum += this.trainY[rows[i]];

    for (let j = 0; j "lt" numColsToUse; ++j) {
      let colIdx =colIndices[j];
      for (let i = 0; i "lt" nRows; ++i) {
        let r = rows[i];
        sortedRows[i] = r;
        featureKeys[i] = this.trainX[r][colIdx];
      } // i

      // sort featureKeys, sortedRows in parallel
      // based on order of featureKeys
      this.sortParallel(featureKeys, sortedRows);
      
      let leftSum = 0.0;
      for (let i = 0; i "lt" nRows-1; ++i) {
        leftSum += this.trainY[sortedRows[i]];
        let leftCount = i + 1;
        let rightCount = nRows - leftCount;

        if (featureKeys[i] == featureKeys[i+1])
          continue;  // duplicate

        if (leftCount "lt" this.minLeaf ||
          rightCount "lt" this.minLeaf)
          continue;

        let rightSum = totalSum - leftSum;

        let gain = (leftSum * leftSum / leftCount) +
          (rightSum * rightSum / rightCount);

        if (gain "gt" bestGain) {
          bestGain = gain;
          bestColIdx = colIdx;
          bestThresh = featureKeys[i] + 
            (featureKeys[i+1] - featureKeys[i]) / 2.0;
        }
      } // i
    } // j

    return [bestColIdx, bestThresh];
  } // bestSplit()

  // --------------------------------------------------------

  next()
  {
    let x = Math.sin(this.seed++) * 10000; // avoid stuck 0
    return x - Math.floor(x);
  }

  // -------------------------------------------------------

  nextInt(lo, hi)
  {
    return Math.trunc((hi - lo) * this.next() + lo);
  }

  // --------------------------------------------------------

  treeTargetMean(rows)
  {
    let sum = 0.0;
    for (let i = 0; i "lt" rows.length; ++i) {
      sum += this.trainY[rows[i]];
    }
    return sum / rows.length;
  }

  // --------------------------------------------------------

  sortParallel(keys, values)
  {
    // sort keys and values in-place based on order of keys
    let n = keys.length;
    // let idxs = this.argsort(keys);

    let augmented = [];
    for (let i = 0; i "lt" n; ++i) {
      augmented[i] = [keys[i], i]; // add indices to values
    }

    augmented.sort(function(a, b){return a[0] - b[0]});

    let idxs = [];
    for (let i = 0; i "lt" n; ++i) {
      idxs[i] = augmented[i][1];  // fetch the indexes
    }

    let sortedKeys = new Array(n);
    let sortedValues = new Array(n);
    for (let i = 0; i "lt" n; ++i) {
      let idx = idxs[i]; 
      sortedKeys[i] = keys[idx];
      sortedValues[i] = values[idx];
    }
    for (let i = 0; i "lt" n; ++i) {
      keys[i] = sortedKeys[i];
      values[i] = sortedValues[i];
    }

  } // sortParallel()

} // class DecisionTreeRegressor

// ==========================================================
// ==========================================================

function loadTxt(fn, delimit, usecols, comment)
{
  let all = FS.readFileSync(fn, "utf8").trim();
  let lines = all.split("\n");

  let nRows = 0;
  for (let i = 0; i "lt" lines.length; ++i) {
    if (!lines[i].startsWith(comment)) ++nRows;
  }

  let nCols = usecols.length;
  // let result = Array.from({ length: nRows },
  //   () ="gt" new Array(nCols));

  let result = [];
  for (let i = 0; i "lt" nRows; ++i) {
    result[i] = [];
    for (let j = 0; j "lt" nCols; ++j) {
      result[i][j] = 0.0;
    }
  }

  let r = 0; let i = 0;
  while (r "lt" lines.length) {
    if (lines[r].startsWith(comment)) {
      ++r;
    } 
    else {
      let tokens = lines[r].split(delimit);
      for (let j = 0; j "lt" nCols; ++j) {
        result[i][j] = parseFloat(tokens[usecols[j]]);
      }
      ++r;
      ++i;
    }
  }

  if (usecols.length === 1) {
    let vec = new Array(nRows);
    for (let k = 0; k "lt" nRows; ++k)
      vec[k] = result[k][0];
    return vec;
  }

  return result;
}

// ----------------------------------------------------------

function vecShow(v, dec, len)
{
  for (let i = 0; i "lt" v.length; ++i) {
    if (i != 0 && i % len == 0) {
      process.stdout.write("\n");
    }
    if (v[i] "gte" 0.0) {
      process.stdout.write(" ");  // + or - space
    }
    process.stdout.write(v[i].toFixed(dec));
    process.stdout.write("  ");
  }
  process.stdout.write("\n");
}

// ----------------------------------------------------------

function main()
{
  console.log("\nJavaScript bagging tree regression ");

  // 1. load data
  console.log("\nLoading synthetic train (200), test (40)");
  let trainFile = ".\\Data\\synthetic_train_200.txt";
  let testFile = ".\\Data\\synthetic_test_40.txt";

  let trainX = loadTxt(trainFile, ",", [0,1,2,3,4], "#");
  let trainY = loadTxt(trainFile, ",", [5], "#");
  let testX = loadTxt(testFile, ",", [0,1,2,3,4], "#");
  let testY = loadTxt(testFile, ",", [5], "#");
  console.log("Done ");

  console.log("\nFirst three train x: ");
  for (let i = 0; i "lt" 3; ++i)
    vecShow(trainX[i], 4, 8);  // 4 decimals 8 wide
  console.log("\nFirst three train y: ");
  for (let i = 0; i "lt" 3; ++i)
    process.stdout.write(trainY[i].toString() + " ");
  console.log("");

  // 2. create and train model
  let numTrees = 100;
  let maxDepth = 6;
  let minSamples = 2;
  let minLeaf = 1;
  let numSplitCols = -1;  // use all columns
  let nRows = 150;
  let seed = 0;

  console.log("\nSetting numTrees = " +
    numTrees.toString());
  console.log("\nSetting maxDepth = " +
    maxDepth.toString());
  console.log("Setting minSamples = " +
    minSamples.toString());
  console.log("Setting minLeaf = " +
    minLeaf.toString());
  console.log("Setting nRows = " +
    nRows.toString());
  // console.log("Using default numSplitCols = -1 (all)");

  console.log("\nCreating and training bagging " +
    "tree regressor model ");
  let btr = 
    new BaggingTreeRegressor(numTrees, nRows, maxDepth,
      minSamples, minLeaf, numSplitCols, seed);

  btr.train(trainX, trainY);
  console.log("Done ");

  // 3. evaluate model
  console.log("\nEvaluating model ");
  let trainAcc = btr.accuracy(trainX, trainY, 0.10);
  let testAcc = btr.accuracy(testX, testY, 0.10);
  console.log("\nTrain acc (within 0.10) = " +
    trainAcc.toFixed(4).toString());
  console.log("Test acc (within 0.10) = " +
    testAcc.toFixed(4).toString());

  let trainMSE = btr.mse(trainX, trainY);
  let testMSE = btr.mse(testX, testY);
  console.log("\nTrain MSE = " +
    trainMSE.toFixed(4).toString());
  console.log("Test MSE = " +
    testMSE.toFixed(4).toString());

  // 4. use tree to make prediction
  let x = trainX[0];
  console.log("\nPredicting for x = ");
  vecShow(x, 4, 9);
  let predY = btr.predict(x);
  console.log("y = " + predY.toFixed(4).toString());

  console.log("\nEnd bagging tree regression demo ");
}

main();

Training data:

# synthetic_train_200.txt
#
-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
-0.8299, -0.9219, -0.6603,  0.7563, -0.8033,  0.7955
 0.0663,  0.3838, -0.3690,  0.3730,  0.6693,  0.3206
-0.9634,  0.5003,  0.9777,  0.4963, -0.4391,  0.7377
-0.1042,  0.8172, -0.4128, -0.4244, -0.7399,  0.4801
-0.9613,  0.3577, -0.5767, -0.4689, -0.0169,  0.6861
-0.7065,  0.1786,  0.3995, -0.7953, -0.1719,  0.5569
 0.3888, -0.1716, -0.9001,  0.0718,  0.3276,  0.2500
 0.1731,  0.8068, -0.7251, -0.7214,  0.6148,  0.3297
-0.2046, -0.6693,  0.8550, -0.3045,  0.5016,  0.2129
 0.2473,  0.5019, -0.3022, -0.4601,  0.7918,  0.2613
-0.1438,  0.9297,  0.3269,  0.2434, -0.7705,  0.5171
 0.1568, -0.1837, -0.5259,  0.8068,  0.1474,  0.3307
-0.9943,  0.2343, -0.3467,  0.0541,  0.7719,  0.5581
 0.2467, -0.9684,  0.8589,  0.3818,  0.9946,  0.1092
-0.6553, -0.7257,  0.8652,  0.3936, -0.8680,  0.7018
 0.8460,  0.4230, -0.7515, -0.9602, -0.9476,  0.1996
-0.9434, -0.5076,  0.7201,  0.0777,  0.1056,  0.5664
 0.9392,  0.1221, -0.9627,  0.6013, -0.5341,  0.1533
 0.6142, -0.2243,  0.7271,  0.4942,  0.1125,  0.1661
 0.4260,  0.1194, -0.9749, -0.8561,  0.9346,  0.2230
 0.1362, -0.5934, -0.4953,  0.4877, -0.6091,  0.3810
 0.6937, -0.5203, -0.0125,  0.2399,  0.6580,  0.1460
-0.6864, -0.9628, -0.8600, -0.0273,  0.2127,  0.5387
 0.9772,  0.1595, -0.2397,  0.1019,  0.4907,  0.1611
 0.3385, -0.4702, -0.8673, -0.2598,  0.2594,  0.2270
-0.8669, -0.4794,  0.6095, -0.6131,  0.2789,  0.4700
 0.0493,  0.8496, -0.4734, -0.8681,  0.4701,  0.3516
 0.8639, -0.9721, -0.5313,  0.2336,  0.8980,  0.1412
 0.9004,  0.1133,  0.8312,  0.2831, -0.2200,  0.1782
 0.0991,  0.8524,  0.8375, -0.2102,  0.9265,  0.2150
-0.6521, -0.7473, -0.7298,  0.0113, -0.9570,  0.7422
 0.6190, -0.3105,  0.8802,  0.1640,  0.7577,  0.1056
 0.6895,  0.8108, -0.0802,  0.0927,  0.5972,  0.2214
 0.1982, -0.9689,  0.1870, -0.1326,  0.6147,  0.1310
-0.3695,  0.7858,  0.1557, -0.6320,  0.5759,  0.3773
-0.1596,  0.3581,  0.8372, -0.9992,  0.9535,  0.2071
-0.2468,  0.9476,  0.2094,  0.6577,  0.1494,  0.4132
 0.1737,  0.5000,  0.7166,  0.5102,  0.3961,  0.2611
 0.7290, -0.3546,  0.3416, -0.0983, -0.2358,  0.1332
-0.3652,  0.2438, -0.1395,  0.9476,  0.3556,  0.4170
-0.6029, -0.1466, -0.3133,  0.5953,  0.7600,  0.4334
-0.4596, -0.4953,  0.7098,  0.0554,  0.6043,  0.2775
 0.1450,  0.4663,  0.0380,  0.5418,  0.1377,  0.2931
-0.8636, -0.2442, -0.8407,  0.9656, -0.6368,  0.7429
 0.6237,  0.7499,  0.3768,  0.1390, -0.6781,  0.2185
-0.5499,  0.1850, -0.3755,  0.8326,  0.8193,  0.4399
-0.4858, -0.7782, -0.6141, -0.0008,  0.4572,  0.4197
 0.7033, -0.1683,  0.2334, -0.5327, -0.7961,  0.1776
 0.0317, -0.0457, -0.6947,  0.2436,  0.0880,  0.3345
 0.5031, -0.5559,  0.0387,  0.5706, -0.9553,  0.3107
-0.3513,  0.7458,  0.6894,  0.0769,  0.7332,  0.3170
 0.2205,  0.5992, -0.9309,  0.5405,  0.4635,  0.3532
-0.4806, -0.4859,  0.2646, -0.3094,  0.5932,  0.3202
 0.9809, -0.3995, -0.7140,  0.8026,  0.0831,  0.1600
 0.9495,  0.2732,  0.9878,  0.0921,  0.0529,  0.1289
-0.9476, -0.6792,  0.4913, -0.9392, -0.2669,  0.5966
 0.7247,  0.3854,  0.3819, -0.6227, -0.1162,  0.1550
-0.5922, -0.5045, -0.4757,  0.5003, -0.0860,  0.5863
-0.8861,  0.0170, -0.5761,  0.5972, -0.4053,  0.7301
 0.6877, -0.2380,  0.4997,  0.0223,  0.0819,  0.1404
 0.9189,  0.6079, -0.9354,  0.4188, -0.0700,  0.1907
-0.1428, -0.7820,  0.2676,  0.6059,  0.3936,  0.2790
 0.5324, -0.3151,  0.6917, -0.1425,  0.6480,  0.1071
-0.8432, -0.9633, -0.8666, -0.0828, -0.7733,  0.7784
-0.9444,  0.5097, -0.2103,  0.4939, -0.0952,  0.6787
-0.0520,  0.6063, -0.1952,  0.8094, -0.9259,  0.4836
 0.5477, -0.7487,  0.2370, -0.9793,  0.0773,  0.1241
 0.2450,  0.8116,  0.9799,  0.4222,  0.4636,  0.2355
 0.8186, -0.1983, -0.5003, -0.6531, -0.7611,  0.1511
-0.4714,  0.6382, -0.3788,  0.9648, -0.4667,  0.5950
 0.0673, -0.3711,  0.8215, -0.2669, -0.1328,  0.2677
-0.9381,  0.4338,  0.7820, -0.9454,  0.0441,  0.5518
-0.3480,  0.7190,  0.1170,  0.3805, -0.0943,  0.4724
-0.9813,  0.1535, -0.3771,  0.0345,  0.8328,  0.5438
-0.1471, -0.5052, -0.2574,  0.8637,  0.8737,  0.3042
-0.5454, -0.3712, -0.6505,  0.2142, -0.1728,  0.5783
 0.6327, -0.6297,  0.4038, -0.5193,  0.1484,  0.1153
-0.5424,  0.3282, -0.0055,  0.0380, -0.6506,  0.6613
 0.1414,  0.9935,  0.6337,  0.1887,  0.9520,  0.2540
-0.9351, -0.8128, -0.8693, -0.0965, -0.2491,  0.7353
 0.9507, -0.6640,  0.9456,  0.5349,  0.6485,  0.1059
-0.0462, -0.9737, -0.2940, -0.0159,  0.4602,  0.2606
-0.0627, -0.0852, -0.7247, -0.9782,  0.5166,  0.2977
 0.0478,  0.5098, -0.0723, -0.7504, -0.3750,  0.3335
 0.0090,  0.3477,  0.5403, -0.7393, -0.9542,  0.4415
-0.9748,  0.3449,  0.3736, -0.1015,  0.8296,  0.4358
 0.2887, -0.9895, -0.0311,  0.7186,  0.6608,  0.2057
 0.1570, -0.4518,  0.1211,  0.3435, -0.2951,  0.3244
 0.7117, -0.6099,  0.4946, -0.4208,  0.5476,  0.1096
-0.2929, -0.5726,  0.5346, -0.3827,  0.4665,  0.2465
 0.4889, -0.5572, -0.5718, -0.6021, -0.7150,  0.2163
-0.7782,  0.3491,  0.5996, -0.8389, -0.5366,  0.6516
-0.5847,  0.8347,  0.4226,  0.1078, -0.3910,  0.6134
 0.8469,  0.4121, -0.0439, -0.7476,  0.9521,  0.1571
-0.6803, -0.5948, -0.1376, -0.1916, -0.7065,  0.7156
 0.2878,  0.5086, -0.5785,  0.2019,  0.4979,  0.2980
 0.2764,  0.1943, -0.4090,  0.4632,  0.8906,  0.2960
-0.8877,  0.6705, -0.6155, -0.2098, -0.3998,  0.7107
-0.8398,  0.8093, -0.2597,  0.0614, -0.0118,  0.6502
-0.8476,  0.0158, -0.4769, -0.2859, -0.7839,  0.7715
 0.5751, -0.7868,  0.9714, -0.6457,  0.1448,  0.1175
 0.4802, -0.7001,  0.1022, -0.5668,  0.5184,  0.1090
 0.4458, -0.6469,  0.7239, -0.9604,  0.7205,  0.0779
 0.5175,  0.4339,  0.9747, -0.4438, -0.9924,  0.2879
 0.8678,  0.7158,  0.4577,  0.0334,  0.4139,  0.1678
 0.5406,  0.5012,  0.2264, -0.1963,  0.3946,  0.2088
-0.9938,  0.5498,  0.7928, -0.5214, -0.7585,  0.7687
 0.7661,  0.0863, -0.4266, -0.7233, -0.4197,  0.1466
 0.2277, -0.3517, -0.0853, -0.1118,  0.6563,  0.1767
 0.3499, -0.5570, -0.0655, -0.3705,  0.2537,  0.1632
 0.7547, -0.1046,  0.5689, -0.0861,  0.3125,  0.1257
 0.8186,  0.2110,  0.5335,  0.0094, -0.0039,  0.1391
 0.6858, -0.8644,  0.1465,  0.8855,  0.0357,  0.1845
-0.4967,  0.4015,  0.0805,  0.8977,  0.2487,  0.4663
 0.6760, -0.9841,  0.9787, -0.8446, -0.3557,  0.1509
-0.1203, -0.4885,  0.6054, -0.0443, -0.7313,  0.4854
 0.8557,  0.7919, -0.0169,  0.7134, -0.1628,  0.2002
 0.0115, -0.6209,  0.9300, -0.4116, -0.7931,  0.4052
-0.7114, -0.9718,  0.4319,  0.1290,  0.5892,  0.3661
 0.3915,  0.5557, -0.1870,  0.2955, -0.6404,  0.2954
-0.3564, -0.6548, -0.1827, -0.5172, -0.1862,  0.4622
 0.2392, -0.4959,  0.5857, -0.1341, -0.2850,  0.2470
-0.3394,  0.3947, -0.4627,  0.6166, -0.4094,  0.5325
 0.7107,  0.7768, -0.6312,  0.1707,  0.7964,  0.2757
-0.1078,  0.8437, -0.4420,  0.2177,  0.3649,  0.4028
-0.3139,  0.5595, -0.6505, -0.3161, -0.7108,  0.5546
 0.4335,  0.3986,  0.3770, -0.4932,  0.3847,  0.1810
-0.2562, -0.2894, -0.8847,  0.2633,  0.4146,  0.4036
 0.2272,  0.2966, -0.6601, -0.7011,  0.0284,  0.2778
-0.0743, -0.1421, -0.0054, -0.6770, -0.3151,  0.3597
-0.4762,  0.6891,  0.6007, -0.1467,  0.2140,  0.4266
-0.4061,  0.7193,  0.3432,  0.2669, -0.7505,  0.6147
-0.0588,  0.9731,  0.8966,  0.2902, -0.6966,  0.4955
-0.0627, -0.1439,  0.1985,  0.6999,  0.5022,  0.3077
 0.1587,  0.8494, -0.8705,  0.9827, -0.8940,  0.4263
-0.7850,  0.2473, -0.9040, -0.4308, -0.8779,  0.7199
 0.4070,  0.3369, -0.2428, -0.6236,  0.4940,  0.2215
-0.0242,  0.0513, -0.9430,  0.2885, -0.2987,  0.3947
-0.5416, -0.1322, -0.2351, -0.0604,  0.9590,  0.3683
 0.1055,  0.7783, -0.2901, -0.5090,  0.8220,  0.2984
-0.9129,  0.9015,  0.1128, -0.2473,  0.9901,  0.4776
-0.9378,  0.1424, -0.6391,  0.2619,  0.9618,  0.5368
 0.7498, -0.0963,  0.4169,  0.5549, -0.0103,  0.1614
-0.2612, -0.7156,  0.4538, -0.0460, -0.1022,  0.3717
 0.7720,  0.0552, -0.1818, -0.4622, -0.8560,  0.1685
-0.4177,  0.0070,  0.9319, -0.7812,  0.3461,  0.3052
-0.0001,  0.5542, -0.7128, -0.8336, -0.2016,  0.3803
 0.5356, -0.4194, -0.5662, -0.9666, -0.2027,  0.1776
-0.2378,  0.3187, -0.8582, -0.6948, -0.9668,  0.5474
-0.1947, -0.3579,  0.1158,  0.9869,  0.6690,  0.2992
 0.3992,  0.8365, -0.9205, -0.8593, -0.0520,  0.3154
-0.0209,  0.0793,  0.7905, -0.1067,  0.7541,  0.1864
-0.4928, -0.4524, -0.3433,  0.0951, -0.5597,  0.6261
-0.8118,  0.7404, -0.5263, -0.2280,  0.1431,  0.6349
 0.0516, -0.8480,  0.7483,  0.9023,  0.6250,  0.1959
-0.3212,  0.1093,  0.9488, -0.3766,  0.3376,  0.2735
-0.3481,  0.5490, -0.3484,  0.7797,  0.5034,  0.4379
-0.5785, -0.9170, -0.3563, -0.9258,  0.3877,  0.4121
 0.3407, -0.1391,  0.5356,  0.0720, -0.9203,  0.3458
-0.3287, -0.8954,  0.2102,  0.0241,  0.2349,  0.3247
-0.1353,  0.6954, -0.0919, -0.9692,  0.7461,  0.3338
 0.9036, -0.8982, -0.5299, -0.8733, -0.1567,  0.1187
 0.7277, -0.8368, -0.0538, -0.7489,  0.5458,  0.0830
 0.9049,  0.8878,  0.2279,  0.9470, -0.3103,  0.2194
 0.7957, -0.1308, -0.5284,  0.8817,  0.3684,  0.2172
 0.4647, -0.4931,  0.2010,  0.6292, -0.8918,  0.3371
-0.7390,  0.6849,  0.2367,  0.0626, -0.5034,  0.7039
-0.1567, -0.8711,  0.7940, -0.5932,  0.6525,  0.1710
 0.7635, -0.0265,  0.1969,  0.0545,  0.2496,  0.1445
 0.7675,  0.1354, -0.7698, -0.5460,  0.1920,  0.1728
-0.5211, -0.7372, -0.6763,  0.6897,  0.2044,  0.5217
 0.1913,  0.1980,  0.2314, -0.8816,  0.5006,  0.1998
 0.8964,  0.0694, -0.6149,  0.5059, -0.9854,  0.1825
 0.1767,  0.7104,  0.2093,  0.6452,  0.7590,  0.2832
-0.3580, -0.7541,  0.4426, -0.1193, -0.7465,  0.5657
-0.5996,  0.5766, -0.9758, -0.3933, -0.9572,  0.6800
 0.9950,  0.1641, -0.4132,  0.8579,  0.0142,  0.2003
-0.4717, -0.3894, -0.2567, -0.5111,  0.1691,  0.4266
 0.3917, -0.8561,  0.9422,  0.5061,  0.6123,  0.1212
-0.0366, -0.1087,  0.3449, -0.1025,  0.4086,  0.2475
 0.3633,  0.3943,  0.2372, -0.6980,  0.5216,  0.1925
-0.5325, -0.6466, -0.2178, -0.3589,  0.6310,  0.3568
 0.2271,  0.5200, -0.1447, -0.8011, -0.7699,  0.3128
 0.6415,  0.1993,  0.3777, -0.0178, -0.8237,  0.2181
-0.5298, -0.0768, -0.6028, -0.9490,  0.4588,  0.4356
 0.6870, -0.1431,  0.7294,  0.3141,  0.1621,  0.1632
-0.5985,  0.0591,  0.7889, -0.3900,  0.7419,  0.2945
 0.3661,  0.7984, -0.8486,  0.7572, -0.6183,  0.3449
 0.6995,  0.3342, -0.3113, -0.6972,  0.2707,  0.1712
 0.2565,  0.9126,  0.1798, -0.6043, -0.1413,  0.2893
-0.3265,  0.9839, -0.2395,  0.9854,  0.0376,  0.4770
 0.2690, -0.1722,  0.9818,  0.8599, -0.7015,  0.3954
-0.2102, -0.0768,  0.1219,  0.5607, -0.0256,  0.3949
 0.8216, -0.9555,  0.6422, -0.6231,  0.3715,  0.0801
-0.2896,  0.9484, -0.7545, -0.6249,  0.7789,  0.4370
-0.9985, -0.5448, -0.7092, -0.5931,  0.7926,  0.5402

Test data:

# synthetic_test_40.txt
#
 0.7462,  0.4006, -0.0590,  0.6543, -0.0083,  0.1935
 0.8495, -0.2260, -0.0142, -0.4911,  0.7699,  0.1078
-0.2335, -0.4049,  0.4352, -0.6183, -0.7636,  0.5088
 0.1810, -0.5142,  0.2465,  0.2767, -0.3449,  0.3136
-0.8650,  0.7611, -0.0801,  0.5277, -0.4922,  0.7140
-0.2358, -0.7466, -0.5115, -0.8413, -0.3943,  0.4533
 0.4834,  0.2300,  0.3448, -0.9832,  0.3568,  0.1360
-0.6502, -0.6300,  0.6885,  0.9652,  0.8275,  0.3046
-0.3053,  0.5604,  0.0929,  0.6329, -0.0325,  0.4756
-0.7995,  0.0740, -0.2680,  0.2086,  0.9176,  0.4565
-0.2144, -0.2141,  0.5813,  0.2902, -0.2122,  0.4119
-0.7278, -0.0987, -0.3312, -0.5641,  0.8515,  0.4438
 0.3793,  0.1976,  0.4933,  0.0839,  0.4011,  0.1905
-0.8568,  0.9573, -0.5272,  0.3212, -0.8207,  0.7415
-0.5785,  0.0056, -0.7901, -0.2223,  0.0760,  0.5551
 0.0735, -0.2188,  0.3925,  0.3570,  0.3746,  0.2191
 0.1230, -0.2838,  0.2262,  0.8715,  0.1938,  0.2878
 0.4792, -0.9248,  0.5295,  0.0366, -0.9894,  0.3149
-0.4456,  0.0697,  0.5359, -0.8938,  0.0981,  0.3879
 0.8629, -0.8505, -0.4464,  0.8385,  0.5300,  0.1769
 0.1995,  0.6659,  0.7921,  0.9454,  0.9970,  0.2330
-0.0249, -0.3066, -0.2927, -0.4923,  0.8220,  0.2437
 0.4513, -0.9481, -0.0770, -0.4374, -0.9421,  0.2879
-0.3405,  0.5931, -0.3507, -0.3842,  0.8562,  0.3987
 0.9538,  0.0471,  0.9039,  0.7760,  0.0361,  0.1706
-0.0887,  0.2104,  0.9808,  0.5478, -0.3314,  0.4128
-0.8220, -0.6302,  0.0537, -0.1658,  0.6013,  0.4306
-0.4123, -0.2880,  0.9074, -0.0461, -0.4435,  0.5144
 0.0060,  0.2867, -0.7775,  0.5161,  0.7039,  0.3599
-0.7968, -0.5484,  0.9426, -0.4308,  0.8148,  0.2979
 0.7811,  0.8450, -0.6877,  0.7594,  0.2640,  0.2362
-0.6802, -0.1113, -0.8325, -0.6694, -0.6056,  0.6544
 0.3821,  0.1476,  0.7466, -0.5107,  0.2592,  0.1648
 0.7265,  0.9683, -0.9803, -0.4943, -0.5523,  0.2454
-0.9049, -0.9797, -0.0196, -0.9090, -0.4433,  0.6447
-0.4607,  0.1811, -0.2389,  0.4050, -0.0078,  0.5229
 0.2664, -0.2932, -0.4259, -0.7336,  0.8742,  0.1834
-0.4507,  0.1029, -0.6294, -0.1158, -0.6294,  0.6081
 0.8948, -0.0124,  0.9278,  0.2899, -0.0314,  0.1534
-0.1323, -0.8813, -0.0146, -0.0697,  0.6135,  0.2386