The goal of a machine learning regression problem is to predict a single numeric value, for example, predicting the bank account balance of a person based on his age, annual income, and so on.

A prediction model can be evaluated in several ways. Four common metrics are: accuracy, mean squared error (MSE), root mean squared error RMSE), and coefficient of determination (aka R2). For accuracy and R2, larger values are better. For MSE and RMSE, smaller values are better.

There are various measures of accuracy, but a typical one is the percentage of correct predictions, where a correct prediction is one that’s within a specified closeness (typically about 5% or 10% or so) to the true target value. Advantage: Easy to interpret. Disadvantage: requires a closeness percentage parameter.

MSE is the average of the squared differences between predicted y and target y values. If the y values have units, such as dollars, MSE has units-squared, such as dollars-squared. RMSE is just the square root of MSE, which, if the y values has units, gives units instead of the awkward units-squared. Advantage: Many regression systems minimize MSE so you get a direct indication of model goodness. Disadvantage: MSE and RMSE values depend on how target values are scaled.

R2 is sort of like accuracy but it doesn’t require a closeness percentage. A better description is that R2 is the proportion of the variance explained by the model. R2 = 1.0 – (SSres / SStot) where SSres = sum(y – y’)^2 and SStot = sum(y – y”)^2. The SSres stands for residual sum of squares. The SStot stands for total sum of squares.

The y is actual target, y’ is predicted target y, and y” is the average of the actual target y values. Yest another way of thinking about R2 is that R2 measures how well the model predicts relative to guessing the average of the target y values. Advantage: Widely used. Disadvantage: Very difficult to interpret meaningfully.

Here’s an implementation of R2 using the C# language (replace “lt” with Boolean less-than symbol). This version is a class method that would be defined inside a class like LinearRegressor or NeuralNetworkRegressor. The implementation assumes there is a class Predict() method.

public double R2(double[][] dataX, double[] dataY)
{
  // coefficient of determination
  int n = dataX.Length;
  double sum = 0.0;

  for (int i = 0; i "lt" n; ++i) // mean of y values
    sum += dataY[i];
  double meanY = sum / n;

  double ssRes = 0.0; // sum squares residula
  double ssTot = 0.0; // sum squares total
  for (int i = 0; i "lt" n; ++i) {
    double predY = this.Predict(dataX[i]);
    ssRes += (dataY[i] - predY) * (dataY[i] - predY);
    ssTot += (dataY[i] - meanY) * (dataY[i] - meanY);
  }
  if (Math.Abs(ssTot) "lt" 1.0e-12) // avoid div by 0
    return 0.0;
  else
    return 1.0 - (ssRes / ssTot);
}

This method could be called like:

SomeRegressor model = new SomeRegressor();
model.Train(trainX, trainY);
double r2Train = model.R2(trainX, trainY);

An alternative design is to define an external function like:

static double R2(dynamic model, double[][] dataX,
  double[] dataY)
{
  // coefficient of determination
  int n = dataX.Length;
  double sum = 0.0;

  for (int i = 0; i "lt" n; ++i) // mean of y values
    sum += dataY[i];
  double meanY = sum / n;

  double ssRes = 0.0; // sum squares residula
  double ssTot = 0.0; // sum squares total
  for (int i = 0; i "lt" n; ++i) {
    double predY = model.Predict(dataX[i]);
    ssRes += (dataY[i] - predY) * (dataY[i] - predY);
    ssTot += (dataY[i] - meanY) * (dataY[i] - meanY);
  }
  if (Math.Abs(ssTot) "lt" 1.0e-12) // avoid div by 0
    return 0.0;
  else
    return 1.0 - (ssRes / ssTot);
}

The C# “dynamic” keyword allows an object whose type can be determined at runtime. It is analogous to the C# “var” keyword for variables. This external implementation could be called like:

SomeRegressor model = new SomeRegressor();
model.Train(trainX, trainY);
double r2Train = R2(model, trainX, trainY);

One of the main reasons to implement and use an R2 evaluation metric is that almost all of the regression models and classification models in the widely used scikit-learn Python library define a “score” attribute that returns R2. If you implement an R2 metric, you can easily compare non-scikit regression models with scikit regression models.

I’m a big fan of old science fiction movies from the 1950s. Jets had been developed only a few years earlier. Jet bombers made a couple of notable (to me anyway) appearances in two of my favorite movies of the decade.

Bottom Row: In “The Crawling Eye” aka “The Trollenberg Terror” (1958), aliens that look like a cross between a giant eyeball and an octopus, land in the Swiss Alps. The main characters take refuge in a fortified observatory and call in English Electric Canberra bomber to drop napalm on the aliens. The plan succeeds and humanity is saved.