The goal of a machine learning regression problem is to predict a single numeric value. Quantile regression is a variation where you are concerned with under-prediction or over-prediction. I’ll phrase the rest of this blog post in terms of scenarios where you mostly care about under-prediction.

Suppose you want to predict how much liquor to order for an important charity fund raising event. There’s a large negative consequence if your prediction is too low (angry people who don’t get a drink) but minor consequence if your prediction is too high (you just hold the excess liquor until the next party). In such a scenario, you could make a 95th percentile regression prediction, which means 95% of your predictions will be correct or sightly high (and therefore only 5% of your predictions would be too low). You can also loosely phrase this as, “There’s a 95% chance my prediction will meet the demand.”

Most machine learning regression techniques over-predict about 50% of the time, and under-predict about 50% of the time. Put another way, most regression techniques are implicitly 50th percentile quantile techniques.

A naive approach is to start by creating a standard regression model. You can then modify the model’s Predict() method by adding some constant to the normal predicted values so that 80% of the predictions are greater than the associated target values. This naive quantile regression approach is simple and often works reasonably well, but a better approach is to create a different model.

Quantile regression: naive add-constant approach vs. quantile loss function.

The ideas are illustrated by the graph above. The hypothetical graph shows a linear prediction system, but the ideas are the same for other regression techniques such as a neural network. There are 10 training data points. A standard linear regression model, the solid red prediction line, under-predicts 5 of the data points and over-predicts 5 of the data points.

The naive quantile approach, the dashed green prediction line, adds a constant 0.35 to the standard linear predictions, which is just enough so that 8 out of 10 data points are below the line and therefore you do not under-predict except for 2 of the data points.

However, the naive modified prediction line now does not fit the data as well as a different prediction line, the dashed purple line. Instead of adding a fixed constant to the predict method, you can maintain the 80 percentile requirement and get the better prediction model by using a special quantile loss function during training that penalizes under-predictions more than over-predictions .

There are many ways to define a quantile loss function. One example is:

error = target_y - predicted_y
loss = Max((q - 1.0) * err), (q * error))

For example, suppose the specified quantile is q = 0.80, and a target y value is 0.45. If the predicted y is 0.35 (an unwanted under-prediction of 0.10), the loss is:

error = 0.45 - 0.35 = 0.10
loss = Max(-0.20 * 0.10, 0.80 * 0.10)
     = 0.08

If the predicted y is 0.55 (a not-so-bad over-prediction of 0.10), the loss is:

error = 0.45 - 0.55 = -0.10
loss = Max(-0.20 * -0.10, 0.80 * -0.10)
     = 0.02

And so the 0.10 under-prediction is penalized more than the 0.10 over-prediction.

Now, all of this said, in some scenarios, instead of using a somewhat complicated quantile loss function, the simplicity of the naive add-constant technique for quantile regression is good enough.

Quantile regression was one of many steps in the evolution of modern machine learning techniques.

The first automatic scoring with lights on the backglass appeared in 1935 on the “Rocketlite” by Bally. Automatic scoring with reels didn’t appear until 1953 in “Army Navy” by Williams. Therefore, some pinball experts consider Army Navy the first modern machine. Before automatic scoring, players had to keep track of their own score (balls remained in pockets in the playing field).