Suppose you have a tiny dataset of nine items like this:

Sex Hand  Age  Height   City      Score (y)
-------------------------------------------
 M  right  24   tall    Anaheim    0.60		
 F  left   49   short   Anaheim    0.70
		
 M  right  33   medium  Boulder    0.20		
 M  ambi   38   short   Boulder    0.30		
 F  left   51   medium  Boulder    0.40
 M  right  28   tall    Boulder    0.50
		
 M  right  38   medium  Chicago    0.20		
 F  left   62   short   Chicago    0.30		
 M  ambi   27   tall    Chicago    0.40

Your goal is to predict Score from Sex, Hand, Age, Height, and City. And you want to use a tree-based regression system (bagging tree regression, random forest regression, AdaBoost.R2 regression, or gradient boosting regression).

Tree-Based Regression with Categorical Predictor Variables

The numeric Age variable can be left as-is, or it can be normalized (for example by dividing all age values by 100).

But the Sex, Hand, Height, and City values must be encoded into numeric values. (Note: Some library implementations of tree-based regression have built-in support for categorical predictors.)

The simplest possible approach is to always use ordinal encoding for any categorical predictor. Sometimes this works well, but sometimes a more sophisticated encoding approach works better.

Here is my standard set of five techniques to encode categorical variables for tree-based regression. Note that these are my personal heuristics, not some sort of generally accepted heuristics.

cardinality    encoding
================================================
natural order  plain ordinal
2 (binary)     0-1
3 to 4         one-hot
5 to 10        one-hot (or target-sorted ordinal)
10+            smoothed-target (aka impact)

The Sex variable is binary because it has exactly two possible values. Therefore it can be encoded as M = 0, F = 1 (or vice versa).

The Height variable has natural math order, with short is less than medium is less than tall. Therefore it can be plain ordinal encoded: short = 0, medium = 1, tall = 2.

The Hand variable has 3 possible values (where the values don’t have an inherent order) so it can be one-hot encoded: left = 1 0 0, right = 0 1 0, ambi (ambidextrous) = 0 0 1. (Where the order is arbitrary).

The City variable has just 3 possible values, so in a non-demo scenario it should be one-hot encoded. But suppose there were 5 through 10 possible cities. It could be encoded as target-sorted. The average of the target y values for each city is computed, the averages are sorted, and then the sort order is used as the encoding. For this data, the average y value for Anaheim is (0.60 + 0.70) / 2 = 0.65. For Boulder the average is 0.35. For Chicago the average is 0.30. If the averages are sorted from smallest to largest, the order is Chicago (0.30), then Boulder (0.35), then Anaheim (0.65). Therefore the target-sorted ordinal encoding is Chicago = 0, Boulder = 1, Anaheim = 2.

The target-sorted ordinal encoding is somewhat controversial. Many of my colleagues avoid it, and use one-hot encoding for up to 10 possible values.

Now suppose there were 10 or more cities. They would be smoothed-target encoded (also called impact encoded). The equation to encode a particular categorical value is:

x’ = [(n_cat * y_cat) + (m * y_global)] / (n_cat + m)

The n_cat is the number of times the value occurs in the dataset. The y_cat is the average of the y values for the value (just like target-sorting above). The y_global is the average of all the y values in the dataset. The m value is a hyperparameter. It is often set to 10, so I’ll use m = 10.

For this data, the overall average y value is 0.4000. The averages for Anaheim, Boulder, Chicago are 0.65, 0.35, 0.30 respectively. The counts of each values for Anaheim, Boulder, Chicago are 2, 4, 3 respectively.

Therefore, if m is set to 10, the encoding calculations are:

Anaheim: [(2 * 0.65) + (10 * 0.4000)] / (2 + 10) = 0.4417
Boulder: [(4 * 0.35) + (10 * 0.4000)] / (4 + 10) = 0.3857
Chicago: [(3 * 0.30) + (10 * 0.4000)] / (3 + 10) = 0.3769

Let me re-emphasize that I’m demonstrating smoothed-target (aka impact) encoding on just three values (rather than when there are 10 or more possible values) just to keep the example calculations simple.

In theory, the value of m should be determined using trial-and-error experimentation. But in practice, the value of 10 is often just used. Larger values of m produce stronger shrinkage toward the global mean, but the actual effect of using different values of m can vary depending on the exact nature of the dataset being used.

In practice, when using smoothed-target encoding, to prevent the model from memorizing rare values, you should consider grouping low-frequency categories (for example occurring less than x% of the time, where the x% depends on many factors) into an ‘Other’ bucket before encoding.

When using smoothed-target encoding, there is a danger of target leakage. Therefore, you should perform a train-test split first, use the training data to calculate the smoothed-target encoding, and then use those encoding values to transform the test data too. You should not encode the entire source dataset and then perform a train-test split.

In machine learning, data preparation is almost always by far the most time-consuming part of the process.

Most of my friends, and I, are fascinated by all kinds of strange things that don’t interest normal people. For reasons that are essentially unknown to me, I am intrigued by movies that feature a creature that makes noises that are translated to “chittering” in the closed caption text. The are an amazing number of chittering instances in science fiction movies, but you’ll only notice these chitterings if you’re actively looking for them.

Right: In “Project Hail Mary” (2026), the Sun is being eaten slowly but surely by some sort of space organisms. A crew of three travel to the distant star Tau Ceti to discover why that star is not being consumed. Only one member of the crew, Ryland, survives the trip. Upon arriving at Tau Ceti, Ryland discovers another spaceship from planet Erid with the same mission. Only one of the alien crew, Rocky, survived the trip. For the next 20 minutes or so, until Ryland and Rocky learn how to communicate, Rocky is in non-stop chittering mode. This movie got excellent reviews. I rarely agree with reviews but I really liked this movie when I watched it with my pal Ken. My personal grade = A.