Bottom line: Encoding categorical predictor variables for use in tree-based regression systems is a surprisingly complex topic. There are two main encoding schemes:

1. Always use simple ordinal encoding.
2. Use ordinal, or one-hot, or smoothed target encoding:
  a. binary predictor = use 0-1 encoding (ordinal encoding)
  b. categorical with natural ordering = use ordinal encoding
  c. categorical no ordering, 3-10 values = use one-hot encoding
  d. categorical no ordering, 10+ values = smooth target encoding

The first scheme is extremely simple. The second scheme is quite complicated. Sometimes the first, simple, encoding scheme works better. Sometimes the second, complex encoding scheme works better.

This blog post discusses the first, simple encoding scheme. The second, complex, encoding scheme is discussed in other blog posts. Search for “smoothed target”.

Suppose you want to predict a single numeric value (a regression problem) using a machine learning technique that is based on collections of decision trees — specifically, bagging regression, random forest regression, AdaBoost.R2 regression, or gradient boost regression. (The encoding techniques for math-based regression techniques such as kernel ridge regression and neural network regression is a completely different story).

And suppose you have a dataset of people that looks like:

Sex Age  State  Height  Income    Politics
-------------------------------------------------
M   41   GA     68.5   59100.00   moderate
F   23   AL     61.0   36800.00   conservative
F   18   CA     64.0   42600.00   liberal
M   56   CT     71.0   62400.00   moderate
. . .

Your goal is to predict Income from the other five variables. The numeric Age and Height predictors, and the target Income variable can be left as-is or normalized. But the categorical Sex, State, and Politics variables must be encoded. Encoding categorical variables for use with tree-based regression systems seems like it should be simple but in fact is an astonishingly complicated topic.

Note: There are some regression tree systems, such as CART trees, that have built-in support for categorical variables.

Example of Gradient Boost Regression on Ordinal Encoded Data

I will cut to the chase and say that, surprisingly, when all factors are taken into account, using simple ordinal encoding often works as well as the much more complicated ordinal, one-hot, smoothed target encoding. So for Sex, Male = 0, Female = 1. For State, Alabama = 0, Alaska = 1, . . Wyoming = 9. (Assuming only 9 States). For Politics, conservative = 0, moderate = 1, liberal = 2. Simple.

If numeric variables are divide-by-constant normalized, and the categorical variables are ordinal encoded, the data becomes:

Sex Age  State  Height  Income   Politics
-------------------------------------------------
0   0.41  9     0.685   0.59100   1
1   0.23  1     0.610   0.36800   0
1   0.18  4     0.640   0.42600   2
0   0.56  6     0.710   0.62400   1
. . .

Ordinal encoding imposes an arbitrary math ordering such as, “Wyoming is greater than California”. But a tree splits data using simple threshold checks (e.g., State less-than-or-equal 14). It can chain these splits down a branch to isolate specific numbers or groups of numbers, effectively carving out individual states regardless of their arbitrary integer values.

A simple, but less obvious alternative is count-encoding. You encode according to the counts of each value. For example, if the dataset has 324 items/people from California (because it’s a populous state) but just 9 people from Wyoming, then California = 324 and Wyoming = 9.

Count encoding can be better than ordinal encoding when the frequency of occurrence contains meaningful real-world signal that directly correlates with your target variable, in this case State population size, which might be a predictor of Income.

A huge advantage of ordinal-encoding and count-encoding is that you can encode categorical variables in the source dataset and then split into training and test sets, as opposed to many encoding techniques such as “target-mean with OOF and smoothing”, where you must split first, then fold, then encode — a real nightmare in practice. Put another way, there is no data leakage with ordinal encoding and count-encoding.

The target-mean encoding with OOF and smoothing technique often appeals to beginners because it looks very impressive with all kinds of equations and related math concepts. But in a production environment, sometimes simplicity trumps fancy math.

Machine learning encoding transforms categorical data into numeric data.

Right: Stone Boy could . . . well, turn himself into stone. But then he couldn’t move or speak. He applied to the Legion in Adventure Comics #306 (March 1963) but was rejected. Other hilarious rejectees included Antennae Boy (radio antennae on his head and radio in his brain), Chlorophyll Kid (make plants grow fast), Camera Kid (a video camera in his forehead), Flat Boy (make himself flat like a rug), Leadfeather Lad (make bird feathers very heavy), and Spider Girl (expanding hair). In a later issue, Adventure Comics #315 (December 1963), Stone Boy showed great courage and was offered membership in the Legion, but he declined so he could remain a member of the Legion of Substitute Heroes.