Among my software developer colleagues, there’s a lot of interest in machine learning with neural networks. Neural networks are essentially complex math functions, so if you have predictor variables that can take categorical values, like (red, yellow, blue, green), you must encode those values as numbers.

For neural networks, by far the most common way to encode categorical values, is 1-of-(N-1) encoding. For example, for a variable “color” that can be one of (red, yellow, blue, green):

red    = (  1,  0,  0 )
yellow = (  0,  1,  0 )
blue   = (  0,  0,  1 )
green  = ( -1, -1, -1 )

Notice that with N=4 possible values, the encoding only uses N-1 values of -1, 0, or +1. Pretty simple.

Developers sometimes seem to have difficulty with data encoding for neural networks. I suspect that the main source of this difficulty is that classical statistics uses data encoding too, but there are three common forms: dummy coding, effect coding, and orthogonal coding. And the “effect coding” used by classical statistics is exactly the same as the 1-of-(N-1) coding used in machine learning, but the motivations are slightly different. So when a developer searches the Internet for information about data encoding, he will quite likely end up with information for classical statistics encoding rather than encoding for machine learning.

In my talk I wondered about a possibility that I’ve never seen discussed — using the (statistics) orthogonal encoding technique for neural network predictor values. I strongly suspect that orthogonal encoding wouldn’t make any difference, either better or worse, but lots of things in machine learning are tricky, so only some experiments would reveal the truth for sure.

“The Papal Palace Avignon”, Paul Signac (1900)