I recently gave a lecture on back-propagation. I’ve spoken on this topic before and it’s always a challenge because back-propagation has many interrelated ideas.

Even defining back-propagation is quite tricky because you can think of it in many ways. Back-propagation is a technique (or algorithm) to find the values of the weights and biases of a neural network. But this definition only makes sense if the audience fully understands everything about the neural network input-output mechanism.

Back-propagation is based on the Calculus gradient of the neural network error function. Again, for this to make any sense, the audience has to completely understand neural network error — both squared error and cross entropy error — and Calculus gradients, which are a form of Calculus derivatives.

And so on, and so on. There are multiple concepts that all have dependencies on other concepts.

That said, to understand back-propagation, you just have too keep looking at it. I remember when I was first learning back-prop many years ago, it seemed like the learning process would never end. But eventually it did.

I think the real challenge is for a person to determine at how deep a level they need to understand back-propagation. It’s sort of like learning about an automobile. You have to have basic knowledge in order to use a car, deeper knowledge to perform basic maintenance like change oil, deeper knowledge yet to work on a transmission, and so on.

In the same way, a machine learning practitioner doesn’t need to have a profound depth of knowledge to use a neural network library that uses back-propagation. But the more you know, the better in general — but I can imagine scenarios where knowing too much could work against you.

Director Alfred Hitchcock did two entirely different movies, with the same title of “The Man Who Knew Too Much”, one in 1934 and one in 1956. I prefer the 1956 version.