One of my background tasks is writing a deep neural network (DNN) from scratch. As part of my intellectual journey, I want to code a back-propagation training method for my DNN.

For some reason that’s not clear to me, I have a lot of trouble remembering exactly how back-propagation works. I can figure it out, usually after several hours, then code it up, but the following week I don’t remember how back-prop works. I’ve been through this figure-it-out then forget-it cycle many times.

Anyway, before attempting to do back-prop on a DNN, I figured I’d better revisit back-prop on a regular (non-deep) neural network. This time I made a diagram. See below.

Well, to be honest, the diagram didn’t help me nearly as much as I thought it would. In the end, once again, I can best understand back-prop by looking at code.