Comment by sebastianconcpt 2 days ago
This comment:
> “Why do we have to write the backward pass when frameworks in the real world, such as TensorFlow, compute them for you automatically?”
worries me because is structured with the same reasoning of "why we have to demonstrate we understand addition if in the real world we have calculators"
But you went all the way to the extreme. There is a gradient of know-how that is essential for mastery.
You can ignore every detail of a transistor, but knowing it can be like a valve is enough to understand that it can model consistently a logic gate of a stream of electrons and that can be used to compute.
If wouldn't be for that, then the laws that govern your whole computer become magic for you brain.
I fear the consequences of minds that value more their laziness rather to understand, let's say with 5 degrees deep of "why", the things in reality work the way they do. The alternative is that they are hallucinating all the time on narratives that can be psychotic or real and they are not equipped to discern which is which.
The counter-argument would be that you can make excellent arguments for why we should understand what the compiler is doing, understand what the transistors are doing (e.g. to understand the limitations and risks of overclocking), understand how sort algorithms work (doing sort algorithms manually was at one time a typical interview question). It's not that these things aren't useful, but whether or not that learning would be useful is not the right question.
Is this _more_ useful than the other things, which I could be learning but won't because I spent the time and effort to learn this instead?
We have a finite capacity for learning, if for no other reason then at least because we have a finite amount of time in this life, and infinite topics to learn (and there are plenty of other constraints besides time). The reason given for learning this topic, is that it has hidden failure modes which you will not be on the lookout for if you didn't know how it worked "under the hood".
Is this a good enough reason to spend time learning this rather than, say, how to model the physics of the system you're training the neural network to deal with? Tough question; maybe, maybe not. If you have time to learn both, do that, but if not, then you will have to choose which is most important. And in our education system, we do things like teach calculus but not intermediate statistics, and it would have been better to do the opposite for something like 90% of the people taking calculus.
That said, I've implemented backpropagation multiple times, it's a good way to evaluate a new language (just complex enough to reveal problems, not so complex that it takes forever).