Comment by ActorNightly

>Every time you pit the sheer violent force of end to end backpropagation against compartmentalization and lines drawn by humans, at a sufficient scale, backpropagation gets its win.

I fully agree, but your statement is quite ironic.

For driving, humans drive well because we operate more like Mu Zero model does - we can "visualize" the possibilities of the future states depending on what we do and pick the most optimal path. We don't need to know what the specific object is on the road, the fact that we can recognize that its there, in our path, and understand the physical interacting of a car hitting something that is taller than a bump means we can avoid it.

The way to implement self driving is exactly that - train your model to take sensor data and reconstruct a 3d space in latent dimensions, train another model to predict evolutions on the 3d space given time history with probabilistic output, and then your inference is a probabilistic guided search in that space with time constraints based on hardware. Mu Zero is nothing new, and already proved that you don't even need a hardcoded model of environment to operate in.

And you don't even need human driving data for this, as the model will be able to predict things collisions solely based on pure physics. And as a bonus, as you enhance it with things like physical models of the cars, where it can reconcile what it thinks the system is going to do versus what the physics calculations predict, you can even make it drive well in snow with low traction.

The irony of your statement is that everyone who is going end to end is manually hand coding all these hacks (like image warping in the case of Comma AI) to make the training work, all because the training data is just not sufficient, which is the exact same exercise as humans drawing lines.

And if you doubt that what Im saying is true, again, Mu Zero was proven to work. Driving is just another game where you can easily define a winning scenario, the board, and moves you can make, and apply the same concepts. The only technical part becomes accurately determining the board from sensor data.