I personally think it'll be fine. FSD is perfect for all my driving. It's been 3000-4000 miles since my last takeover. I know an uber driver who's driven his 16k miles without a safety intervention. And this is without them superfinetuning and doing custom navigation/mapping on a specific town.
You could argue its not necessary to achieve performance for an L2+ product and so keeps BOM cost down while still achieving goals. I'm not personally opposed though, the systems I worked on did have other sensors we could use.
That wouldn't resolve the concern around debugging/root-causing and remediating failures more quickly though. You still have a black box system that is difficult to simulate closed loop.
I'm a huge proponent of e2e learning for robotics (worked at two places doing e2e before Tesla adopted it) and personally believe its the right approach long term. I also have FSD on my Model 3 and love it for L2+. That said, my experience with disengagements is very different than yours.. I have a few a week for things like road works, school zones, route map following. Perfectly fine for L2+, L4 it would be unacceptable.
If these robotaxis end up looking more like my experience than yours then another layer of trouble will be root causing and fixing failure modes. Training models e2e makes both of these much more difficult.