Comment by zhyder
It's all about the chip economics. I don't know how the _manufacturing cost_ of Google's TPUs compares to Nvidia's GPUs, for inference of equivalent token throughput.
But at the moment Nvidia's 75-80% gross margin is slowly killing its customers like OpenAI. Eventually Nvidia will drop its margins, because non-0 profit from OpenAI is better than the 0 it'll be if OpenAI doesn't survive. Will be interesting to see if, say, 1/3 the chip cost would make OpenAI gross margin profitable... numbers bandied in this thread of $20B revenue with $115B cost imply they need 1/6 the chip cost, but I doubt those numbers are right (hard to get accurate $ numbers for a private company for the benefit of us arm-chair commenters).
Yes, from the first principles perspective this AI thingy is just about running electricity through some wires printed on silicon by a Taiwanese company using a Dutch machine. Which means, up until the Taiwanese you have plenty of room to cut margins up until that point the costs are mostly greed based. That is Nvidia is asking for the highest price the customer can pay and they have quite a way to the cost that define their min price. Which means AI companies can actually keep getting better deals until the devices delivered to them are priced close to TSMCs bulk wafer printing prices.