Comment by hypeatei a day ago
Well, patience as a consumer might pay off in the next year or so when the music stops and hyperscalers are forced to dump their inventories.
There still isn't a clear path to profitability for any of these AI products and the capital expenditure has been enormous.
> Consumer DDR5 motherboards normally take UDIMMs. Server DDR5 motherboards normally take RDIMMs. They're mechanically incompatible, and the voltages are different.
All you need is a fixed-latency, dumb translator bridge where the adapter forces everything into a simplified JEDEC-compliant mode.
CA/CK Line Translator with a Fixed Retimer as the biggest mismatch between RDIMM/UDIMM is the command/address path.
RDIMMs route CA/CK to RCD to DRAM, and the UDIMMs route CA/CK to DRAM directly, take the UDIMM CA/CK, delay + buffer + level shift it, feed it into a "RCD" like input using a delay locked loops (DLL).
Throw in a SPD translator, PMIC and voltage correction, DQ line conditioning and some other stuff into a 10–12-layer PCB, retimer chips, vrm, and level shifters.
It would cost about $40 million to fab and about $100 per adapter but would make bank with all the spare UDIMMs when the bubble bursts.
I imagine the cost is primarily in the actual DRAM chips on the DIMM. So availability of RDIMMs on the market will affect DRAM prices anyway. These days lots of motherboards come with Oculink, etc. and you can get a U.2 PCIe card for rather cheap.
I put together a small server with mostly commodity parts.
I see it a bit differently. In marketing, companies like AppLovin with the Axon Engine and Zeta Global with Athena are already showing strong profitability, both in earnings and free cash flow. They’re also delivering noticeably higher returns on ad spend compared to pre-AI tools for their customers. This is the area I’m researching most closely, so I can only speak for marketing, but I’d love to hear from others seeing similar results in their industries.
Its a bit of a shame these AI GPUs don't actually have displayport/hdmi output ports because they would make for nice cheap and powerful gaming GPUs with a lot of VRAM, they would potentially be really good graphics cards.
Will just have to settle for insanely cheap second hand DDR5 and NVMe drives I guess.
AI GPUs suck for gaming, I have seen a video from a guy playing Red Dead Redemption 2 on a H100 at a whooping 8 FPS! That's after some hacks, because otherwise it wouldn't run at all.
AI GPUs are stripped away of most things display-related to make room for more compute cores. So in theory, they could "work", but there are bottlenecks making that compute power irrelevant for gaming, even if they had a display output.
So there aren't actually GPUs, but more like some other architecture for CPUs.
If you can't afford the electricity to afford to run the model on free hardware, you'd certainly never be able to afford the subscription to the same product as a service!
But anyway, the trick is to run it in the winter and keep your house warm.
I don't think I am. I don't think economies of scale on hardware will drive costs below free, and while subscription providers might be willing to offer services below the cost of the hardware that runs them, I don't think they'll offer services below the cost of the electricity that runs them.
And while data centers might sign favorable contracts, I don't think they are getting electricity that far below retail.
A single machine for personal inference on models of this size isn't going to idle at some point so high that electricity becomes a problem and for personal use it's not like it would be under load often and if for some reason you are able to keep it under heavy load presumably it's doing something valuable enough to easily justify the electricity.
> Well, patience as a consumer might pay off in the next year or so when the music stops and hyperscalers are forced to dump their inventories.
Their inventories are not what consumers use.
Consumer DDR5 motherboards normally take UDIMMs. Server DDR5 motherboards normally take RDIMMs. They're mechanically incompatible, and the voltages are different. And the memory for GPUs is normally soldered directly to the board (and of the GDDRn family, instead of the DDRn or LPDDRn families used by most CPUs).
As for GPUs, they're also different. Most consumer GPUs are PCIe x16 cards with DP and HDMI ports; most hyperscaler GPUs are going to have more exotic form factors like OAM, and not have any DP or HDMI ports (since they have no need for graphics output).
So no, unfortunately hyperscalers dumping their inventories would be of little use to consumers. We'll have to wait for the factories to switch their production to consumer-targeted products.
Edit: even their NVMe drives are going to have different form factors like E1.S and different connectors like U.2, making them hard for normal consumers to use.