Comment by matt-p
If you’re using 7280-SR3 switches, they’re certainly a fine choice. However, have you considered the 7280-CR3(K) range? They're much better $/Gbps and more relevant edge interfaces.
At this scale, why did you opt for a spine-and-leaf design with 25G switches and a dedicated 32×100G spine? Did you explore just collapsing it and using 1-2 32×100G switches per rack, then employing 100G>4×25G AOC breakout cables and direct 100G links for inter-switch connections and storage servers?
Have you also thought about creating a record on PeeringDB?https://www.peeringdb.com/net/400940.
By the way, I’m not convinced I’d recommend a UniFi Pro for anything, even for out-of-band management.
All valid points - and our ideas for Gen 2 sound directionally similar - but those are at crayon drawing stage.
When we started, we didn't have much of an idea about what the rack needs to look like. So we chose a combination of things we thought we could pull this off. We're mostly software and systems folks, and there's a dearth of information out there on what to do. Vendors tend to gravitate towards selling BGP+EVPN+VXLAN or whatever "enterprise" reference designs; so we kinda YOLO'ed the Gen 1. We decided to spend extra money if we could get to a working setup sooner. When the clock is in cloud spend, there's uh... lots of opportunity cost :D.
A lot of the chipset and switch choices were bets and we had to pick and choose what we gambled on - and what we could get our hands on. The main bets this round were eBGP to the hosts with BGP unnumbered, SONiC switches - this lets us do a lot of networking with our existing IPv6/Wireguard/eBPF overlay and a debian based switch OS + FRR (so fewer things to learn). And ofc. figuring out how to operationalise the install process and get stuff running on the hardware as soon as possible.
Now we've got a working design, we'll start iterating a bit more on the hardware choice and network design. I'd love for us to write about it when we get through it. Plus I think we owe the internet a rant on networking in general.
Edit: Also we don't use UniFi Pro / Uniquity gear anywhere?