Comment by toddmorey

Once worked in a "DC" in a converted cow shed in the English countryside. Hot takes that align with your experiences:

    - A key microwave link kept going down with intermittent packet errors way down in the data link layer. A short investigation discovered that a tree across the road had come into leaf, and a branch was blowing into the line of sight of the kit on our building. A step-ladder, a saw and 10 minutes later we restored connectivity
    - Our main (BGP-ified) router out of the DC - no, there wasn't a redundant device - kept rebooting. A quick check showed the temp in the DC was so high, cooling so poor, that the *inlet* fan had an air temp of over 60C. We pointed some fans at it as a temporary measure. 
    - In a similar vein, a few weeks later the air con in another room just gave up and started spewing water over the Nortel DMS-100 (we were a dial-in ISP with our own switch). Wasn't too happy to be asked to help mop it up (thought the water could potentially be live), but what to do?

And a few years after that I also got to commission a laser beam link over Manchester's Oxford Road (at the time, the busiest bus route in Europe), to link up an office to a University campus. Fun times.

It was all terrific fun, but I'm so glad I now only really do software.

> Data center science has... well improved since the earlier days

You say that, but...

> There was one day we knocked out windows and purchased box fans because servers were literally catching on fire

This happened to Equinix's CH1 datacenter in Chicago Jan24 (not the literal fire part). Took down Azure ExpressRoute.

Apparently it got too cold and the CRACs couldn't take it? I'm told they had all the doors and windows open trying to keep things cold enough, but alas. As the CRAC goes, so goes the servers

Many years ago I had a BlackDiamond dropped on my foot during installation at INTX LON1 for LINX, disabling me for hours. The switch in question was evidently cursed: later that week a spanning tree misconfiguration on the same unit then disabled LINX for hours, throwing half of Britain's ISP peering into temporary chaos, and everyone else involved in that project was dead within two years.

We had a bird land on a transformer up on a pole and blew fuses. A couple years later, I toured the facility and the fried carcass was still there on the ground below it.

> There was one day we knocked out windows and purchased box fans because servers were literally catching on fire.

Pointing the fans in or out?

> Data center science has... well improved since the earlier days. We worked with Facebook on the OpenCompute Project that had some very forward looking infra concepts at the time.

Am a bit surprised Meta doesn't offer a cloud provider yet to compete with AWS/GCP. Especially considering how much R&D they've put into their infra.

In the bad old days I had a server at blue host in Dallas. Went to the dc once and there extension cords accross the racks suspended about 1ft off the ground that I had to step over to get to my server. Hey at least it was cheap :)

When it comes to Internet service we're living in the early 2000s in the some parts of the manufacturing world

I recall getting a DC tour of LON3 and being totally blown away by it all as a 20-something web dev. Good times.

I attended an OCP lecture by someone involved in building a facebook DC.

One of the stories was learning that stuff on top gets hotter than stuff on bottom.

This is, like, basic stuff here, guys. I've never understood the hiring practices in these projects.

> and purchased box fans because servers were literally catching on fire

Ah yes, or a collection of R2D2 portable air conditioners, with the tails draped out through the window.

Or a coolant leak that no one noticed until the sub-floor was completely full and the floor panels started to float!