Comment by bob1029

Unpowered in a warehouse is a huge latency problem.

For storage especially we now build enough redundancy into systems that we don't have to jump on every fault. That reduces the chance of human error when trying to address it, and pushing the hardware harder during recovery (resilvering, catching up in a distributed concensus system, etc).

When the entire box gets taken out of the rack due to hitting max faults, then you can piece out the machine and recycle parts that are still good.

You could in theory ship them all off to the backend of nowhere, but it seems that Glacier is all the places where AWS data centers are, so it's not that. But Glacier being durable storage, with a low expectation of data out versus data in, they could and probably are cutting the aggregate bandwidth to the bone.

How good do your power backups have to be to power a pure Glacier server room? Can you use much cheaper in-rack switches? Can you use old in-rack switches from the m5i era?

Also most of the use cases they mention involve linear reads, which has its own recipe book for optimization. Including caching just enough of each file on fast media to hide the slow lookup time for the rest of the stream.

Little's Law would absolutely kill you in any other context but we are linear write, orders of magnitude fewer reads here. You have hardware sitting around waiting for a request. "Orders of magnitude" is the space where interesting solutions can live.

Only if you have low redundancy. RAIDZ is better about this isn’t it? And Backblaze goes a lot farther. They just decommission the rack when it hits the limit for failed disks, and the files on the cluster are stored on m of n racks, so adding a rack and “resilvering” doesn’t even require scanning the entire cluster, just m/n of it.

This is less of a concern with RAID 6, and especially in Glacier's use case where reading any piece of data happens seldom, I'd expect it to be fine.

There’s a really old trick with HDDs where you buy a big disc and then allocate less than half of it. There’s more throughput on the first half of the disk, more tracks per cylinder so fewer seeks, and never having to read half the disk reduces the worst case seek time. All increase IOPs.

But then what do you do with the other half of the disk? If you access it when the machine isn’t dormant you lose most of these benefits.

For deep storage you have two problems. Time to access the files, and resources to locate the files. In a distributed file store there’s the potential for chatty access or large memory footprints for directory structures. You might need an elaborate system to locate file 54325 if you’re doing some consistent hashing thing, but the customer has no clue what 54325 is. They want the birthday party video. So they still need a directory structure even if you can avoid it.