Comment by dewey

Comment by dewey 10 months ago

18 replies

View on Hacker News

Providing robust restoring and backup systems for a system that allows to run any kind of workload is almost impossible. You'd have to provide database backups for all versions of all databases, correct file backup for the volumes etc.

It feels much more dangerous to have such a system instead in place and provide false sense of security. Users know best what kind of data they need to backup, where they want to back it up, if it needs to be encrypted or not, if it needs to be daily or weekly etc.

sgarland 10 months ago

ZFS. Snapshot the entire filesystem, ship it off somewhere. Done. At worst, Postgres is slow to startup from the snapshot because it thinks it’s recovering from a crash.

Reply View 10 replies
  • GauntletWizard 10 months ago

    Postgres is recovering from a crash if it's reading from a ZFS snapshot. It probably did have several of it's database writes succeed that it wasn't certain of, and others fail that it also wasn't certain of, and those might not have been "in order". That's why WAL files exist, and it needs to fully replay them.

    Reply View 0 replies
  • prmoustache 10 months ago

    Most projects/products can survive a few seconds of downtime to have a clean snapshot.

    Reply View 0 replies
  • dewey 10 months ago

    Classic HN reply that’s very disconnected from reality. Most people don’t run ZFS, most people using these tools are using this to self host their apps as it’s cheaper than managed cloud server. Usually on a dedicated or VPS server where by default you run stock Ubuntu and no niche file system.

    Reply View 7 replies
    • FloatArtifact 10 months ago

      TrueNas needs to embody best practice for application backup and restore even though it runs ZFS.

      As a side note, it's pretty interesting TrueNas is heavily designed around the ZFS paradigm. Most people think of applications and time stamps to restore not data pools and the snapshot paradigm accompanying ZFS.

      With the right layer of abstraction on top of snapshots one could have the cake you need it to as it is difficult for beginners to grasp ZFS.

      Reply View 0 replies
    • transpute 10 months ago

      Ubuntu ships tested ZFS modules.

      https://utcc.utoronto.ca/~cks/space/blog/linux/UbuntuKernels...

      https://wiki.ubuntu.com/ZFS

      Reply View 3 replies
      • dewey 10 months ago

        You are missing the point, nobody doubts it's possible but defaults matter and people just commission a new "Ubuntu" server on OVH, Hetzner, DO etc. and don't configure ZFS or snapshotting or even want to know what that is.

        Reply View 2 replies
    • sgarland 10 months ago

      I didn’t say most people run it, I offered it as a solution. If you’re willing to run a server, you should be willing to try ZFS. As long as you follow some best practices, you’ll be fine.

      Reply View 0 replies
    • pmarreck 10 months ago

      > Most people don’t run ZFS

      Well, the people who host the people do. Is that not argument enough in favor of it?

      Reply View 0 replies
sally_glance 10 months ago

A viable strategy, but requires an experienced Linux/Unix admin and quite some planning & setup effort.

There are a lot of non-obvious gotchas with ZFS, and a lot of knobs to turn to make it do what you want. Anecdotally, a coworker of mine set it up on his development machine back when Ubuntu was heavily promoting it for default installs. It worked well until one day his machine started randomly freezing for minutes multiple times a day... He traced the issue back to some improper snapshotting setup, then spend a couple of days trying to fix it before going back to ext4.

For the Postgres data use case in particular, I would be wary of interactions and probably require a lot of testing if we were to introduce it... Though it seems at least some people are having success with it (not exactly plug and play or cheap setup though): https://lackofimagination.org/2022/04/our-experience-with-po...

Reply View 1 reply
  • sgarland 10 months ago

    I think you meant to reply to me.

    There are a ton of ZFS knobs, yes, but you don’t need most of them to have a safe and performant setup. Optimal, no, but good enough.

    It’s been well-tested with DBs for years; Percona in particular is quite fond of it, with many employees writing blog posts on their experiences.

    Reply View 0 replies
cweagans 10 months ago

I don't think that's true. I opened https://github.com/dokku/dokku/issues/5008 a while back and Jose didn't seem to disagree.

Addressing your argument directly though: you know that if you spin up a Postgres database for your app, you need to dump the database to disk to back it up (or if you wanna get fancy, you can do a delta from the last backup + a full backup periodically). Anytime a Postgres database exists, you know the steps you need to take to backup that service.

Same with persistent file storage on disk: if you have a directory of files, you need a snapshot of all of those files.

Each _service_ can know how to back itself up. If you tell a Dokku _app_ to back itself up, what you really mean is that each _service_ attached to that app should do whatever it needs to do to create a backup. Then, dokku only needs to collate all of the various backup outputs, include a copy of the git repository that drives the app, tar/zstd it, and write it to disk.

As you pointed out, the user should probably be able to control the backup cadence, where those backups are shipped off to, the retention period, whether or not they are encrypted, etc, but the actual mechanics of performing a backup aren't exactly rocket science. All of the user configurable values can have reasonable defaults too -- they can/should Just Work (tm). There's value in having that work OOTB even if the backups are just being written to disk on the actual Dokku machine somewhere.

Reply View 3 replies
  • dewey 10 months ago

    I've landed on your issue too at the time when I was building my Dokku setup. I don't disagree that it would be nice but I just disagree with the parent poster making it sound like it's an essential feature that makes the project any less valuable.

    Reply View 2 replies
    • cweagans 10 months ago

      Sure, Dokku is still valuable without it. I've seen a ton of recommendations over the years to "just use Dokku" when people ask about simple hosting setups for their side hustle or whatever though. In that context, Dokku isn't really a viable option unless backups are in place - ideally OOTB. Without having backups OOTB, the idea of Dokku being a really easy way to get going is a little bit diluted, don't you think?

      I sort of see it the same way as "just install Linux Mint, it's great, you'll love it", but when something doesn't work, it's "oh yeah, just open the terminal and ________": opening the terminal is the _last_ thing people want to do if they just wanna read their email or whatever.

      Reply View 1 reply
      • FloatArtifact 10 months ago

        My comment comes from the lack of backup and restore systems for nearly all projects that I've seen, especially open source for services and applications.

        Reply View 0 replies
FloatArtifact 10 months ago

It's worth discussing backup/restore systems abstraction file system such as ZFS versus application layer.

Reply View 0 replies