this post was submitted on 29 Jan 2024
21 points (100.0% liked)

Selfhosted

40040 readers
705 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
 

I currently have two computers, one that has a big zfs raidz pool that I currently back everything up to. Right now, on my local computer I use rsnapshot to do snapshot backups via rsync to the remote zfs pool. I know I'm wasting a ton of space because I have snapshotting in the rsync backup, and then the zfs pool is snapshotted every day.

Does it make sense to just do a regular rsync into a backup directory on the zfs pool and then just rely on the zfs pool snapshotting for snapshotting?

Maybe eventually I will put the local machine on zfs and then just send the local zfs snapshots over, but that will take some time. Thanks!

top 17 comments
sorted by: hot top controversial new old
[–] [email protected] 7 points 9 months ago* (last edited 9 months ago) (3 children)

Using plain rsync sounds sane.

Sending local ZFS snapshots to the remote ZFS might be problematic. Consider accidentally deleting important data locally and nuking all of your local snapshots, then sending that to the remote ZFS. You lost all of your snapshots and there's no way to recover the deleted data. Instead do what I do - keep the two ZFS systems separate and use a non-ZFS mechanism to transfer data - rsync, Syncthing, etc. That way even if you delete everything locally, nuke all local snapshots and send the deletions via rsync remotely, you could still recover your data by restoring the remote ZFS to a snapshot prior to the deletions. For reference I have two ZFS machines doing frequent snapshots and Syncthing replicating data between them on immediate basis.

!selfhosted, please do critique if you find some fundamental issues with this.

[–] [email protected] 3 points 9 months ago (1 children)

Zfs send / receive might be what you want

[–] [email protected] 2 points 9 months ago (1 children)

Wouldn't send/receive also sync snapshots across ZFS instances?

[–] [email protected] 1 points 9 months ago (1 children)

Docs say this , so yeah. "send streams can either be “full”, containing all data in a given snapshot, or “incremental”, containing only the differences between two snapshots. ZFS receive reads these send streams and uses them to re-create identical snapshots on a receiving system. "

[–] [email protected] 1 points 9 months ago* (last edited 9 months ago) (1 children)

Hm, so send doesn't "create the same state, bits and snapshots" on the other side. Instead it "adds net new snapshots" on the other side. 🤔

Perhaps I could use send instead of Syncthing after all. But then again I'm typically syncing net new data so the optimization would be minimal.

[–] [email protected] 1 points 9 months ago

I believe there is a method to do a 1-1 build copy, but my expertise ends at this point

[–] [email protected] 2 points 9 months ago

You don't sync the deletion of snapshots, you use expiry on the remote

[–] [email protected] 2 points 9 months ago (1 children)

This is fantastically helpful, thank you. I will do this.

I don’t know why I thought sending zfs snapshots was the better option

[–] [email protected] 2 points 9 months ago* (last edited 9 months ago)

Cause it makes sense at a glance and it's efficient. Not for backup purposes though.

[–] [email protected] 4 points 9 months ago (1 children)

It'd be worth checking out Borg as an alternative to rsync. Borg will handle snapshotting, and automatically de-dupe on a block-by-block basis.

I use it for all of my remote backups, and it provides a lot of quality of life stuff that rsync isn't going to handle.

[–] [email protected] 1 points 9 months ago

So for this, would i make another zfs pool on my remote backup server that is not snapshotted? Like, the problem i have is that i have snapshotting via rsync, but then the whole remote server zfs pool is further snapshotted so there’s a lot of redundancy.

[–] [email protected] 4 points 9 months ago (1 children)

Don't use filesystem snapshots as backup. They're a safety measure against accidental deletion or casual modifications but they're not backups.

If you want backups then use a proper, dedicated solution like Borg Backup. It connects remotely, takes care of deduplication, compression, encryption etc. and you can fully verify the backups and manage them individually.

[–] [email protected] 1 points 9 months ago

This is the right answer. A better backup strategy is an actual backup strategy. Snapshots, drive mirroring, rsync copies, etc aren't really backups.

[–] [email protected] 4 points 9 months ago (1 children)

That is rsync.net's entire business model.

I still rclone my Borg repos there instead of relying on snapshots though.

[–] [email protected] 1 points 9 months ago (1 children)

I also use rsync.net but as direct host for my borg repos, why rclone after?

[–] [email protected] 1 points 9 months ago

It works the same either way. Borg does a lot of different backups on my home network. I also have more than just Borg backups that I want off-site, so an rclone of everything from that nas share once after everything else is done makes more sense than duplicating Borg everywhere. The rclone'd stuff can be used directly just like if it was put there by Borg itself.

[–] [email protected] 3 points 9 months ago

Have you tried a restore? A non-differential smap snapshot should be fine, but differential snapshots would make a restore difficult to impossible.

A zfssend and zfsrestore with a differential snapshot would be more traditional. If one put mbuffer in the middle, it would even be fast.