this post was submitted on 17 Feb 2025
74 points (98.7% liked)

Selfhosted

42718 readers
582 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
 

Basically title. I'm in the process of setting up a proper backup for my configured containers on Unraid and I'm wondering how often I should run my backup script. Right now, I have a cron job set to run on Monday and Friday nights, is this too frequent? Whats your schedule and do you strictly backup your appdata (container configs), or is there other data you include in your backups?

top 50 comments
sorted by: hot top controversial new old
[–] [email protected] 2 points 2 days ago* (last edited 2 days ago)

I'm always backing up with SyncThing in realtime, but every week I do an off-site type of tarball backup that isn't within the SyncThing setup.

[–] [email protected] 32 points 4 days ago* (last edited 4 days ago) (2 children)

I run Borg nightly, backing up the majority of the data on my boot disk, incl docker volumes and config + a few extra folders.

Each individual archive is around 550gb, but because of the de-duplication and compression it's only ~800mb of new data each day taking around 3min to complete the backup.

Borgs de-duplication is honestly incredible. I keep 7 daily backups, 3 weekly, 11 monthly, then one for each year beyond that. The 21 historical backups I have right now RAW would be 10.98tb of data. After de-duplication and compression it only takes up 407.98gb on disk.

With that kind of space savings, I see no reason not to keep such frequent backups. Hell, the whole archive takes up less space than one copy of the original data.

[–] [email protected] 3 points 3 days ago* (last edited 3 days ago)

+1 for borg


                   Original size      Compressed size    Deduplicated size

This archive: 602.47 GB 569.64 GB 15.68 MB All archives: 16.33 TB 15.45 TB 607.71 GB

                   Unique chunks         Total chunks

Chunk index: 2703719 18695670

[–] [email protected] 5 points 4 days ago

Thanks for sharing the details on this, very interesting!

[–] [email protected] 21 points 4 days ago (1 children)
[–] [email protected] 4 points 4 days ago

Yep. Even if the data I'm backing up doesn't really change that often. Perhapas I should start to back up files from my laptop and workstation too. Nothing too important is stored only on those devices, but reinstalling and reconfiguring everything back is a bit of a chore.

[–] [email protected] 18 points 4 days ago (1 children)
[–] [email protected] 4 points 4 days ago (2 children)
[–] [email protected] 32 points 4 days ago (3 children)

That is what the B in RAID stands for.

[–] [email protected] 6 points 4 days ago

Just like the “s” in IoT stands for “security”

[–] [email protected] 5 points 4 days ago
[–] [email protected] 4 points 4 days ago (1 children)

What's the second B stand for?

[–] [email protected] 4 points 4 days ago (1 children)

Beets.

Or bears.

Or buttsex.

It’s context dependent, like “cool”.

[–] [email protected] 3 points 4 days ago
[–] [email protected] 5 points 4 days ago* (last edited 4 days ago)

If Raid is backup, then Unraid is?

[–] [email protected] 10 points 4 days ago* (last edited 4 days ago) (1 children)

Proxmox servers are mirrored zpools, not that RAID is a backup. Replication between Proxmox servers every 15 minutes for HA guests, hourly for less critical guests. Full backups with PBS at 5AM and 7PM, 2 sets apiece with one set that goes off site and is rotated weekly. Differential replication every day to zfs.rent. I keep 30 dailies, 12 weeklys, 24 monthly and infinite annuals.

Periodic test restores of all backups at various granularities at least monthly or whenever I'm bored or fuck something up.

Yes, former sysadmin.

[–] [email protected] 2 points 2 days ago (1 children)

This is very similar to how I run mine, except that I use Ceph instead of ZFS. Nightly backups of the CephFS data with Duplicati, followed by staggered nightly backups for all VMs and containers to a PBS VM on a the NAS. File backups from unraid get sent up to CrashPlan.

Slightly fewer retention points to cut down on overall storage, and a similar test pattern.

Yes, current sysadmin.

[–] [email protected] 1 points 2 days ago* (last edited 2 days ago)

I would like to play with ceph but I don't have a lot of spare equipment anymore, and I understand ZFS pretty well, and trust it. Maybe the next cluster upgrade if I ever do another one.

And I have an almost unhealthy paranoia after see so many shitshows in my career, so having a pile of copies just helps me sleep at night. The day I have to delve into the last layer is the day I build another layer, but that hasn't happened recently. PBS dedup is pretty damn good so it's not much extra to keep a lot of copies.

[–] [email protected] 12 points 4 days ago

I do not as I cannot afford the extra storage required to do so.

[–] [email protected] 12 points 4 days ago

Daily backups here. Storage is cheap. Losing data is not.

[–] [email protected] 7 points 4 days ago

I use Duplicati for my backups, and have backup retention set up like this:

Save one backup each day for the past week, then save one each week for the past month, then save one each month for the past year.

That way I have granual backups for anything recent, and the further back in the past you go the less frequent the backups are to save space

[–] [email protected] 3 points 4 days ago

rsync from ZFS to an off-site unraid every 24 hours 5 times a week. on the sixth day it does a checksum based rsync which obviously means more stress so only do it once a week. the seventh day is reserved for ZFS scrubbing every two weeks.

[–] [email protected] 4 points 4 days ago* (last edited 4 days ago)

Right now, I have a cron job set to run on Monday and Friday nights, is this too frequent?

Only you can answer this. How many days of data are you prepared to lose? What are the downsides of running your backup scripts more frequently?

[–] [email protected] 6 points 4 days ago (1 children)

Every hour, automatically

Never on my Laptop, because I'm too lazy to create a mechanism that detects when it's possible.

[–] [email protected] 4 points 4 days ago

I just tell it to back up my laptops every hour anyway. If it’s not on, it just doesn’t happen, but it’s generally on enough to capture what I need.

[–] [email protected] 3 points 4 days ago

Backup all of my proxmox-LXCs/VMs to a proxmox backup server every night + sync these backups to another pbs in another town. A second proxmox backup every noon to my nas. (i know, 3-2-1 rule is not reached...)

[–] [email protected] 3 points 4 days ago

I have

  • Unraid back up it's USB
  • Unraid appears gets backed up weekly by a community applications (CA app backup) and I use rclone to back it up to an old box account (100GB for life..) I did have it encrypted but seems I need to fix that..
  • Parity drive on my Unraid (8TB)
  • I am trying to understand how to use Rclone to back up my photos to Proton Drive so that's next.

Music and media is not too important yet but I would love some insight

[–] [email protected] 5 points 4 days ago

Nextcloud data daily, same for the docker configs. Less important/rarely changing data once per week. Automatic sync to NAS and online storage. Irregular and manual sync to an external disk.

7 daily backups, 4 weekly backups, "infinite" monthly backups retained (until I clean them up by hand).

[–] [email protected] 5 points 4 days ago

Boils down to how much are you willing to lose? Personally I do weekly

[–] [email protected] 4 points 4 days ago

If you haven't tested your backups, you ain't got a backup.

[–] [email protected] 3 points 4 days ago

I classify the data according to its importance (gold, silver, bronze, ephemeral). The regularity of the zfs snapshots (15 minutes to several hours) and their retention time (days to years) on the server depends on this. I then send the more important data that I cannot restore or can only restore with great effort (gold and silver) to another server once a day. For bronze, the zfs snapshots and a few days of storage time on the server are enough for me, as it is usually data that I can restore (build artifacts or similar) or is simply not that important. Ephemeral is for unimportant data such as caches or pipelines.

[–] [email protected] 4 points 4 days ago (2 children)

Local zfs snap every 5 mins.

Borg backups everything hour to 3 different locations.

I've blown away docker folders of config files a few times by accident. So far I've only had to dip into the zfs snaps to bring them back.

load more comments (2 replies)
[–] [email protected] 3 points 4 days ago

No backup for my media. Only redundacy.

For my nextcloud data, anytime i made major changes.

[–] [email protected] 2 points 4 days ago

Assuming it is on: Daily

[–] [email protected] 3 points 4 days ago

I have a cron job set to run on Monday and Friday nights, is this too frequent?

Only you can answer that - what is your risk tolerance for data loss?

[–] [email protected] 3 points 4 days ago* (last edited 4 days ago)

Most backup software allow you to configure backup retention. I think I went with some pretty standard once per day for a week. After that they get deleted, and it keeps just one per week of the older ones, for one or two months. And after that it's down to monthly snapshots. I think that aligns well with what I need. Sometimes I find out something broke the day before yesterday. But I don't think I ever needed a backup from exactly the 12th of December or something like that. So I'm fine if they get more sparse after some time. And I don't need full backups more than necessary. An incremental backup will do unless there's some technical reason to do full ones.

But it entirely depends on the use-case. Maybe for a server or stuff you work on, you don't want to lose more than a day. While it can be perfectly alright to back up a laptop once a week. Especially if you save your documents in the cloud anyway. Or you're busy during the week and just mess with your server configuration on weekends. In that case you might be alright with taking a snapshot on fridays. Idk.

(And there are incremental backups, full backups, filesystem snapshots. On a desktop you could just use something like time machine... You can do different filesystems at different intervals...)

[–] [email protected] 3 points 4 days ago

Weekly full backup, nightly incremental

[–] [email protected] 3 points 4 days ago (1 children)

Timeshift creates a btrfs snapshot on each boot for me. And my server gets nightly borg backups.

[–] [email protected] 5 points 4 days ago (3 children)

Just a friendly reminder that BTRFS snapshots are not backups.

[–] [email protected] 3 points 4 days ago (2 children)

You're correct and probably the person you're responding to is treating one as an alternative as another.

However, theoretically filesystem snapshotting can be used to enable backups, because they permit for an instantaneous, consistent view of a filesystem. I don't know if there are backup systems that do this with btrfs today, but this would involve taking a snapshot and then having the backup system backing up the snapshot rather than the live view of the filesystem.

Otherwise, stuff like drive images and database files that are being written to while being backed up can just have a corrupted, inconsistent file in the backup.

[–] [email protected] 3 points 4 days ago (1 children)

btrbk works that way essentially. Takes read-only snapshots on a schedule, and uses btrfs send/receive to create backups.

There's also snapraid-btrfs which uses snapshots to help minimise write hole issues with snapraid, by creating parity data from snapshots, rather than the raw filesystem.

load more comments (1 replies)
[–] [email protected] 2 points 4 days ago

Absolutely, my backup solution is actually based on BTRFS snapshots. I use btrbk (already mentioned in another reply) to take the snapshots and copy them to another drive. Then a nightly restic job backs up the latest snapshot to B2.

load more comments (2 replies)
[–] [email protected] 2 points 4 days ago

Daily toward all my three locations:

  • local on the server
  • in-house but on a different device
  • offsite

But not all three destinations backup the same amount of data due to storage limitations.

[–] [email protected] 2 points 4 days ago (2 children)

I continuous backup important files/configurations to my NAS. That's about it.

IMO people who redundant/backup their media are insane... It's such an incredible waste of space. Having a robust media library is nice, but there's no reason you can't just start over if you have data corruption or something. I have TB and TB of media that I can redownload in a weekend if something happens (if I even want). No reason to waste backup space, IMO.

[–] [email protected] 2 points 4 days ago (1 children)

Maybe for common stuff but some dont want 720p YTS or yify releases.
There are also some releases that don't follow TVDB aired releases (which sonarr requires) and matching 500 episodes manually with deviating names isn't exactly what I call 'fun time'.
Amd there are also rare releases that just arent seeded anymore in that specific quality or present on usenet.

So yes: Backup up some media files may be important.

[–] [email protected] 1 points 3 days ago (1 children)

Data hoarding random bullshit will never make sense to me. You're literally paying to keep media you didn't pay for because you need the 4k version of Guardians of the Galaxy 3 even though it was a shit movie...

Grab the YIFY, if it's good, then get the 2160p version... No reason to datahoard like that. It's frankly just stupid considering you're paying to store this media.

[–] [email protected] 1 points 3 days ago (2 children)

This may work for you and please continue doing that.

But I'll get the 1080p with a moderate bitrate version of whatever I can aquire because I want it in the first place and not grab whatever I can to fill up my disk.

And as I mentioned: Matching 500 episodes (e.g. Looney Tunes and Disney shorts) manually isnt fun.
Much less if you also want to get the exact release (for example music) of a certain media and need to play detective on musicbrainz.

load more comments (2 replies)
[–] [email protected] 2 points 4 days ago (1 children)

It becomes a whole different thing when you yourself are a creator of any kind. Sure you can retorrent TBs of movies. But you can't retake that video from 3 years ago. I have about 2 TB of photos I took. I classify that as media.

[–] [email protected] 1 points 3 days ago

It becomes a whole different thing when you yourself are a creator of any kind.

Clearly this isn't the type of media I was referencing....

load more comments
view more: next ›