avidamoeba

Just to be clear, what I'm referring to here is that a search would occur on a single instance. E.g. searches on lemmy.world occur on the lemmy.world instance, and load lemmy.world's servers. The federated part is in the building the database on lemmy.world. E.g. a crawler or a user on lemmy.ca adds a new web site and that record is federated to lemmy.world to add to its database. Another user on feddit.de upvotes a search result and that upvote is federated to lemmy.world so that the search result shows higher for users searching on lemmy.world. In this kind of model individual search instances could in fact be very large based on their usage. If there's no limit to what's federated, that would put a lower bound on the size of instances. If there's a limit (something dumb like federate only search records for *.fr domains) then that would allow for smaller instances that don't have the compute and storage for the complete index.

You're right, raidz expansion is brand new and I probably wouldn't use it for a few years. I was referring to adding new redundant vdevs to an existing pool which has always been supported as far as I know. E.g. if you have an existing raidz or mirror, you can add another raidz or mirror vdev to the pool. The pool size grows with the usable size of the new vdev. It's just zpool add thepool mirror disk1 disk2 as far as I know. The downside being it results in less usable space - e.g. two raidz1 vdevs remove 2 disks from the usable space, whereas Unraid-raid would remove 1. For example if you have 3x 3TB and 3x 4TB disks, you'd end up with 14TB usable space with ZFS and 17TB with Unraid. On the flip side, the two raidz1 vdevs would have higher reliability since you can have one disk die in each vdev.

just clicking a button on a webUI.

No question. I think TrueNAS offers this too.

ZFS also recently had a major data loss bug so I’m not sure safer is accurate.

Imagine how many of those would be found in Unraid-raid if it was used as widely and for similar loads as ZFS. My argument isn't that there aren't bugs in storage systems. There are, and the more eyes have seen the code and the more users have lost data for more years, the fewer bugs would remain. Assuming similar competence of the system developers, ZFS being much older and ran for production loads makes it more likely to contain fewer data eating bugs than Unraid.

🐸

Not sure how traditional-traditional (hw RAID?) you're referring to but you can use different disk sizes as well as grow LVM/mdraid or ZFS. It does indeed require a bit more thought and reading to do well. On the upside it's probably much safer (for data integrity) and more performant.

This is what I meant. ☝️ If they had merely wrapped LVM/mdraid or ZFS in a nice packaging my argument wouldn't stand. They would have had equivalent data reliability to TrueNAS.

As a software developer (who's looked at ZFS' source to chase a bug,) I would not dare to write my own redundant storage system. I feel like storage is a complex area with tons of hard-learned gotchas, and similar to cryptography, a best practice is to not roll your own unless truly necessary. This is not your run-of-the-mill web app and mistakes eat data. Potentially data with bite marks that gets backed up, eventually fully replacing the original before it's caught. I don't have data for this but I bet the proportion of Unraid users with eaten data from the total Unraid userbase is significantly higher than the equivalent for solutions using industry standard systems. The average web UI user probably isn't browsing through their ods/xlsx files regularly to check whether some 5 became a 13.

I think their scheme does fall under the RAID definition. I don't think being able to access individual drives is something that distinguishes RAID from not-RAID since there are standard RAID schemes in which you can access the data in individual drives.

Than Unraid or DIY? I've considered SCALE but I've never used it. I know they use ZFS so it will almost certainly eat less data than Unraid's homegrown raid.

Yeah, I guess that's the niche. I would still not trust their homegrown raid scheme though. Making storage systems that don't eat data is hard. Making it without bugs is impossible. Bugs are found by having someone's data eaten and fixed over time scaled by the size of the userbase. As a result industry standard systems like mdraid, LVM, ZFS, and more recently Btrfs used in data centers and production applications are statistically guaranteed to eat less data than Unraid's homegrown solution. I've heard it now supports those systems too so if I had to use Unraid, I'd probably be using ZFS for the storage.

Yeah, I've read about that but I couldn't buy it because you could achieve similar results with LVM, ZFS etc. albeit with a bit more thought. For example I used to have a mirror (RAID1) comprised of 1TB, 3TB, 4TB and an 8TB disks. The 1, 3 and 4TB disks were concatenated in an 8TB linear volume (JBOD) and then that was mirrored with the 8TB disk (RAID1). All using standard battle tested software - LVM, mdraid and Ext4. I got 8TB usable from it. I'd have gotten the same in Unraid. The redundancy was equivalent too. With ZFS things are even simpler. Build whatever redundant scheme you have disks for. Use whatever redundancy scheme makes sense for those disks. You could combine multiple schemes. E.g. 1TB + 1TB mirror and a RAIDz1 with 3x 3TB disks, all adding to 7TB of nice contiguous usable space with all the data integrity guarantees of ZFS. Heck if you need to do some 3-disks-in-a-trenchcoat trickery to utilize your obsolete hardware like I did, you can use LVM for that and give it to ZFS to use. When you're ready to expand, buy disks for whatever redundancy scheme you like and just add it to your ZFS pool. No fuss. You like living dangerously? Add disks without redundancy. Can't afford redundancy now but you'd like it later? Add disks without redundancy now, add redundancy later.

Truly. I wonder if ActivityPub could be utilized to create a resilient search engine that shares the cost among federated instances. We already have something like that in Lemmy and Mastodon where federated data can be search from any instance. If the data is pages crawled by some automatic crawler which is then federated across instances which in turn allow to search through it, perhaps it might resemble a search engine. Page ranking beyond text matching could even be done by peoples up/down votes instead of some arbitrary algorithm. Similar to how voting works on StackExchange or Lemmy. 🤔 I'm sure someone is thinking about this.

Perhaps it's becoming clear that search needs to become a common cooperatively managed infrastructure similar to Wikipedia. That this is in the best interest of everyone but advertisers and spammers.

I'm sure there are reasons for using Unraid but the original funky raid alternative they marketed has always struck me as extremely fishy. The kind of solution developed by folks who didn't know enough about the best practices in storage and decided to roll their own. I guess people like web interfaces too. Personally I'd never use it. Get Debian Stable or Ubuntu LTS, learn some Docker, Ansible and Prometheus, deploy and never touch until you break it or the hardware breaks. Throw Webmin on it if you like dancing bears too.