this post was submitted on 26 Oct 2024
106 points (94.2% liked)

Piracy: ꜱᴀɪʟ ᴛʜᴇ ʜɪɢʜ ꜱᴇᴀꜱ

55085 readers
471 users here now

⚓ Dedicated to the discussion of digital piracy, including ethical problems and legal advancements.

Rules • Full Version

1. Posts must be related to the discussion of digital piracy

2. Don't request invites, trade, sell, or self-promote

3. Don't request or link to specific pirated titles, including DMs

4. Don't submit low-quality posts, be entitled, or harass others



Loot, Pillage, & Plunder

📜 c/Piracy Wiki (Community Edition):


💰 Please help cover server costs.

Ko-Fi Liberapay
Ko-fi Liberapay

founded 2 years ago
MODERATORS
 

So, I was having a phone conversation with my boss yesterday. The topic was a clustered filesystem that can hold huge amounts of data, and how we would best allow local users to access this data without yhe storage cluster becoming overloaded (because of various reasons, the I/O is relatively slow. This was built for quantity, not speed).

Rights now there's an SMB share, and we're looking at replacing that so that we can have better control over the data throughput. My suggestion is to simply spin up an FTP server.

Then my boss asks: "I'm just curious, but would rsync or NFS work as a protocol instead?"

Well, it's a valid question, so the only thing I could do was reply with the honest answer as to why I chose FTP. Paraphrased and translated:

"Because some 20 years ago my then username carried a lot of recognition in certain communities revolving around software and media distribution, whose rights holders would not necessarily approve of said distribution. We used FTP, because when you're on an ADSL from 2002, you want to have as much fine control as you can to make sure your internet connection doesn't get flooded with requests. One connection at a time, and only one file at a time, which would be ideal in our particular case."

The response I got was a chuckle and that he couldn't think of a better endorsement of FTP as a preferred transfer protocol.

So there you have it - My career revolves a lot of skills that I picked up whole sailing the high seas. And coincidentally, my career now also involves literally sailing the high seas as these storage clusters are used on survey ships.

top 8 comments
sorted by: hot top controversial new old
[–] [email protected] 39 points 2 months ago (1 children)

Lol @ "some 20 years ago ... ADSL from 2002". Thanks for making me feel old!

[–] [email protected] 7 points 2 months ago

Horses were domesticated some 6000 years ago. I feel so old!

[–] [email protected] 8 points 2 months ago (1 children)
[–] [email protected] 10 points 2 months ago (1 children)
[–] [email protected] 2 points 2 months ago

Do check seaweedfs too! Haven't tried it (yet) but their 'erasure coding' reads as super sophisticated to me ;)

I wonder how it compares to beegfs

[–] [email protected] 5 points 2 months ago* (last edited 2 months ago)

Using FTP (I assume you mean SFTP) will buy you some performance, as would other protocols that are faster and requiring less compute than SMB.

I predict whatever solution you use will only buy you time. Usage is bound to increase so you'll still hit the performance limits for the hardware platform at some point, unless you can constrain the simultaneous connections. File sizes will impact scalability a lot as well.

You can't guess this one. You need to test.

tl;dr - I suspect you can't win.

[–] [email protected] 1 points 2 months ago* (last edited 2 months ago) (1 children)

Was IPFS considered? I've tried it myself but it seems like an unstable product and I'm not sure if it's living up to its promise...

Anti Commercial-AI license

[–] [email protected] 2 points 2 months ago

Unusable in our case