this post was submitted on 17 Mar 2024
244 points (81.8% liked)
Privacy
31975 readers
248 users here now
A place to discuss privacy and freedom in the digital world.
Privacy has become a very important issue in modern society, with companies and governments constantly abusing their power, more and more people are waking up to the importance of digital privacy.
In this community everyone is welcome to post links and discuss topics related to privacy.
Some Rules
- Posting a link to a website containing tracking isn't great, if contents of the website are behind a paywall maybe copy them into the post
- Don't promote proprietary software
- Try to keep things on topic
- If you have a question, please try searching for previous discussions, maybe it has already been answered
- Reposts are fine, but should have at least a couple of weeks in between so that the post can reach a new audience
- Be nice :)
Related communities
Chat rooms
-
[Matrix/Element]Dead
much thanks to @gary_host_laptop for the logo design :)
founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I wonder what a distributed search engine would look like. Basically, the index would be sharded across user computers, and queries would hit some representative sample of that index. This means:
My biggest concern is how to build the index, but if OP is willing to share that, I might start hacking on a distributed version.
Don't start new; contribute to what already exists: https://en.wikipedia.org/wiki/YaCy
Awesome! That's pretty much exactly what I'm looking for, though I'm interested to see how easy it is limit certain peers to certain functions. Not everyone has resources to crawl and index pages, but a lot of people can store the index.
I'm interested in having client-side web storage, so you can participate in the network by just having the search page open (opt-in of course).
I'm honestly not actively working on it, but if OP provides the database and/or crawler, I'll do some research on feasibility.
This is really neat and I’m just hearing about it after over twenty years of development. I need to try it out, thank you. How do you stay in the know about this kind of stuff? I’m curious about all the cool stuff out there I wouldn’t even know I’m curious to find.
By being terminally online, I guess?
More concretely, I've spent (probably too much) time on Slashdot, Reddit and now Lemmy over the years (subscribed to Free Software and privacy-related communities in particular). Also, looking through sites like https://awesome-selfhosted.net/ and https://www.privacytools.io/, wiki-walking through articles about Free Software projects on Wikipedia, browsing the Debian repositories, etc.
I'm sure there are plenty of things I haven't heard of either, though.
I was going to mention YaCy as well if nobody else was, so I can chip in to this somewhat. My method is to keep wondering and researching. In this case it was a matter of being interested in alternative search engines and different applications of peer to peer/decentralized technologies that led me to finding this.
So from this you might go: take something you're even passingly interested in, try to find more information about it, and follow whatever tangential trails it leads to. With rare exceptions, there are good chances someone out there on the internet will also have had some interest in whatever it is, asked about it, and written about it.
Also be willing to make throwaway accounts to get into the walled gardens for whatever info might be buried away there and, if you think others may be interested, share it outside of those spaces.
Isn't that what Searx is/can be?
https://en.wikipedia.org/wiki/Searx#Instances
I admit it's not something I've looked closely at.
No, Searx is a metasearch engine that queries and aggregates results from multiple normal search engines (Google, Bing, etc.)
A distributed search engine would be more like YaCy, which does its own crawling and stores the index as a distributed hash table shared across all instances.
Ah thanks - appreciate the clarification.
Exactly. The main difference I would bring is a web client that hooks into the network, and perhaps an alternative client (e.g. I'm interested in Tauri, so I may rewrite part of the BE to Rust).
But I'm probably not going to start on this project on my own. DDG is good enough for now, so I'm putting my efforts elsewhere.
i feel that decentralized search is an extremely valuable thing to start thinking about. but the devil is in practically every one of the details.
Yup. Even if you trust all your peers (which isn't reasonable), there's still a ton of practical issues that need to be resolved:
It's a really complex problem without getting p2p involved, and p2p just adds a ton of other problems.
So I'm probably going to stick with building my Reddit clone, which I think is simpler (search doesn't need to happen at the start).