Technology

Scraper bots don't read instructions, they just follow links

Right‽ This is ridiculously stupid when you can download the entirety of Wikipedia in a single package and parse it to your hearts desire

The amount of stupid AI scraping behavior I see even on my small websites is ridiculous, they'll endlessly pound identical pages as fast as possible over an entire week, apparently not even checking if the contents changed. Probably some vibe coded shit that barely functions.

If I was running infra for them, I’d just start blacklisting abusive IPs without warning

An HTTP request is a request. Servers are free to rate limit or deny access

i doubt the recent uptick in traffic is from “stealing data” for training but rather from agents scraping them for context, eg Edge Copilot, Google’s AI search, SearchGPT, etc.

poisoning the data will likely not help in this situation since there’s a human on the other side that will just do the same search again given unsatisfactory results. like how retries and timeouts can cause huge outages for web scale companies, poisoning search results will likely cause this type of traffic to increase and further increase the chances of DoS and higher bandwidth usage.

Not in this case, to be fair. The only concern is cost - since Wiki wouldn't be opposed to them getting their actual data - and AI mazes are designed to safeguard more sensitive data, not reducing cost

I still struggle with a use case for artificial intelligence in my own life. I play around with it all and I'm just like, it doesn't do a good job. Also, I think humanity is missing the plot, you know? Like, we don't need government. If government isn't going to do government. Government serves the people, not corporations. Or at least it should. I don't know, I think we're entering in times. At some point, I think people will pray for nuclear war, because life will be so miserable. That it would be better than just to end it all.

even better stop joining major platforms like social media and then they won't be able to create data sets. Be a leach, especially when they give it away for free, but don't contribute to the project. Understand how it works, sure. But it seems like most of humanity says they don't want something, yet they do the contrary. It's like we choose to comply before we even ask to comply for the fear of missing out. But if you look at what is today, what are you really missing out on?

The other day I tried to have it help me with a programming task on a personal project. I am an experienced programmer, but I only "get by" in Python (typically just by looking up the documentation for the standard library). I thought, "OK. This is it. I will ask Llama 3.3 and GPT4 for help."

That shit literally set me back a weekend. It gave me such bad approaches and answers, that I could tell were bad (aforementioned experience in programming, degree in comp sci, etc) that I got confused about writing Python. Had I just done what I usually do, which is to look up the documentation and use my brain, I would have gotten my weekend task done a whole weekend sooner.

It scares me to think what people are doing to themselves by relying on this, especially if they're novices.

AI is useful for basic, mundane tasks and that's about it. Trying to force it to be some sort of Uber search engine is such a bad idea.

I recently started as a graphic designer despite knowing absolutely nothing about it, so i am constantly searching how to do stuff in Adobe suite at work. Half the time Google's AI can't even keep "Cmnd" and "ctrl" straight, telling me to use' "cmnd+shift+H" on Windows or "ctrl+shift+H" on Mac'. I don't even know how it botches that, but it does it about 25% of the time.

AI bros aren't that smart.

Apparently the dump doesn't include media, though there's ongoing discussion within wikimedia about changing that. It also seems likely to me that AI scrapers don't care about externalizing costs onto others if it might mean a competitive advantage (e.g. most recent data, not having to spend time and resources developing dedicated ingestion systems for specific sites).

I want to stress this: it's not that "tech bros" are just stupid—even though a lot of them are revoltingly unappreciative of the giants whose sholders they stand on—it's that they don't care.

Well I get the analogy, but also I think they didn't use pencils because of the graphite and complications with filtering air or something.

This pen / pencil thing has been corrected so many times for so many decades that it's ludicrous people are still bringing it up.

https://www.scientificamerican.com/article/fact-or-fiction-nasa-spen/

Random bits of pencil lead floating around in a high tech environment is such a poor idea that even the Soviet's quit using pencils once Fisher's Space Pen was available. A pen which Fisher itself paid to develop and then sold to both NASA and the Soviet Space Program.

Pen, pencil. Both are dangerous in the wrong hands.