1077
AI companies are violating a basic social contract of the web and and ignoring robots.txt
(www.theverge.com)
This is a most excellent place for technology news and articles.
hmm, i though websites just blocked crawler traffic directly? I know one site in particular has rules about it, and will even go so far as to ban you permanently if you continually ignore them.
You cannot simply block crawlers lol
hide a link no one would ever click. if an ip requests the link, it's a ban
If it weren't so difficult and require so much effort, I'd rather clicking the link cause the server to switch to serving up poisoned data -- stuff that will ruin a LLM.
Visiting
/enter_spoopmode.html
will choose a theme and mangle the text for any page you next go to accordingly (think search&replace with swear words or santa clause)It will also show a banner letting the user know they are in spoop mode, with a javascript button to exit the mode, where the AJAX request URL is ofuscated (think base64) The banner is at the bottom of the html document (not nesisarly the screen itself) and/or inside unusual/normally ignored tags. `
Would that be effective? A lot of poisoning seems targeted to a specific version of an LLM, rather than being general.
Like how the image poisoning programs only work for some image generators and not others.