Technology

Thanks for added background! I haven't been monitoring this area very closely so wasn't aware, but I'd have thought a publication that has been would then be more skeptical and at least mention some of this, particularly highlighting disputes over the efficacy of the Glaze software. Not to mention the others they talked to for the article.

Figures that in a space rife with grifters you'd have ones for each side.

who illegally stole GPLv3 code from an open source program called DiffusionBee for his proprietary Glaze software (reddit link), and when pressed, only released the code for the “front end” while still being in violation of GPL?

Oh, how I wish the FSF had more of their act together nowadays and were more like the EFF or ACLU.

Thank you, Margot Robbie! I'm a big fan!

They don't even need to detect them - once they are common enough in training datasets the training process will "just" learn that the noise they introduce are not features relevant to the desired output. If there are enough images like that it might eventually generate images with the same features.

Trying to detect poisoned images is the wrong approach. Include them in the training set and the training process itself will eventually correct for it.

I think if you build more robust features

Diffusion approaches etc. do not involve any conscious "building" of features in the first place. The features are trained by training the net to match images with text features correctly, and then "just" repeatedly predict how to denoise an image to get closer to a match with the text features. If the input includes poisoned images, so what? It's no different than e.g. compression artifacts, or noise.

These tools all try to counter models trained without images using them in the training set with at most fine-tuning, but all they show is that models trained without having seen many images using that particular tool will struggle.

But in reality, the massive problem with this is that we'd expect any such tool that becomes widespread to be self-defeating, in that they become a source for images that will work their way into the models at a sufficient volume that the model will learn them. In doing so they will make the models more robust against noise and artifacts, and so make the job harder for the next generation of these tools.

In other words, these tools basically act like a manual adversarial training source, and in the long run the main benefit coming out of them will be that they'll prod and probe at failure modes of the models and help remove them.

Haven't read the paper so not sure about the specifics, but if it relies on subtle changes, would rounding color values or down sampling the image blur that noise away?

So tl;dr he/his team did two things:

1. argue the way AI uses content to train is legal
2. provide artists a tool to prevent their content being used to train AI without their permission

On the surface it sounds all good, but I can't help but notice a future conflict of interest for Zhao should Glaze ever become monetized. If it were to be ruled illegal to train AI on content without permission, tools like Glaze would be essentially anti-theft devices, but while it remains legal to train AI this way, tools like Glaze stand to perhaps become necessary for artists to maintain the pre-AI status quo w/r/t how their work can be used and monetized.

Speaking of jpeg I miss the "needs more jpeg" bot that used to run on reddit, that shit was hilarious.

No, that's something else entirely. Adversarial training is where you put an ai against a detector AI as a kind of competition for results.

Its called adversarial attack, this is an old video (5 years) explaining how it works and how you can potentially do it charging just one pixel on the image.

https://youtu.be/SA4YEAWVpbk?si=xObPveXTT2ip5ICG

"We like to call them poison pixels."

Or the patterned vinyl wraps they used on test cars that interferes with camera autofocus.

"Invisible changes to pixels" means "a human can't tell the difference with a casual glance" - you can still embed a shit-ton of data in an image that doesn't look visually like it's been changed without careful inspection of the original and the new image.

If this data is added in certain patterns it will cause ML models trained against the image to draw incorrect conclusions. It's a technical hurdle that will slow a casual adversary, someone will post a model trained to remove this sometime soon and then we'll have a good old software arms race and waste a shit ton of greenhouse emissions adding and removing noise and training ever more advanced models to add and remove it.

You can already intentionally poison images so that image recognition draws incorrect conclusions fairly easily, this is the same idea but designed to cripple ML model training.

I'm sure others know more about it than i do but I thought pixels were very simple things.

You're right, in that pixels are very simple things. However, you and I can't tell one pixel from another in an image, and at the scale of modern digital art (my girlfriend does hers at 300dpi), shifting a handful of pixels isn't going to make much of a visible difference to a person, but a LLM will notice them.