Benn Jordan's AI poison pill and the weird world of adversarial noise

To add, all the new audio models (partially) use diffusion methods that are exactly the same methods as used on images - the audio generation can be thought of as an image generation of a spectrogram of an audio file.

For early experiments people literally took Stable Diffusion and fine tuned it on labelled spectrograms of music snippets, then used the fine tuned model to generate new images of spectrograms guided by text, and then took those images and turned them back into audio via re-synthesis of that spectral image to a .wav.

Riffusion was one of the first to experiment with this, 2 years ago now: https://github.com/riffusion/riffusion-hobby

The more advanced music generators out now I believe have more of a 'stems' approach and a larger processing pipeline to increase fidelity and add tracking vocal capability but the underlying idea is the same.

Any adversarial attack to hide information in the spectrograph to fool the model into categorizing the track as something it is not isn't different than the image adversarial attacks which have been found to have ways to be mitigated.

Various forms of filtering for inaudible spectral information coupled with methods that destroy and re-synthesize/randomize phase information would likely break this poisoning attack.

The short answer is that they are applying the same defense to audio as to images, and so we should expect that the same attacks will work as well.

More specifically, there are a few moving parts here - the GenAI model they're trying to defeat, the defense applied to data items, and the data cleaning process that a GenAI company may use to remove the defense. So we can look at each and see if there's any reason to expect things to turn out differently than they did in the image domain. The GenAI models follow the same type of training, and while they of course have slightly different architectures to ingest audio instead of images, they still use the same basic operations. The defenses are exactly the same - find small perturbations that are undetectable to humans but produce a large change in model behavior. The cleaning processes are not particularly image-specific, and translate very naturally to audio. It's stuff like "add some noise and then run denoising".

Given all of this, it would be very surprising if the dynamics turned out to be fundamentally different just because we moved from images to audio, and the onus should be on the defense developers to justify why we should expect that to be the case.

>o it seems a logical leap to say they know it doesn't and are doing this as a scheme?

In some of the earlier image protection articles the people involved seemed rather shady about the capabilities. Would have to do some HN searching for those articles.

But everything at the end of the day will be a scheme if the end result is for humans to listen to it. You cannot make a subset of music that can be heard by humans (and actually sounds good) that cannot be prefiltered to be learned by AI. I've said the same thing about images, the same thing will be true about audio, movies, actions in real leave, et al.

These schemes will likely work for a few of the existing models, then fall apart quickly the moment a new model arrives. What is worse for defense is audio quality for humans is remaining the same while GPU speeds and algorithms increase in speeds over time meaning the time until a model beats the new defense will trend to unity.

Are you sure you mean "stealing"? As in deprive him of his own recordings?

I am curious if anyone read Harry Potter in bootleg form from a LLM. I mean, LLMs are the worst tools for infringing - they are approximate, expensive and slow, while copying is instant, perfect and free. You can apply the same logic for other modalities.

Moreover, who's got the time to see someone else's AI shit when they can generate their own, perfectly customized to their liking? I personally generated a song about my cat and kid. It had zero commercial value but was fun for 2-3 people to listen.

They’ve been doing that since the recording studio process developed the model in the 1920s or so. They would hire songwriters to make generic pop music with generic lyrics, and keep it in the vault until you have some attractive young singer you want to use for marketing then you give them an album of these songs to sing. And they are sure to sell because you’ve been priming the american ear for these chord progressions for a long time, and you fill all the air in the room with your marketing for this singer leaving people little option but to hear the latest carefully crafted earworm. Still happens today maybe even more perfected with psychological studies intersecting with music and marketing. The best musicians have never and will never be a product of that machine. Seek out live music.

Soulless cheap Muzak already exists and has for a long time.

Any musician these days that thinks there is money in music by selling songs is delusional. Sad but true.

But it's not intended as a watermark, it's an attempt at disruption. And with some models it simply doesn't work.

For instance, I've seen somebody experiment with Glaze (the image AI version of this). Glaze at high levels produces visible artifacts (see middle image: https://pbs.twimg.com/media/FrbJ9ZTacAAWQQn.jpg:large ).

It seems some models ignore it and produce mostly clean images on the output (looking like the last image), while others just interpret is as a texture, the character is just wearing a funny patterned shirt. This is while the intended result is fooling the model to generate something other than the intended character.

My point is that this entire situation has to be framed in the narrative that copyright demands it be framed in. It's "you" the participant of copyright.

When someone creates art, copyright says that there is a countable result we can refer to as their "work". Copyright also says that that artist has a monopoly over the distribution and sale of that work. The implication is that the way for an artist to get paid for their labor is for them to leverage the monopoly they have been granted, and negotiate a distribution scheme that involves paying them.

When an artist chooses to work outside the copyright model, that means they must predetermine part of their distribution negotiation. That might be the libertarian option (gratis distribution with no demands), or it might be the copyleft option, where the price is demanded, but also set to 0. The artist may find payment for their labor by other means, but that's challenging to do in an economy where copyright participants dominate.

Indeed! I've definitely been a fan of his for a while, and I laud him for trying to make things work in a space where all the cards are stacked against him.

I do wish, though, that he would have introduced that perspective of the situation in this particular video. Leaving it out feels like making a video about learning to swim, set in the middle of the ocean.

> Exclusive rights over their published work encourages artists and inventors to publish their work

Do they? Please cite your studies.

People have been publishing their work for thousands of years before IP even existed. We don't know if the system we have is the best one we could have, I doubt it though.

This! Benn Jordan hasn't published anything in months because of scrapers for AI modeling.

>They don't, what's happening here is their music is being fed to a computer program in a for-profit venture.

I believe that no artist has the right to tell anyone what to do with the art they have published. It does not matter what happens inside the algorithm with that art. Whether NNs (LLMs, by definition are about language) learn like humans or not is totally irrelevant to my point.

I am not against IP because it does or doesn't benefit artists. I am against the idea because it does not make sense. It gives ownership and control over imaginary things to people, a song you create and public isn't "yours". You do not get to decide what others do with it and how they use it.

I believe that artists see the current IP laws critically, of course they do as it directly impacts how they finance themselves and how they bargain. But I do not care how good/bad the bargain for the artist is. IP laws should be abolished regardless of what artists want.

Yes.

I also pirate every single book I read. Sometimes I buy them though.

I also pirate every single show I watch. I never buy them.

Music is a bit difficult, but I pay for Spotify, but I wouldn't mind paying for the service if Spotify had no rights to the songs and wasn't compensating the artists.

I believe that a creator has no right to dictate what other people do with his creations.

I thought this was the most common anti-IP sentiment.

It makes me so uncomfortable that the relatively informed people on HN seem to equate a human learning to AI learning.

> What ML does is not in any way akin to human learning in methodology. When we call it learning that's an analogy.

Of course it's different, but if we look closely, it is not copying. The model itself is smaller, sometimes 1000x smaller, than the training set. Being made of billions of examples, the impact of any one of them is very small (de minimis).

If you try to replicate something closely with AI it fails. If regurgitation was a huge problem we'd see lots of lawsuits on output, but we see most suits for input (training). That means authors can't identify cases of infringement in the outputs.

> You can not argue a legal case from analogy alone.

Yes they can because the analogy directly applies. The technical details of how a computer learns, vs a human learns doesn't really matter here, and is an irrelevant difference.

The reason why the analogy directly applies is because in both cases it is about how IP cannot really control how someone uses the IP.

Just like how IP laws cannot prevent you from listening to music while standing on your head, it also cannot prevent you from training your models on it (while also standing on your head, lets say!).

Instead, IP laws only prevents the publishing of copies of the IP.

So the point and the analogy stands.

They can work for their living like I do or get a following/sponsors which pays for their creative output. The later works quite well in the case of YouTube. YouTubers generally have proven that it is absolutely possible to commercialize an artistic output that is given out for free, removing IP laws would have essentially zero effect on them.

Nobody owes artists the ability to make a living just for being artists.