No elephants: Breakthroughs in image generation

> For many, many websites this is going to be good enough.

It was largely a solved problem though. Companies did not seem to have an issue with using stock photos. My current company's website is full of them.

For business use cases, those galleries were already so extensive before AI image generation, that what you wanted was almost always there. They seemingly looked at people's search queries, and added images to match previously failed queries. Even things you wouldn't think would have a photo like "man in business suit jump kicking a guy while screaming", have plenty of results.

My problem with finding enjoyment in this is the same problem I have when using cheat codes in games: the doing part is the fun part, getting to the end or just permutations of the end gets really boring.

>I love using LLMs to generate pictures. I'd call myself rather creative, but absolutely useless in any artistic craft. Now, I can just describe any image I can imagine and get 90% accurate results

May I ask what you use? I'm not yet even a paid subscriber to any of the models, because my company offer a corporate internal subscription chatbot and code integration that works well enough for what I've been doing so far but has no image generation.

I have tried image generation on the free tier but run out of free use before I get anyway pleasing.

What do you pay for?

If you use this technology, you're actively harming creative labor.

I've used stock photo sites occasionally but I use vector art and icon sites multiple times a week. Even today, I used an few different sites while designing some stuff on Canva.

The reason I don't use AI is because it gives me far less reliable and impossible to specify results than just searching through the limited lists of human made art.

Today, for undisclosed reasons, I needed vector art of peanuts. I found imperfect but usable human made art within seconds from a search engine. I then spent around 15 - 25 minutes trying to get something closer to my vision using ChatGPT, and using the imperfect art I'd found as a style guide. I got lots of "huh that's cool what AI can do" but nothing useful. Nothing closer to my vision than what I started with.

By coincidence it's the first time I'vr tried making art with AI in about a year, but back then I bought a Midjourney account and spent a month making loads of art, then installed SD on my laptop and spent another couple of weeks playing around with that. So it's not like I'm lacking experience. What I've found so far is that AI art generators are great for generating articles like this one. And they do make some genuinely cool pictures, it blows my mind that computers can do this now.

It's just when I sit down with a real world task that has specific, concrete requirements... I find them useless.

Their main application appears to be taking blog posts and internal memos and making them three times longer and use ten times the bandwidth to convey no more information. So exactly the application AI is ‘good’ at.

>so I suppose it's no surprise I have no real use for "generate any image on demand".

Other than stock photos, porn is the killer app for that, but most of the AI companies don't want to allow that.

How about remove blur from your photo, remove blocking items, denoise darks and fixing whiteouts. Granted it's not quite there yet for everything but it's pretty close.

Have it interview you (as-in a job interview) on your specialisation. Works your interviewer skills.

Ask it to teach you a language.

DnD works really well (the LLM being the game-master).

Unless I'm doing something simple like writing out some basic shell script or python program, it's often easier to just do something myself than take the time to explain what I want to an LLM. There's something to be said about taking the time to formulate your plan in clear steps ahead of time, but for many problems it just doesn't feel like it's worth the time to write it all out.

Some of the image generation systems are very fast.

>Image generation is still very slow.

Only because the free ones slow things down.

They insist on feeding me AI generated videos about "HOA Karens" for some odd reason.

True, I do enjoy watching the LawTubers and sometimes they talk about HOAs but that is a far stretch from someone taking a reddit post and laundering it through the robots.

Youtube Studio is has build in AI thumbnail functionality. Google actively encourages use of AI to clickbait and to generate automatic AI replies to comments ala onlyfaps giving your viewers that feeling of interaction without reading their comments.

All my music cover images are AI generated. At the same time I refuse to listen to AI music. We're all going to sink alone on this one.

What's frustrating me is if I tell the Youtube algo 'don't recommend' to AI music video channels it stops giving me any music video channels. That's not what I want, I just don't want the AI. They need to seperate the two. But of course they need to not do that with AI cover images because otherwise it would harm me. :)

Probably is your algorithm as mine is pretty good in not showing me those low effort channels. Check out extensions like PocketTube, SponsorBlock, and DeArrow to manage your YouTube feeds better.

That blog post is a year old.

There has been a lot of progress since then: https://doubiiu.github.io/projects/ToonCrafter/

BUT it's doing a stunningly better job replicating previous scenes than it did before. I asked it just now for a selfie of two biker buddies on a Nevada highway, but one is a quokka and one is a hyrax. It did it. Then I asked for the same photo with late afternoon lighting, and it did a pretty amazing job of preserving the context where just a few months ago it would have had no idea what it had done before.

Also, sweet jesus, after more than a year of hilarious frustration, it now knows that a flying squirrel is a real animal and not just a tree squirrel with butterfly wings.

I thought the selection tool allows you to limit the area of the image that a revision will make changes to, but I tested it and I still see changes outside of the selected area which is good to know.

As an example the tape spindles, among other changes, are different: https://chatgpt.com/share/67f53965-9480-800a-a166-a6c1faa87c...

https://help.openai.com/en/articles/9055440-editing-your-ima...

It just means that you comp it together manually. That's still much better than having to set up some inpainting pipeline or whatever.

Fwiw pixlr is a good pairing with GPT 4o for just this. Generate with 4o then use pixlr AI tools to edit bits. Especially for removals pixlr (and I'm sure others) are much much faster and quite reliable.

Actually, almost everything changes slightly - the number, shape and pattern of the chairs, the number and pattern of the pillows, the pattern of the curtains, the scene outside the window, the wooden part of the table, the pattern of the carpet... The blue couch stays largely the same, it just loses some detail...

Yes, first a still life and something impressionist, then a blob and a blob, then a smear and a smear. And what about the reflections and transparency of the glass table top? It gets very indistinct. Keep working at the same image and it looks like you'll end up with some Deep Dream weirdness.

I think the fireplace might be turning into some tiny stairs leading down. :)

> Using the prompt to detect and choose the most appropriate model checkpoint and LoRa(s) along with rewriting a prompt to most appropriately suit the chosen model has been pretty bog standard for a long time now.

Which players are doing this? I haven't heard of this approach at all.

Most artistic interfaces want you to visually select a style (LoRA, Midjourney sref, etc.) and will load these under the hood. But it's explicit behavior controlled by the user.

Generated autoregressively and generated in one shot are not the same. There is a possibility that there is a feedback loop here. Personally, I wouldn't be surprised if there was a small one, but not nearly the complex agentic workflow that OP may be thinking of.

All the stable diffusion software I've used names the files after some form of the prompt, and probably because SD weights the first tokens higher than the last tokens, probably as a side effect of the way the CLIP/BLIP works.

I doubt any of these companies have rolled their own interface to stable diffusion / transformers. It's copy and paste from huggingface all the way down.

I'm still waiting for a confirmed Diffusion Language Model to be released as gguf that works with llama.cpp

I mean, it could very well be that it generates image patches autoregressively, but in a pyramidal way (first a very low resolution version, the "canvas", and then each individual patch). This is very similar to VAR [1]

We can't really be sure until OpenAI tells us.

[1] https://arxiv.org/pdf/2404.02905

I'd take it farther to say that copyright and intellectual property is a legal fiction that ultimately benefits the wealthy[those who can pay to legally enforce it] over small artists.

Small artists get paid to create the art; corporations benefit from exclusivity.

> Its also not clear for example that Studio Ghibli lost by having their art style plastered all over the internet. I went home and watched a Ghibli film that week, as I'm sure many others did as well. Their revenue is probably up quite a bit right now?

This sounds like a rewording of "You won't get paid, but this is a great opportunity for you because you'll get exposure".

>Its also not clear for example that Studio Ghibli lost by having their art style plastered all over the internet.

Maybe Studio Ghibli is much more than merely a style. Maybe people aren't looking at their production just for the style.

Most people dislike wearing fake clothes and the dislike wearing fake watches or fake jewelry. Because it isn't just about the style.

Nearly every artist I've spoken to or have seen talk about this technology says it's evil, so at least among the victims of this corporate abuse of the creative community, there's wide consensus that it's bad.

> but I certainly don't think that AI without restrictions is going to lead to fewer people with art jobs.

It's great that you think that but in reality a lot of artists are saying they're getting less work these days. Maybe that's the result of a shitty economy but I find it very difficult to believe this technology isn't actively stealing work from people.

The Ghibli style took humans decades to refine and create. All that respect and adoration for the craft and artists and the time it took, is now gone in an instant, making it a shallow trivial thing. Worse is to have another company exploit it with no regard for the ones who helped make it a reality.

The threat of AI produced art will forever trivialise human artistic capabilities. The reality is: why bother when it can be done faster and cheaper? The next generation will leverage it, and those skills will be very rare. It is the nature of technology to do this.

Studio Ghibli might not have been affected yet, but only because the technology is not there yet. What's going to happen when someone can make a competing movie in their style with just a prompt? Should we all just be okay with it because it's been decided that Studio Ghibli has made enough money?

If the effort required to create that can just be ingested by a machine and replicated without consequence, how would it be viable for someone to justify that kind of investment? Where would the next evolution of the art form come from? Even if some company put in the time to create something amazing using AI that does require an investment, the precedent is that it can just be ingested and copied without consequence.

I think aside from what is legal, we need to think about what kind of world we want to live in. We can already plainly see what social media has done to the world. What do you honestly think the world will look like once this plays out?

Companies like Studio Ghibli are not being harmed by AI, small freelance artists are.

> "How can we monetize art" remains an open question for society

Yet much of the best art imho is in the wild to the element while being at home at some random place. Or perhaps in someone's collection forgot and displaced. Art's worth will always be an open question.

You don't see a massive difference in the shear number of images that the AI can look at and the speed at which it can imitate it as a fundamental difference between AI and a human copying works or styles?

For a human it took a lot of practice and a lot of time and effort. But now it takes practically no time or effort at all.

Right the difference is that it’s a large company looking at it then copying it and reselling it without credit, which basically everyone would understand as bad without the indirection of a model.

Edit: the key words here are “company” and “reselling”

False, copyism as a career has been always looked down at in the arts community. Learning and reinterpreting is a qualitatively different process.

https://news.ycombinator.com/item?id=42720749

>There are plenty of us that think that dramatically expanding the power of copyright would be a huge mistake that would primarily benefit larger companies and do little to protect or fund small artists.

The status quo also primarily benefits larger companies, and does little (exactly nothing, if we're being earnest) to protect or fund small artists.

It's reasonable to hold both opinions that: 1) artists aren't being compensated, even though their work is being used by these tools, and 2) massive expansion of copyright isn't the appropriate response to 1).

> The former seems to be pretty much decided in favor of AI

None of the cases against AI companies have been decided afaik. There's a ton of ongoing litigation.

> but doesn’t matter much while investors’ money is so plentiful.

More and more people are realizing how wasteful this stuff is every day.

It's already gone quite a bit further than "style". https://www.404media.co/listen-to-the-ai-generated-ripoff-so...

In music, someone can sing the same style as another, but if they imitate it to the point that there is brand confusion, where the consumer believes the product came from X when it actually came from Y, that's clearly crossing the line.

Yeah, of course their style isn't static, but I was taking Kiki's Delivery Service (1989) as a point where much of their visual style was pretty well-established.

>o something like when you make $X million, the work becomes public domain.

https://en.wikipedia.org/wiki/Hollywood_accounting

No, no metrics that can be gamed.

It’s a super-interesting idea, but GoT seems highly cherry picked: the vast majority of all works would never leave copyright if the requirement was that they clear even $1000.

I'm all for paying artists for their work. Unfortunately, same as tattoo artists, some just heavily overcharge for mediocre results (been tattooing myself AND I know a few things about art). Like, sorry, but if you want to earn money doing art, please be good at it...

There are a lot of programmers on this platform (myself included), and I love that my work has an impact on others.

I have a number of public repos and I have benefitted greatly from other public repos. I hope LLMs made some use of my code.

I wrote blogs for years without any monetization. I hope my ideas influenced someone and would be happy if they made some impact on the reasoning of LLMs.

I'm aware of patent trolls and know people with personal experience with them.

So I generate a lot more content that the typical person and I am still in favor of much looser IP rights as I think they have gone overboard and the net benefit for me, a content creator, is much greater having access to the work of others and being able to use tools like LLMs trained on their work.

My personal preference is for (say) 15-20 years.

And, as a content creator, I practice what I preach - at least when it comes to my poetry: https://rikverse2020.rikweb.org.uk/blog/copyrights

Not that it should impact the validity of my argument, but I have sold commercial software in the past, and it is absurd that that software will be copyrighted through most of the 21st-century.

If you make a blog with nice original long form articles it may take much longer to gain traction. Reproducing the content in "your own" wording quickly gets fuzzy.

I like the practical angle. Any formula that requires monitoring what everyone is doing is unworthy of consideration. Appeal to tradition should not apply.

The entitlement of the modern artist/musician is unprecedented. Never have I seen so many people expect to be handed a living because they've posted some "content". Musicians and artists now have global distribution with a plethora of platforms. You have to harness that and then work and grind it out. You have to travel and play gigs and set up a booth at art shows.

There's this new expectation that you should just be able to post some music on Spotify or set up an Etsy shop and get significant passive income. It has never ever worked that way and I feel this new expectation comes from the hustle/influencer types selling it.

Most art is crap and most music isn't worth listening to. In the modern age, it's easy for anyone to be a band or artist and the ability to do this has led to a ton of choice, the market is absolutely flooded. If anyone can do a thing (for very loose values of "do") it's inherently worth less. Only the very best make it and it will always be that way.

Source: made a living as a musician for 20 years. The ones who make it are relentlessly marketing themselves in person. You have to leave the house, be a part of a scene, and be constantly looking for opportunities. No one comes knocking on your door, you must drive your product and make yourself stand out in some way. You make money on merch and gigs, and it's always been that way.

This is all to say that copyright law only affects the top 0.1%. The avg struggling artist will never have to worry about any of this. It's like Bob the mechanic worrying about inheritance taxes. Pipe dream at best.

I mean, this is about as useful as saying anti-slavery people should become slave owners so they understand the hardships of making money.

My example is extreme to the absurd, so how about we go with

>It's difficult to get a man to understand something when his salary depends on not understanding it.

Correct. This is why Stirner is the best Philosopher. https://en.wikipedia.org/wiki/The_Ego_and_Its_Own

There is nothing other than Egoism.

Either abolish it, or make it stronger so it applies to everyone. (OpenAI included.)

"Copyright for thee but not for me" is the worst of all worlds.

You forgot:

2024: What do you mean I can't copyright AI generated artwork?

Honestly, I'm pretty encouraged by all of the projects and efforts within legislation and organizations regarding clear lines being drawn - i.e., through watermarking to clearly label whether something is AI-generated or not - as well as the efforts by industries for livelihoods to be protected, specifically in the creative space, where human intentionality and feeling are still of the utmost essentiality. We've seen, are seeing, and will see cultural and societal acceptance and backlash against one thing or another, but I'm confident that we will adapt. Ultimately, pushback, thanks to the Web itself, is already pretty monumental among artists and even other AI researchers in many respects - regulations for the internet, largely due to lack of the Web, were far slower to materialize, on an exponential scale. I remain optimistic that we will find the niches where AI is needed, where it isn't, and where it is detrimental.

Just like text results from AI, when images like this get better, the more and more subtle yet absolute wrongness is going to be a nightmare to deal with.

Imagine a ask AI to show me a sewer cap that's less than a foot wide (or whatever, I dunno, watch TMNT right now). And it does, just by showing a sewer cap that looks photorealistic and a ruler, that has markings from one end to the other that only go up to 8 inches. That doesn't mean sewer caps come in that size, it just means you can produce a rendered image to fit what you asked for.

I don't get how this would produce consistent images. In the article, the text could be on a grid, but the window and doorway and sofa don't seem to be grid-aligned. (Or maybe the text is overlaid?)

> I haven't see any details on how OpenAI's model works

Exactly. People just confidently make things up. There are many possible ways, and without details, "native generation" is just a marketing buzzword without clear definition. It's a proprietary system, there is no code release, there is no publication. We simply don't know how exactly it's done.

While it’s not 100% perfect (no horn from the forehead of his helmet) I’d say this is far beyond 90%. I can imagine reading books from project Gutenberg on a future reader app that automatically makes pictures of each scene which are consistent with each other and faithful to the text, on the fly as you read.

You make it sound like an unfair wish, which I would say is unfair itself. I like the book, I like visualising it in my head, and I fantasise what a scene would look like. AI won't generate a true visual. It's all fantasy anyway and AI can actually do it the way I have it in my head. It will solidify that thought.

For this book in particular, I read the comic version and I didn't like the visuals very much. I have a different idea of babel fish. Vogons look different. I would love to see the visual that's in my head on paper.

Vortex may be not so much but there are other hard to visualize things. I am on the third book, and I have no idea what Beeblebrox's two heads look like. Second head is often mentioned in passing. Sometimes its mentioned as its always there, other times it feels like its just pops out of somewhere, otherwise, it's like it doesn't exist.

There is the scene when they see themselves on the beach on first rescue by the ship. That was hard to grasp. Or the insides of the ship itself, the bridge, the panels etc. Also that black ship they stole.

But may be its just me having a hard time with these concepts.

It's not just about scene being difficult to visualize, even if I can see them in my head, I want to see them on paper too because those thing excite me.