Results of the "Can you tell which images are AI generated?" survey

So if the average is roughly 10/20, that's about the same as responding randomly each time, does that mean humans are completely unable to distinguish AI images?

One thing I'm not sure if it skews anything, but technically ai images are curated more than anything, you take a few prompts, throw it into a black box and spit out a couple, refine, throw it back in, and repeat. So I don't know if its fair to say people are getting fooled by ai generated images rather than ai curated, which I feel like is an important distinction, these images were chosen because they look realistic

Did you not check for a correlation between profession and accuracy of guesses?

I still don’t believe the avocado comic is one-shot AI-generated. Composited from multiple outputs, sure. But I have not once seen generative AI produce an image that includes properly rendered text like this.

Something I'd be interested in is restricting the "Are you in computer science?" question to AI related fields, rather than the whole of CS, which is about as broad a field as social science. Neural networks are a tiny sliver of a tiny sliver

I got a 17/20, which is awesome!

I’m angry because I could’ve gotten an 18/20 if I’d paid attention to the thispersondoesnotexists’ glasses, which in hindsight, are clearly all messed up.

I did guess that one human-created image was made by AI, “The End of the Journey”. I guessed that way because the horses had unspecific legs and no tails. And also, the back door of the cart they were pulling also looked funky. The sky looked weirdly detailed near the top of the image, and suddenly less detailed near the middle. And it had birds at the very corner of the image, which was weird. I did notice the cart has a step-up stool thing attached to the door, which is something an AI likely wouldn’t include. But I was unsure of that. In the end, I chose wrong.

It seems the best strategy really is to look at the image and ask two questions:

• what intricate details of this image are weird or strange?
• does this image have ideas indicate thought was put into them?

About the second bullet point, it was immediately clear to me the strawberry cat thing was human-made, because the waffle cone it was sitting in was shaped like a fish. That’s not really something an AI would understand is clever.

One the tomato and avocado one, the avocado was missing an eyebrow. And one of the leaves of the stem of the tomato didn’t connect correctly to the rest. Plus their shadows were identical and did not match the shadows they would’ve made had a human drawn them. If a human did the shadows, it would either be 2 perfect simplified circles, or include the avocado’s arm. The AI included the feet but not the arm. It was odd.

The anime sword guy’s armor suddenly diverged in style when compared to the left and right of the sword. It’s especially apparent in his skirt and the shoulder pads.

The sketch of the girl sitting on the bench also had a mistake: one of the back legs of the bench didn’t make sense. Her shoes were also very indistinct.

I’ve not had a lot of practice staring at AI images, so this result is cool!

Wow, what a result. Slight right skew but almost normally distributed around the exact expected value for pure guessing.

Assuming there were 10 examples in each class anyway.

It would be really cool to follow up by giving some sort of training on how to tell, if indeed such training exists, then retest to see if people get better.

One thing I'd be interested in is getting a self assessment from each person regarding how good they believe themselves to have been at picking out the fakes.

I already see online comments constantly claiming that they can "totally tell" when an image is AI or a comment was chatGPT, but I suspect that confirmation bias plays a big part than most people suspect in how much they trust a source (the classic "if I agree with it, it's true, if I don't, then it's a bot/shill/idiot")

And this is why AI detector software is probably impossible.

Just about everything we make computers do is something we're also capable of; slower, yes, and probably less accurately or with some other downside, but we can do it. We at least know how. We can't program software or train neutral networks to do something that we have no idea how to do.

If this problem is ever solved, it's probably going to require a whole new form of software engineering.

Wow, I got a 12/20. I thought I would get less. I'm scared for the future of artists

Having used stable diffusion quite a bit, I suspect the data set here is using only the most difficult to distinguish photos. Most results are nowhere near as convincing as these. Notice the lack of hands. Still, this establishes that AI is capable of creating art that most people can't tell apart from human made art, albeit with some trial and error and a lot of duds.

I have it on very good authority from some very confident people that all ai art is garbage and easy to identify. So this is an excellent dataset to validate my priors.

Curious which man made image was most likely to be classified as ai generated

Sketches are especially hard to tell apart because even humans put in extra lines and add embellishments here and there. I'm not surprised more than 70% of participants weren't able to tell that one was generated.

Thank you so much for sharing the results. Very interesting to see the outcome after participating in the survey.

Out of interest: do you know how many participants came from Lemmy compared to other platforms?

I guess I should feel good with my 12/20 since it's better than average.

What I learnt is that I’m bad at that task.

I did 12 and I thought it was a pretty bad result. Apparently I was wrong!

I tried to answer by instinct, without careful examination, to make it more realistic

It'd be interesting to also see the two non-AI images the most people thought were.

I got 11/20 and there were a couple of guesses in there that I got right and wrong. Funny how there are some man-made ones that seem like AI. I think it's the blurry/fuzziness maybe?

Oof. I got about 65% on the images I hadn’t seen in the post. I must be pretty close to being replaceable by an adversarial network.

You put 'tell' twice in the title 😅

I would second the comment from the original post which said that they didn't pay much attention in the latter half.

Interesting. So you've given us a 50/ 50 chance. Usually you've given us the art that was used and then the AI has attempted it's own version? Did you train the ai using the art ?

Are you allowed to do that ? Is the art in the public sphere?

I have no knowledge or understanding of ai.

Can people tell tell

Is there any chance that the human artists experimented with AI and posted the result along with their usual work? :)

To me 4 was pretty obvious as a lot of the lines made no sense, such as the ones right below her face.

Also, is there a correlation between image generator and accuracy?

That avocado and tomato post took me out, that and the Legos. Very impressive.

The most obvious ai one for me was the last cat picture, somehow it just screamed ai

Huh, I felt the 12/20 was a bit low but I guess not so much. As someone that has never used an image generator (or an LLM for that matter, chatGPT not even once baby) nor has actually worked at tech (though I have been learning programming on my own) and doesn't even know how to draw... I guess I didn't do too bad.

No shot the bottom left was ai gen without human help. AI has so much trouble delivering words and text.

Luckily we've already begun developing tools to help detect AI Images with higher accuracy than a human curator.