Stubsack: weekly thread for sneers not worth an entire post, week ending Sunday 1 September 2024

The poison floor not hurting the player trapping him forever was a good thing to end on.

Oh it's definitely comparable. See, I'll compare:

The visual quality of GameNGen is worse than that of the original game.

only real gamers want to shoot Thongmonster Pro. coming soon to a doom WAD near you

Man, that looks like ass.

Sorry, looks like an ass.

and yet, with zero evidence to support the claim, the paper’s authors are confident that their model can be used to create new game logic and assets:

Today, video games are programmed by humans. GameNGen is a proof-of-concept for one part of a new paradigm where games are weights of a neural model, not lines of code. GameNGen shows that an architecture and model weights exist such that a neural model can effectively run a complex game (DOOM) interactively on existing hardware. While many important questions remain, we are hopeful that this paradigm could have important benefits. For example, the development process for video games under this new paradigm might be less costly and more accessible, whereby games could be developed and edited via textual descriptions or examples images. A small part of this vision, namely creating modifications or novel behaviors for existing games, might be achievable in the shorter term. For example, we might be able to convert a set of frames into a new playable level or create a new character just based on example images, without having to author code.

the objective is, as always, to union-bust an industry that only recently found its voice

yeah, this is weirdly sneerable for a 404 article, and I hope this isn’t an early sign they’ve enshittifying. let’s do what they should have and take a critical look at, ah, GameNGen, a name for their research they surely won’t regret

Diffusion Models Are Real-Time Game Engines

wow! it’s a shame that creating this model involved plagiarizing every bit of recorded doom footage that’s ever existed, exploited an uncounted number of laborers from the global south for RLHF, and burned an amount of rainforest in energy that also won’t be counted. but fuck it, sometimes I shop at Walmart so I can’t throw stones and this sounds cool, so let’s grab the source and see how it works!

just kidding, this thing’s hosted on github but there’s no source. it’s just a static marketing page, a selection of videos, and a link to their paper on arXiv, which comes in at a positively ultralight 10 LaTeX-formatted letter-sized pages when you ignore the many unhelpful screenshots and graphs they included

so we can’t play with it, but it’s a model implementing a game engine, right? so the evaluation strategy given in the paper has to involve the innovative input mechanism they’ve discovered that enables the model to simulate a gameplay loop (and therefore a game engine), right? surely that’s what convinced a pool of observers with more-than-random-chance certainty that the model was accurately simulating doom?

Human Evaluation. As another measurement of simulation quality, we provided 10 human raters with 130 random short clips (of lengths 1.6 seconds and 3.2 seconds) of our simulation side by side with the real game. The raters were tasked with recognizing the real game (see Figure 14 in Appendix A.6). The raters only choose the actual game over the simulation in 58% or 60% of the time (for the 1.6 seconds and 3.2 seconds clips, respectively).

of course not. nowhere in this paper is their supposed innovation in input actually evaluated — at no point is this work treated experimentally like a real-time game engine. also, and you pointed this out already — were the human raters drunk? (honestly, I couldn’t blame them — I wouldn’t give a shit either if my mturk was “which of these 1.6 second clips is doom”) the fucking thing doesn’t even simulate doom’s main gameplay loop right; dead possessed marines just turn to a blurry mess, health and armor don’t make sense in any but the loosest sense, it doesn’t seem to think imps exist at all but does randomly place their fireballs where they should be, and sometimes the geometry it’s simulating just casually turns into a visual paradox. chances are this experimental setup was tuned for the result they wanted — they managed to trick 40% of a group of people who absolutely don’t give a fuck that the incredibly short video clip they were looking at was probably a video game. amazing!

if we ever get our hands on the code for this thing, I’m gonna make a prediction: it barely listens to input, if at all. the video clips they’ve released on their site and YouTube are the most coherent this thing gets, and it instantly falls apart the instant you do anything that wasn’t in its training set (aka, the instant you use this real-time game engine to play a game and do something unremarkably weird, like try to ram yourself through a wall)

Oh god is this the first time we have to sneer at a 404 article? Let’s hope it will be the last.

My intention was more to sneer at the research:

Diffusion Models Are Real-Time Game Engines