Neural Game Engines: The End of the Traditional Render Pipeline

We have spent the last forty years optimizing triangles. From the jagged polygons of the PS1 era to the path-traced, hyper-realistic lighting of Unreal Engine 5, the core philosophy of video game rendering hasn’t actually changed: a CPU calculates the game logic, feeds a massive list of geometry to a GPU, and the GPU paints the pixels on your screen.

Before you read: We previously looked at how Sora and Runway are upending Hollywood’s VFX pipeline. But while those models generate passive, static video clips, a new breed of AI is doing something far wilder: generating interactive video games in real-time.

Table of contents

• Hallucinating the Simulation
• The Object Permanence Problem
• The Hardware Economics
• Fun Facts
• The Death of Esports, The Birth of Endless Worlds
• Sources

Hallucinating the Simulation

What happens when you throw out the geometry entirely? That is the question driving neural game engines. Instead of rendering a 3D world, these models simply predict what the next frame of a video game should look like based on your controller inputs.

Think of it like playing Dungeons & Dragons. The game engine isn’t a physics simulator; it’s a Dungeon Master hallucinating the visual reality on the fly. You press the “forward” key, and the neural network generates a frame showing the world moving closer to you. You press “shoot,” and it hallucinates a muzzle flash.

In August 2024, Google researchers unveiled GameNGen, a neural model that simulates the classic 1993 shooter DOOM at 20 frames per second using a single TPUv5. It doesn’t run the DOOM engine code. It has just ingested so much DOOM gameplay footage that it knows exactly how the pixels should react when you pull the virtual trigger. If there is one constant in software engineering, it’s that we will find increasingly expensive ways to run DOOM.

The Object Permanence Problem

Google’s breakthrough was quickly followed by Decart’s Oasis, an AI model that generates a playable Minecraft clone in real-time. Oasis is fascinating because you can literally play it in your browser right now.

But playing Oasis exposes the massive, glaring flaw in neural rendering: AI has terrible object permanence.

Because the engine is just predicting the next frame based on the previous few frames, it quickly forgets what is behind you. You can build a dirt house, turn around to look at a pig, and when you turn back, your house has morphed into a tree. It is exactly like playing Minecraft while aggressively sleep-deprived. The world feels dream-like, unstable, and deeply weird.

The Hardware Economics

Right now, neural game engines are a horribly inefficient parlor trick. Generating a blurry, 20 FPS version of a game from 2011 requires massive AI accelerators that burn through electricity.

But the hardware industry is already pivoting. Decart partnered with Etched, a chip startup building ASICs specifically designed to run transformer models. Etched claims their upcoming Sohu chip can run Oasis at 4K resolution and 60 frames per second. We are watching the birth of a completely parallel hardware race. Nvidia built its empire on rasterization and ray tracing; the next trillion-dollar company might be the one that perfects real-time frame inference.

Fun Facts

• GameNGen was trained by having an AI agent play DOOM for thousands of hours, generating the massive dataset of “actions and resulting frames” needed to teach the neural network how the game works.
• Neural engines don’t actually “know” the rules of the games they simulate. They have no concept of health points, gravity, or collision detection—they just mimic the visual consequences of those rules perfectly.

The Death of Esports, The Birth of Endless Worlds

Will neural engines replace Unreal Engine tomorrow? Absolutely not. You cannot run a competitive esports tournament like Counter-Strike on an engine where the hitboxes are probabilistically generated. Competitive gaming requires deterministic math.

But for single-player exploration? The potential is staggering. A traditional game is limited by the assets a human artist can manually create and store on a hard drive. A neural engine could generate an infinite, dynamically shifting universe that reacts to your voice, your playstyle, and your choices, creating entirely new biomes and mechanics on the fly.

We are currently in the “pong” era of generative interactive media. The graphics are blurry, the rules are shaky, and the compute costs are astronomical. But the foundational proof of concept is here: we no longer need to render worlds. We can just dream them.

Sources

• Google Research: GameNGen
• Decart: Introducing Oasis
• Etched: The Sohu Architecture