The Battle for Experience: The Shader Front
On the streaming team at Ludeo, we are the first impression players have of the games we work with. They jump straight into a moment of a game, streamed from the cloud. Behind the scenes, in those few seconds, we are finding the best hardware available for that game in the data center closest to the user.
Now you are in the moment, playing the final boss fight of the game, using your character’s best skill on him. And then the game lags and gets stuck, you die from the boss, and you lose the fight.
This is the first post in a series on our never-ending battle: the fight to make that experience perfect. Here’s one of its fronts.
Know your enemy: the lag
Pretty much everything on screen, the lighting, the smoke, the shine on a sword, gets drawn by tiny GPU programs called shaders. They’re what turn the game’s graphics data into the pixels you actually see, and a modern game has thousands of them.
The thing is, a shader can’t run the instant the game asks for it. The GPU has to compile it first, and the very first time, that happens while you’re already playing. One shader is nothing, microseconds. But a few thousand of them compiling at once, in your first minutes in the game, is exactly the lag you feel while the GPU catches up.
And it doesn’t only happen at the start. Anytime you do something new, walk into an area nobody’s explored, fire off a skill you haven’t used, the GPU may have to stop and compile a fresh shader right then, and you feel another hitch. A warm cache means that work is already done, so the game stays smooth the whole way through, not just after the opening few minutes.
The fix is simple: compile each shader once, save it to disk, and load it from there next time instead of building it again. Do that, and the game is smooth from the very first frame. Really, that’s the whole job: warm the cache before the player shows up.
On paper, simple. In practice…
Keeping one cache warm on one machine is easy. Ours is the opposite: each game tells us which GPUs it runs best on, and since we choose the hardware in the cloud, we have to warm a cache for every one of them, not just one.
And that’s just one game on one machine. We run many games, on many machines, across a few GPU families (Turing, Ada Lovelace, Blackwell), a pile of driver versions, and more than one cloud. Each of those combinations is its own separate cache, because a compiled shader only works on the exact hardware and driver it was built for. And none of it holds still: games get patched, drivers get updated, and somewhere a fresh machine is always hitting something for the first time, and that first time is the slow one.
So “just keep the cache warm” is actually a hard distributed-systems problem. Who compiles the shaders? Where do they live? How do we know when they’ve changed? And how does every machine end up with the right version before it launches a game? Answering those is the system we built.
How a shader actually gets made
One more thing about our setup, and it came from a call we made early on. The games we stream are Windows titles, built in Unity or Unreal, but running Windows in the cloud carries a license cost on every VM, and across a fleet our size that’s real money. So we run on Linux instead, which is far cheaper. A Windows game doesn’t run on Linux by itself, so we use a compatibility layer called Proton that (through DXVK) translates the game’s DirectX shaders into Vulkan as it plays. Vulkan is just the graphics standard the GPU speaks on Linux, and the shaders it runs come in a low-level format called SPIR-V. Those translated SPIR-V shaders are what we end up collecting.
The old battle plan
In the early days this nearly cost us. When a lot of people showed up at once, we’d spin up a brand-new machine for whoever was next in line. But a cold machine has to load the game, load the Ludeo moment, and compile every shader from scratch, and that took long enough that people in the queue gave up and closed the tab. We were losing players at the door, before the game had even started, and paying for the machine time it took to lose them.
So we started warming the machines up ahead of time, collecting the shaders before anyone landed on one. At first that meant doing it by hand. To onboard a game, an engineer would spin up a machine with the right GPU, launch the game, and play through enough of it to make the GPU compile its shaders. Then they’d copy the cache files off the disk and upload them by hand. Half an hour to an hour, for one game on one GPU.
And it was never one-and-done. The same game had to be replayed for every GPU type, recollected whenever NVIDIA pushed a new driver, and redone every time the studio patched the game. Every new game added more of this, and every driver update threw out caches we’d already made. The bigger we got, the more there was to redo, and by hand we couldn’t keep up.
The battle plan
When we designed this, we had one rule: the machine running your game should never wait on us. So we split the work. A site-controller back in our cloud makes the calls, which game, which version, who’s allowed to save. An agent on each GPU machine carries them out, doing only fast, local things so it never slows the game down. (A thin per-cloud adapter sits between them just to hand out signed upload and download links, nothing more.)
From there it’s three moves, all running in the background: Detection spots new shaders, Saving decides if they’re worth keeping, and Game-start makes sure a machine is warm before anyone plays. Each cache is filed by game, GPU family, and driver, and given a version number that goes up over time, so we never serve the wrong hardware.
Maneuver 1 · Detection
When a game starts, the agent calculates a snapshot of the cache folder, basically the size of every file, and keeps it in memory. The player plays. When the game ends, it snapshots again and compares the two.
Early on we got this wrong. The agent saved on every change it noticed, and since almost every session nudges the cache a little, we ended up cutting new versions constantly for diffs that made no real difference. So now a session only counts if a brand-new cache file shows up, or if the existing ones grow past a set amount. Anything smaller is just noise we ignore, and only a real change goes up to the controller.
Maneuver 2 · Saving
Before it saves anything, the controller runs through a few questions. Is this change actually worth keeping? Is another machine already uploading the same thing? Is what we already have newer than what this machine is offering? Only if it clears all of them does the controller hand out a new version number and a signed link to upload to. The agent sends its cache straight to storage, and only once that upload succeeds does the controller bump the version. If it fails, nothing changes.
Maneuver 3 · Game-start
Right before a game launches, the controller tells the machine which version to use and hands it a signed download link. The agent checks if it already has that version on disk; if it does, it skips the download and just goes. If not, it downloads the cache, unpacks it, and the game loads from it. Either way the game opens warm, and any new shaders this session happens to compile feed straight back into Detection.
Boots on the ground: the agent on the GPU machine
It starts with the site-controller. When a player shows up, it boots a VM in the closest site, on the best GPU available in that data center. The machine’s OS starts the agent, and the agent reports back which GPU driver it’s running. Now we know exactly what we’re working with, and can load the cache built for it.
From there the agent does the rest on the machine itself: it launches the game and watches the cache the game writes into, sending anything new back to storage.
All of this runs in the background while someone’s playing, so they never notice the agent is there at all.
No friendly fire: no machine overwrites another
This is the trickiest part to get right. At any moment we might have dozens of machines running the same game on the same hardware, each compiling its own shaders. They finish around the same time, and they all want to save. Three rules keep this safe:
Versions are immutable. Saving never writes over what’s already there. It always creates the next version instead, so the cache a machine started from is never touched. Worst case, nothing is ever lost.
Only one machine saves at a time. When several finish at once for the same game and hardware, one agent gets the lock and the rest are ignored, so only one upload happens instead of many.
Stale work can’t win. The problem is a slow machine that finishes long after a newer version was released, when its shaders are already out of date. So before it can save, the controller checks one thing: is what we already have newer than what this machine has? If it is, the late save is dropped, so an older cache can never overwrite a newer one.
Counting the cost
On the streaming team, two numbers sit on a dashboard we all watch: cost per play and machine utilization. Almost everything we build is trying to move one of them, and shaders happen to hit both.
We rent these GPU machines by the second, and every second one spends compiling instead of rendering is a second we pay for with nobody playing. That’s money spent while the player just waits. A cold start can be minutes of that. Warm the cache first and the session is playing almost immediately, so that wasted time turns back into play time.
A few seconds a session doesn’t sound like much until you multiply it by every session we stream. Then it’s real movement on both numbers: cost per play drops, and utilization climbs, because the machines spend more time serving players and less time warming up.
Results from the field
We measured it both ways on the same hardware, cache off versus cache on: how long the game takes to load, how long a Ludeo moment takes to load, and how steady the frame rate holds once you’re playing.
Ground gained
Next to the old way, here’s what changed: a job that used to eat an engineer’s afternoon now runs itself, and gets better every time someone plays.
The battle continues, and our players are the soldiers
Perfecting the player’s experience is a constant battle. A new driver, a new game build, a GPU a title hasn’t run, and a new Moment that never been played yet on the cloud, could require more shaders collection.
Our users are the soldiers helping us win the battle, with every new session they explore. we gather more shaders, and improving the experience shader by shader, for the next players.
