Why Owning GPUs Isn’t Enough to Serve AI | AIEEV Developer Kitae Yoo
There is a GPU in the development workstation, and another one in the lab server. The edge box at a branch office may have an accelerator in it too. But the moment you try to load a model, the story changes. The driver version may not match, the SDK you need may be missing, or a network restriction may block access from outside. The hardware is right there, and yet the service cannot use it.
Kitae Yoo of the AIEEV Dev Team builds the distributed infrastructure that connects GPUs and NPUs scattered across these places and turns them into capacity for real AI inference. He checks whether each machine is actually in a state to run a model, groups the verified resources into a single pool, and builds and operates the structure that serves inference requests on top of it. On September 1 he shared what he has learned at KCD x Ceph x OpenInfra Day Korea 2026, in a talk titled “Ray Serve in the Wild: Autoscaling and Operating GPU/NPU-based AI Inference.” We asked him why a GPU you already own often cannot be used right away, why his team chose Ray over the alternatives, and which problems only showed up once the system had been running for a while.
👋 Hi Kitae. Could you introduce yourself and what you work on?
Hi, I’m Kitae Yoo, and I build and operate distributed AI infrastructure on Ray and Kubernetes at AIEEV. My part is turning distributed GPUs and NPUs into resources a real service can use. Having a GPU doesn’t mean you can simply pick it up and run on it. We first check whether the model we want actually runs on that machine, and only the verified resources go into a single pool. Models are deployed on top of that pool to serve real requests. When requests pile up we add resources, and when traffic drops we scale back down. If one machine drops out along the way, the service still has to keep running. My talk was about building and operating that kind of system.
🎤 It started with a question: “We have the GPUs, so why can’t we use them?”
What was the problem you most wanted to talk about in this session?
These days any conversation about AI infrastructure starts with how many GPUs you have secured. New GPU supply is of course still necessary. But once you work in the field, you learn that securing a GPU and making that GPU usable by a service are two different problems. Putting a GPU in a rack does not start an AI service. You need servers and power, and a cooling environment that can handle the heat. To take real requests you also need the network and security setup around it. So even when new GPUs arrive, it takes time before they become real serving capacity. The reverse is true as well: GPUs that are already installed are not always in use. The hardware is there, but the rest of the environment isn’t ready, so it sits idle. That is where we asked ourselves, “Is using the idle time of the GPUs that already exist just as important as securing new ones?” This session started from what we went through connecting those resources and running them as a service.
GPUs and NPUs need different runtime environments depending on the type, and even two machines of the same type can run different models depending on the drivers and software installed on them. So what mattered to us was not how many GPUs we had, but whether the model we wanted actually ran on a given machine. Say we find one NPU. That fact alone is not enough to put it into service. We have to check whether it supports the software we need, then actually load a model and run inference on it. That is why we don’t register a machine as a resource the moment we discover it. Our own agent checks the machine and its runtime environment, verifies it against a real workload, and only the machines that pass go into the resource pool. At that point you have usable hardware. And then the next problem begins: how do you make machines sitting in different places behave like a single service?
🧩 Kubernetes and Slurm were both on the table. Why Ray?
To run many machines as one service, you need something that decides which job runs on which machine. We started by comparing Kubernetes, Slurm, and Ray.
Kubernetes is strong at reliably running the APIs and management systems that have to stay up in the center. But the resources we wanted to connect are more than data center servers. Development workstations and edge devices in completely different environments can join too. Managing all of them as Kubernetes nodes would mean normalizing the network and runtime environment of every single machine, and that is a heavy burden.
Slurm fits HPC environments well, where you allocate a few GPUs at a time and run training jobs. But we were not building a system that runs a job once and finishes. User requests keep coming in, and we have to scale models up and down with the request volume. Even if one machine drops out along the way, the service address has to stay the same.
In this environment, Ray fit what we were trying to do. So we run the central control plane and the APIs on Kubernetes, and leave the job of running and distributing AI workloads on the actual GPUs and NPUs to Ray. It was less a matter of choosing one over the other, and more a matter of splitting the work by what each one does well.
🛠 What we couldn’t see while building it, and only saw after running it
What was the hardest problem in actual operation?
At first it is easy to think the hardest part is done once you have connected several machines, a model comes up, and requests get served. But after running the service for a while, what came after that turned out to be harder. Resource placement is a good example. Two machines that look identical, each with one NPU, can differ in practice, where only one of them can run a particular model. So we could not place jobs by GPU or NPU count alone. We had to judge whether a machine could actually run that specific model before sending work to it. Scaling back down when traffic drops was similar. If one machine has just started loading a model and another is already warmed up and taking requests, which one you shut down first matters. Shut down the ready one and the next request has to load the model from scratch again. So we adjusted the behavior to keep already warm resources alive as long as possible.
What happens when a machine suddenly disappears or fails?
That was the trickier problem in real operation. In a distributed environment you cannot assume any machine is always connected. A machine can be powered off, a network link can drop, and the node that was routing requests can vanish. At one point, after the controller that manages request routing restarted, the new controller came up fine, but some proxies were still looking at information from the old controller. On the surface everything looked healthy, while requests were not actually being delivered. We also hit cases where the node the proxy was running on disappeared entirely. The existing system could handle a proxy shutting down, but when the node itself was gone it could not create a new proxy, and recovery stalled. Both problems came down to the same thing. When something fails, restarting the dead process is not enough. You have to clear the stale state that is left behind and confirm that the service as a whole has returned to a correct structure.
You mentioned that a feature built for failure handling ended up blocking recovery.
Yes, that happened. It came up while running GCS and Redis, which hold the state of the Ray cluster. To put it simply, Redis stores the information needed to recover the cluster, such as which nodes exist and how much capacity they have. We had set it up so that another server takes over if Redis fails, so at first we assumed there would be no problem as long as Redis itself recovered. In practice, there were cases where Redis came back and GCS still failed to reconnect to the new Redis. The store was alive, but the side that had to read from it was still holding on to a dead connection. In the worst cases, the very feature we had built for failure recovery was what stopped the whole recovery. So we changed it so that GCS does not terminate immediately when the Redis connection drops, and added a path for it to find and reconnect to a new Redis. That experience taught us that preventing failures matters, and so does the process by which a system returns to a healthy state after a failure.
Why contribute these fixes back to Ray instead of only patching them inside AIEEV’s own service?
If all we cared about was getting the service healthy again, we could have worked around it in our own code. But following the cause sometimes led to a place where fixing Ray’s own behavior was the right answer. The problems we hit were less about some unusual feature we had added, and more about what surfaces when you run distributed resources as a single service over a long period. If you want to keep GPUs and NPUs in many locations as one resource pool the way we do, a machine has to be able to leave and rejoin, and a failure along the way has to be recoverable without rebuilding the whole service. So when a problem would need solving in any other environment using Ray the same way, we thought it was right to fix it in Ray rather than work around it inside our own service. Some of what we found and fixed was contributed back to Ray upstream.
🔧 After going through a number of failures, did your operating standards change?
They definitely did. In the beginning, what we thought about most was how to bind many GPUs into one pool. Now we look more at how to keep the pool we bound running correctly. We don’t treat a machine as a resource the moment it is discovered. Only machines verified down to actual execution get used. We also don’t assume a machine that joined once will stay in the same state. We operate on the premise that anything can join or leave at any time. How we judge a failure changed too. It used to be easy to think recovery was done once the failed process came back up. In reality the controller can be alive while a proxy is still looking at old information, and Redis can come back while GCS never reconnects. So now we look at whether the service as a whole has returned to its original state, not whether each individual system is alive.
💻 So what was the problem you were really solving?
At first we thought of it as binding scattered GPUs into one. But once we built and ran a real service, the harder part was what came next. When a machine with a different environment joins, you have to judge whether it is actually usable, and place models on the right machines. Even if a machine disappears or fails along the way, the service has to come back to a healthy state. What we are building turned out to be more than a system that connects GPUs. It is about making compute resources in different places keep behaving like one service. That process is what I most wanted to talk about in the session.
Finally, what do you hope Air Cloud becomes?
When people hear “distributed resources,” there is usually a concern that comes first. The machines are spread across many places, so can you use them reliably, and is it safe to put company data and workloads on them? But the distributed infrastructure we have in mind is not simply gathering GPUs from here and there and handing them out. The core of what we have been building is closer to making a large number of resources in different environments manageable as a single infrastructure. Plenty of companies already have good GPUs in house that they are not fully using. Those resources can be grouped into a private pool inside the company without being shared externally. Workloads where security or data governance matters can be handled on internal resources. And when your own resources are not enough, you can bring in Air Cloud’s external capacity alongside them, using the company’s own GPUs most of the time and extending outward at the moments you need to. Not every company has to own infrastructure sized for its peak demand.
I would like Air Cloud to be the service that offers those options naturally. Infrastructure where a company with its own resources uses them better, and connects to outside resources only as much as it needs when it runs short. In the end, what matters is less where the resources sit and more how reliably you can use the resources you need inside one service. The problems we are solving now, connecting and operating all these different GPUs and NPUs, are part of getting there. Letting each company split private and public, or use both together, and shape its infrastructure the way its situation calls for. That is where the strength of Air Cloud’s distributed infrastructure shows best.
.
.
.
Experience the distributed GPU infrastructure Kitae Yoo is building — try Air Cloud yourself. 👉 Try Air Cloud
Meet more stories from the AIEEV team.
