Why was there an issue with vLLM before TRL v0.18.0?

Before TRL v0.18.0, vLLM ran as a separate process on different GPUs from the training job, causing GPU inefficiencies due to idle time during frequent model generation.

What is the 'ping-pong' issue between training and generation?

The 'ping-pong' issue refers to the wasted GPU time on both sides during the continuous communication between the trainer and vLLM server for model generation.

How does the co-located vLLM in TRL address these issues?

Co-located vLLM in TRL allows training and inference to run on the same GPUs, reducing idle time and increasing efficiency by sharing resources.

What is GRPO and why is it significant with co-located vLLM?

GRPO is an online learning algorithm used in TRL for training LLMs. Co-located vLLM is particularly important for GRPO because generation is a critical step and major bottleneck in the training loop.

What are the benefits of the co-located vLLM approach?

The co-located vLLM approach reduces waste, increases efficiency, and lowers costs by sharing GPUs between training and inference.

Efficiency Boost: Co-located vLLM in TRL Reduces GPU Waste and Costs

TRL's new co-located vLLM approach improves efficiency by running training and inference on the same GPUs, reducing waste, increasing speed, and lowering costs for online learning methods like GRPO.

No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL

The Problem

Before TRL v0.18.0, vLLM was only supported in server mode, running as a separate process on different GPUs from the training job. It communicated with the training script over HTTP, which made the setup modular and easy to use — but also introduced GPU inefficiencies.

During training, the model needs to generate completions frequently. The trainer sends a request to the vLLM server, which runs on its own GPUs. While vLLM generates, the training GPUs sit idle and wait. Once generation is done, vLLM GPUs become idle, and training resumes.

This “ping-pong” between training and generation causes wasted GPU time on both sides, increased demand for extra GPUs just to run inference, reduced overall throughput, and higher cost. In online learning methods like GRPO — where generation happens constantly — this inefficiency becomes even more painful. You spend more on hardware, but don't get the performance you'd expect.

The Opportunity

The main issue was that training and inference ran on separate GPUs, leading to idle time and underutilization. The natural solution? Run both on the same GPUs. Instead of having vLLM operate as a standalone server in its own process and devices, what if vLLM could run alongside the training code, within the same distributed process group?

This would let us launch a single distributed job where training and inference share the same devices, switching between tasks efficiently without wasting resources. This approach is known as co-located vLLM in TRL.

Background and Context

TRL supports training LLMs using GRPO, an online learning algorithm recently introduced in the DeepSeekMath paper. In GRPO, the model learns from its own outputs: it generates responses during training, receives feedback, and uses that feedback to improve itself over time.

This makes generation a critical step in the training loop — and also a major bottleneck. To speed up generation, TRL integrates with vLLM. This combination lets you train powerful models more efficiently in GRPO setup.

Why it Matters

The co-located vLLM approach has significant implications for the industry. By sharing GPUs between training and inference, we can reduce waste, increase efficiency, and lower costs. This is particularly important for online learning methods like GRPO, where generation happens constantly.

With co-located vLLM in TRL, you can train powerful models more efficiently, without sacrificing performance or increasing costs. This opens up new possibilities for researchers and developers working with LLMs.

What Comes Next

The integration of co-located vLLM in TRL is a significant step forward for the industry. As researchers and developers continue to explore this approach, we can expect to see even more efficient and effective training methods emerge.

In the near future, we can expect to see more widespread adoption of co-located vLLM in TRL, as well as further research into its applications and limitations. This will help to drive innovation and progress in the field of LLMs.

Key Facts

vLLM was previously only supported in server mode, running on separate GPUs from the training job.
The co-located vLLM approach shares GPUs between training and inference, reducing waste and increasing efficiency.
TRL supports two modes for integrating vLLM during training: server mode and colocate mode.
Co-locate mode runs vLLM inside the trainer process and shares GPU memory with the training model.
The co-located vLLM approach is particularly important for online learning methods like GRPO, where generation happens constantly.