Characterizing and provisioning the environment bubble in agentic-RL LLM rollout

Reinforcement-learning (RL) post-training of tool-using LLM agents leaves expensive rollout GPUs idle while each trajectory waits on CPU-side environment work: sandbox cold-start, code execution, retrieval, and external API calls. We term this idle the environment bubble and measure it directly on 8×A100 hardware using a 20 Hz NVML profiler that avoids the kernel-residency pitfall in the utilization counter and attributes each idle instant to one of three states: env-wait, straggler, or pipeline. Our central finding is that the magnitude of the bubble is governed by the environment tool’s cold-start cost rather than by the RL system. Two real-GPU regimes bracket this behavior. Under an aggressive injected coldstart (warm_s=8 s, representative of heavy tools such as fresh containers, VM or browser spin-up, and a costly env.reset), env-wait is large and decreases from 0.80 at k=4 to 0.23 at k=64. Under a real python:3.11-slim Docker codeexecution backend, whose cold-start we measure at 0.3–1.2 s, env-wait is small, decreasing from 0.049 to 0.008 over the same range, while effective utilization rises to 0.42–0.55. The large values therefore require heavy tool cold-start; under realistic sub-second tools only a few percent of idle is recoverable, and the tool latency of the target system should be measured before optimization. Increasing the number of sandbox workers reduces the bubble, but the benefit diminishes as coldstart decreases. We model the tier as a closed finite-source (M/M/k//C) queue; because the model overstates the achievable worker savings, we size k from the measured idle-vs-k curve. A lightweight next-tool-call predictor anticipates calls on real rollout token streams at AUC 0.81 without an additional forward pass. Finally, on real hardware speculative prewarming does not recover meaningful GPU idle in either regime, and a naive simulator (a persistent warm pool evaluated with too few seeds) can indicate otherwise. This is a measurement and reality-check study, and each reported value is labeled by how it was obtained.