Alibaba’s QwenGyre Tackles The Million-Token Problem With Elastic GPU Scheduling For XLong RL
Alibaba has launched QwenGyre, a reinforcement studying framework designed for what the analysis group calls “XLong-horizon” agent coaching — duties wherein a single rollout can run for hours, contain a whole bunch of mannequin–setting interactions, and eat shut to 1 million tokens. As LLM brokers transfer from resolving particular person repository points to constructing total codebases, standard on-line RL pipelines wrestle with two issues: extreme variance in rollout length leaves GPUs idle whereas a number of straggler executions end, and branching execution paths generate huge redundant coaching knowledge.
The first element, the Elastic Scheduler, addresses GPU underutilization. Traditional frameworks both alternate a whole GPU pool between rollout and coaching (colocate), leaving {hardware} idle throughout lengthy rollout tails, or statically partition GPUs between the 2 levels (async), stopping both facet from borrowing the opposite’s spare capability. QwenGyre as an alternative organizes {hardware} into independently switchable “cells” that reallocate between inference and coaching as the amount of unfinished rollout work modifications. Live harness executions are by no means interrupted: mannequin requests are rerouted to remaining rollout engines mid-flight, with KV caches migrated over RDMA to keep away from recomputation. Newly freed cells be part of an ongoing coaching step as extra data-parallel replicas, pulling the authoritative weights from a core cell, whereas streaming coaching distributes micro-batches in keeping with every group’s progress so cells end in step.
Role transitions are quick — averaging 8.5 seconds from rollout to coaching and three.5 seconds again — negligible in opposition to rollout executions lasting hours.
Taming Branching Trajectories
The second element, the Trajectory Processor, handles the coaching knowledge itself. At XLong horizons, agent harnesses compact saturated contexts, delegate subtasks to sub-agents, and retry failed paths, so one execution yields a branching tree of divergent context paths fairly than a linear trajectory. Naively increasing each path duplicates shared prefixes and inflates coaching value. QwenGyre reconstructs the execution as a prefix-shared trajectory tree, preserving every mannequin output’s precise unique conditioning context. Only tokens truly generated by the coverage are skilled on, and shared targets are counted as soon as by on-tree masking.
The framework additionally scores partial progress: executions that hit their timeout are nonetheless evaluated on their preserved workspace state, so partial makes an attempt carry significant rewards fairly than being discarded. At coaching admission, a bounded variety of trajectories — capped by function precedence (most important agent first, then sub-agents, then abstract paths) — is chosen per execution, with token losses averaged inside every execution to forestall path depend from skewing weighting.
Scaled to Alibaba’s flagship Qwen 3.8-2.4T mannequin at roughly 700K tokens per rollout, QwenGyre improved the NL2RepoBench cross fee from 52.5% to 58.5% in 48 coaching steps, finishing them in 75.4 hours versus 134.6 for async and 91.5 for colocate baselines — speedups of as much as 1.78× and 1.85× respectively, achieved whereas matching baseline coaching scores. Across the three evaluated workloads — NL2RepoBench, DeepSWE, and TerminalBench — positive factors held persistently beneath equal GPU budgets.
The put up Alibaba’s QwenGyre Tackles The Million-Token Problem With Elastic GPU Scheduling For XLong RL appeared first on Metaverse Post.
