The authors of Game2Player report a framework that scales verifiable gameplay data for visual agents by co-developing game environments with the players that test them. By automating the adaptation of web games into state-verified environments and distilling 15.7K trajectories into a 9B model, the team reports doubling task progress on a held-out benchmark and outperforming several proprietary models on an external test set.
Environment–player co-development
Collecting high-quality gameplay data for training visual agents traditionally requires expensive human recordings or lossy inference from video. Existing interactive benchmarks also cover a narrow range of titles and frequently lack explicit tasks or verifiable success signals. Game2Player addresses this by treating environment construction and player testing as a single, iterative loop.
The framework begins with a Builder agent that locates web games capable of offline, single-player execution. The Builder adapts each title into a sandboxed runtime, hooks into the underlying engine, and exposes internal state through two distinct interfaces. The state probe exposes scoring variables like health or resources, while the player observation exposes only what a human could infer from the screen, such as object positions. The Builder then defines tasks as tuples containing a natural language instruction, a start routine, a progress function, and a success predicate. Both functions read exclusively from the state probe, enabling automated scoring without human or model judges.
Rather than trusting the Builder's initial implementation, the framework deploys two complementary players to test every task. A Coding Player writes a Python policy that reads the player observation and executes in real time without model calls. A VLM Player acts directly on screenshots, pausing the game at each step to reason and select an action. If either player encounters a broken menu, a missing state variable, or a progress measure that fails to update, it files a defect report or an observation request. The Builder replays the recorded episode to verify the claim, then repairs the environment code, state probe, or control guide accordingly. This cycle repeats until the environment passes a scripted gate and the code policy solves a sufficient share of tasks.
Illustrative image: Surprising_Media / Pixabay
"Play serves as both environment validation and training data collection."
Defect reports typically flag issues like menus hiding critical UI elements, keys with no mapped effect, or success predicates firing prematurely. Observation requests come exclusively from the Coding Player, naming state variables that a code policy requires but cannot infer from the screen. The pipeline processed games from itch.io, Newgrounds, and similar portals, producing 568 accepted environments across 25 genres and seven major engines. The team holds out 50 of these games, comprising 287 tasks, as PlayerBench, a dedicated test set for evaluating game agents on unseen titles.
Progress-weighted distillation and benchmark results
Once an environment is accepted, both players record trajectories for training. The Coding Player generates high-volume demonstrations by rerunning policies from new seeds, while the VLM Player provides visually grounded action notes and reasoning traces. The two sources combine into a single dataset of 15.7K trajectories, totaling 307 hours of gameplay. Code policies supply 87.7% of the steps, running 22 times faster than the VLM Player and incurring lower inference costs after roughly 12 minutes of gameplay. A VLM Player costs between $3.20 and $54.70 per hour of gameplay depending on the model, while a code policy costs $0.13 to $11.60 to write and $0.13 per hour to rerun.
The authors distill this data into a visual agent using progress-weighted supervised fine-tuning. Standard training treats all steps equally, but gameplay trajectories contain long stretches of waiting, wandering, or recovery that do not advance the objective. The framework calculates a weight for each step based on the measured progress signal. Actions immediately preceding a rise in progress receive higher credit, while repeated actions that yield no improvement or the final steps of failed episodes are down-weighted. Instead of applying a continuous loss weight, the framework rounds each weight to an integer multiplicity and resamples the training set, effectively forcing the model to prioritize milestone-advancing decisions.
Training a 9B model on this weighted dataset more than doubles its baseline performance. On PlayerBench, progress rises from 14.3% to 31.2%. The gains transfer to the external GameWorld benchmark, where the distilled model reaches 53.1% progress, up from 30.0% for the base model. The authors note that this score exceeds the published results of several proprietary models on the same benchmark.
Illustrative image: biancamentil / Pixabay
| Benchmark | Base Model Progress | Game2Player-9B Progress |
|---|---|---|
| PlayerBench | 14.3% | 31.2% |
| GameWorld | 30.0% | 53.1% |
The framework also isolates the contribution of each player type. Ablations indicate that the code policies drive the bulk of the step volume, but the VLM Player's reasoning notes and visual grounding are necessary for tasks where no code policy can be written. The authors further test the co-development loop against a generic agent setup that lacks player feedback. The full loop increases the share of correctly implemented tasks from 65% to 72% and allows code policies to solve 46 tasks compared to 34 in the no-reports setting.
Licensing and scope
The Game2Player collection relies on adapting existing web games rather than generating synthetic environments from scratch. The authors acknowledge that only a small share of these titles carry open-source licenses. Most are freely playable on public portals but remain subject to their original creators' rights. The team does not redistribute the games, their assets, or the instrumented environments, releasing only the framework source code, the trained models, and the task specifications.
The dataset is restricted to single-player games that run offline, explicitly avoiding multiplayer titles whose terms of service typically prohibit automated play. The authors also ran a human play study on PlayerBench to establish a baseline for first-time users. Fifteen participants played tasks in their own browsers for up to 10 minutes per attempt. The recorded inputs and state traces confirmed that the progress functions accurately reflected advancement toward the stated goals, and no scoring errors were found. Because humans play in real time while the model pauses the game at every step, human scores serve as a reference for first-time play rather than a hard upper bound.
The results are reported on an unreviewed arXiv preprint, and the benchmarks measure progress toward state-based objectives rather than raw win rates. The authors provide the code and models for independent replication, but the proprietary nature of the underlying game assets limits direct commercial application of the collected environments.
Sources
PNEUMETRON EDITORIAL
Pneumetron's pipeline pairs AI-assisted drafting with human editorial review before publishing — our goal is to make staying informed easier, not to replace real reporting.




