Paper review · October 9, 2026
Kuration SDK: Closing the Virtual2Real Gap with Data Curation
Kuration SDK: Addressing the Virtual2Real Gap via Data Curation
By comparing diffusion world models trained on CS:GO and CS2 gameplay, the authors showed that FVD and LPIPS do not reflect actual playing quality, and used a data diagnostic tool to narrow the cause down to differences in how much the screen changes in response to actions.
Problem
Quality metrics for action-conditioned world models are not yet established. It is also unclear whether video similarity metrics such as FVD, LPIPS and JEDi reflect actual 'playability' (teleportation, passing through walls, control responsiveness, and so on). Benchmarks based on action semantics or physics also require game-specific estimators or probes, so they do not provide a general-purpose signal. The authors call this gap the Virtual2Real gap.
Method
The authors first reproduced DIAMOND on public CS:GO data to confirm that their training and evaluation pipeline matched the original paper. They then trained an uncurated baseline_v1 on CS2 data matched in map (de_dust2), frame rate, action encoding and scale. Next, they used the open-source Kuration SDK to measure diagnostic properties of the data. These include pixel_shift, which measures optical flow displacement right after a key press, action_state_consistency, collision detection, and SLAM-based trajectory reconstruction. Based on these measurements, they compared four curation strategies: coverage-based, rare-trajectory upweighting, collision-aware, and action-state-consistency-based.
Results
- In the reproduction, FVD was 4.4% lower and LPIPS 4.2% higher than DIAMOND's reported values, which the authors reported as within evaluation noise. Evaluating the public weights with their own pipeline matched the original paper's numbers to within 1%.
- On the CS2 test set, baseline_v1 had an LPIPS of 0.4785, 20.9% better than DIAMOND (0.6049), and its FVD was similar or better. However, the authors reported that its playability was lower in interactive rollouts. The best FVD came at epoch 60 out of 600, so it was also unsuitable as a criterion for when to stop training.
- In the action-state consistency analysis, the response magnitude to a single key press in CS:GO was 1.4–2.1× that of CS2. After the key was released, movement magnitude in CS:GO actually grew to 112–126% of that during the press, while in CS2 it fell to 76–78%. Backward/forward symmetry was 26% for CS:GO and 80% for CS2.
- Of the four curation strategies, coverage-based, rare-trajectory and collision-aware improved some metrics but did not change playability. Only action-state consistency selection, based on the diagnostic results, was reported to qualitatively improve playability over baseline_v1.
Limitations
Playability was judged not by a formal protocol but by the authors themselves watching rollouts without blinding, so expectation bias cannot be ruled out. For the action-state consistency strategy, quantitative metrics and the training schedule are not yet finalized, so only qualitative results exist. LPIPS, FVD and JEDi could not be measured for some strategies because of a GPU shortage. There is no controlled ablation separating response magnitude from momentum and symmetry, and no negative control such as curation in the opposite direction. The two datasets also come from different engine generations (Source vs Source 2), whose effect may be partly mixed in.
Why it matters
The paper gives a concrete example showing that choosing data or models for game and robot world models by video quality metrics alone can miss real controllability problems. It also offers a practical diagnostic direction, measuring the action-response properties of data before training, along with public tools.
Key terms
- Action-conditioned World Model
- A model that takes past frames and user input (actions) and predicts the next screen. It is used for game simulation and as a virtual environment for training robot policies.
- Virtual2Real gap
- A term defined in this paper for the mismatch between scores on metrics such as FVD and LPIPS and the quality you see when actually playing. The paper treats it as a problem stemming from the training data distribution rather than from simulator accuracy.
- FVD / LPIPS
- FVD and LPIPS are quality metrics for generated video: FVD measures the distance between distributions of video features, and LPIPS measures frame-level perceptual differences. Neither catches physics violations that span multiple frames, such as teleporting or passing through walls, very well.
- Action-State Consistency
- A data property describing whether the same input always produces similar on-screen motion. In this paper it is measured by optical flow displacement after a key press.
Original
Authors: Nirmit Desai, Eric Song, Mayank Sengupta, Tejal Bedmutha, Siri Reddy, Sahiti Dharmavaram, Kunal Sawarkar · Posted 2026-10-07 · License CC BY-SA 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.