UserArena: An Interactive User Simulation Benchmark for Long-Horizon Shopping Agents

Problem

Training and evaluating recommendation agents requires simulators that mimic the shopping environment and user responses. However, existing data rarely contains fine-grained real-user trajectories spanning multiple sessions. Evaluation also checks only next-action agreement and rarely verifies whether simulation conclusions match live-service A/B results. Learned simulators also suffer from behavioral distortion, sparse supervision signals and exposure bias.

Method

The authors build a benchmark covering 11,010 users, 64,729 sessions, 566,102 products and 13 action types from a large Asian e-commerce platform. At every step it aligns the user profile, screen, product context and action, and it provides an interactive environment that reconstructs the feed and detail pages. Evaluation has five levels, from L0 (output validity) to L4 (outcome fidelity such as CTR and add-to-cart rate). OPTD uses an LLM to summarize shopping intent from the full trajectory and gives it as privileged information only to a frozen Teacher. The Student follows the Teacher's token distribution with a generalized JSD on responses it generates itself. A per-token KL budget (trust region) limits the influence of the intent, and a teacher-forcing loss on real responses is used alongside.

Results

• The cross-session trajectories consist of 5,768 users and 59,378 sessions, with an average of 10.29 sessions and 86.36 actions per user.
• The authors report that, unlike existing benchmarks (Qilin, OPeRA, OmniBehavior and others), it is the only benchmark that has all of real traces, visual context, the full action space, long-term history and live-service A/B validation.
• In a comparison standardized on Qwen3.5-2B, they report that OPTD achieved the highest overall fidelity among text-based (RecAgent, Agent4Rec, Shop-R1 and others) and GUI-based (A/B Agent, STA) simulators.
• They report that in offline simulation, OPTD most closely reproduced the policy effect observed in a live-service A/B test.

Limitations

The data comes from a single platform, and the live-service A/B validation is based on a comparison of one pair of policies. The main comparisons are limited to the 2B model scale. Intent extraction requires a large LLM (Qwen3.5-397B). The values in the main results table are cut off in the provided text, so detailed metrics could not be checked.

Why it matters

Developing recommendation and shopping agents without costly online experiments requires reliable user simulators. The validation framework that checks simulation results against live-service A/B tests can be used directly by commerce companies to judge how much to trust offline evaluation.

Key terms

User simulator

A model that mimics how a real user would respond to a screen through clicks, scrolls, purchases and so on. It is used to evaluate recommendation policies offline or to train them with reinforcement learning.

On-policy Distillation

A training method in which the Student learns to follow the Teacher's per-token distribution on responses the Student generates itself. It reduces the train-inference mismatch of teacher forcing, which uses only ground-truth prefixes.

Privileged information

Extra information that only the Teacher can see during training. Here it is the shopping intent extracted from the full trajectory, and the Student cannot see it at inference time.

Trust region

A mechanism that limits how much the intent information can change the Teacher's distribution to within a KL divergence budget. It prevents the Student from overfitting to information it cannot see.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Project
• Hugging Face

Authors: Jujia Zhao, Jiakai Tang, Jing Yu, Wen Chen, Jian Wu, Han Zhu, Fei Sun, Suzan Verberne, et al. · Posted 2026-10-22 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.