← Key papers

Paper review · November 19, 2026

AscendGYM: A cost-aware benchmark for Ascend NPU kernel optimization based on real workloads

AscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs

Data & evaluationEfficiency & infraAgents Advanced Upvotes 91 cs.AI SLAI-AITP

The authors built 48 Ascend kernels drawn from real training and inference workloads, plus STS, a metric that accounts for token cost, to measure agents' ability to optimize existing kernels.

Problem

Existing Ascend kernel benchmarks focus on the ability to generate new kernels. They do not evaluate well the ability to make existing implementations faster under the input shapes of real workloads. Looking only at the final speedup also hides the search cost needed to get that result.

Method

The authors profiled real workloads such as MindSpeed-LLM and vLLM-Ascend to select bottleneck kernels. They attached test cases that preserve data types, tensor shapes and execution context, forming 48 tasks across 3 difficulty levels. AscendForge, an execution sandbox based on SandboxFusion, handles compilation, correctness checks, performance measurement and feedback. There are two evaluation modes: one compares models alone using a default harness (AuraKernel), and the other evaluates complete model-harness combinations such as Claude Code and Codex. As metrics, they proposed STS, which penalizes improvements obtained late relative to the final speedup, along with CTG, GM Speedup, fast_p and pass rate.

Results

Limitations

This report evaluated only the easiest batch, Batch 1 (17 tasks), and deferred Batch 2 and 3 results. It is specific to Ascend NPUs. The STS weights (100, 50) are a design choice, and the provided text does not show how stable the rankings are under other weights.

Why it matters

For teams that want to optimize kernels with coding agents, it provides an evaluation framework that looks at both "how much faster did it get" and "how many tokens did it take to get there." Because it uses real workload shapes, gains on the benchmark are more likely to carry over to production performance.

Key terms

Kernel optimization
The work of modifying low-level code that performs the actual computation on NPUs or GPUs to reduce latency. It directly affects the efficiency of LLM training and inference.
STS(Speedup–Token Score)
A metric that rewards the final speedup (log2) and penalizes improvements more the later they are obtained in the token budget. For the same speedup, the one found sooner scores higher.
Harness
An agent execution framework that wraps an LLM and manages tool calls, search strategy, experience accumulation and so on. Examples include Claude Code and Codex.

Original

Authors: AI Training Platform Team, Bingwei Zhang, Jing Li, Amir Abdullin, Sihang Chen, Xin Chen, Liangjun Feng, Yingmeng Gao, et al. · Posted 2026-11-13 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.