Paper review · October 21, 2026
LIBERO-Industry: Benchmarking robot manipulation capability under industrial process constraints
LIBERO-Industry: Benchmarking Robotic Manipulation under Industrial Process Constraints
A 216-task benchmark was built that turns industrial constraints such as conveyor speed, execution windows and placement tolerance into adjustable conditions, and all five general-purpose robot policies recorded a 0% success rate.
Problem
In flexible manufacturing settings such as high-mix low-volume production, robots must adapt to situations where products and production conditions keep changing. Such settings impose process constraints such as product variants, moving material flows, fixed execution windows and tight geometric tolerances. However, existing robot learning benchmarks (LIBERO, CALVIN, Meta-World and others) center on household environments and do not address these timing, motion and placement requirements, while manufacturing benchmarks such as ARIAC do not evaluate general-purpose robot policies.
Method
LIBERO was extended to create three task groups: D2S (picking from a moving conveyor and placing into a fixed jig), S2D (picking from a fixed location and placing into an intermittently moving slot), and D2D (moving from a moving location to a moving location). Combining 4 object types, 3 conveyor speeds, 2 motion directions and 3 difficulty levels produced a total of 216 tasks (180 for training, 36 for evaluation). As difficulty increases, placement precision rises from a wide tray to a recess matching the object's shape to a tight-fitting inlay, and environmental variation also grows. Each task has a fixed rollout time limit derived from the material movement schedule, and the same limit applies to all policies. The metrics are success rate (SR) and two partial completion rates (PCR_T: progress in transferring the target object; PCR_S: penalized for disturbing objects already in place).
Results
- Five public checkpoints fine-tuned on LIBERO (π0.5, OpenVLA-OFT, UniVLA, X-VLA, Cosmos Policy) all scored a 0.0% success rate across 36 evaluation tasks and 360 rollouts per policy.
- The target transfer partial completion rate (PCR_T) was also 0.0% across the board, apart from 0.4% for π0.5. PCR_S was 11.2-12.5%, a score that came from not disturbing objects that were in place from the start.
- The authors report observing a pattern of gradual performance decline as industrial constraints were added one by one and combined.
- The authors report that, under identical fine-tuning and evaluation conditions, co-training with LIBERO-Industry data produced a higher target transfer completion rate in real-robot D2S trials on an AUBO-i5H arm than co-training with the original LIBERO data.
Limitations
Because every policy had a 0% success rate, the benchmark does not discriminate between models, and while it is suitable for measuring zero-shot transfer in its current form, it is hard to use for tracking incremental improvement. It should also be taken into account that the evaluated checkpoints were fine-tuned on household-environment LIBERO and had no opportunity to learn the industrial tasks. The evaluation set is small: 36 tasks with 10 initial states per task. Real-robot validation was limited to the single D2S scenario, and the comparison result was presented as a statement that it was "higher" without specific numbers. The authors state explicitly that the time limits set in the paper are benchmark values, not actual factory takt times.
Why it matters
It shows in numbers that the scores VLA models achieve on household-environment benchmarks do not carry over to factory conditions. When moving materials, fixed timing and narrow tolerances apply together, current general-purpose policies could not complete a single task. It serves as a starting point for diagnosing what is missing for those trying to apply physical AI to manufacturing sites.
Key terms
- VLA (Vision-Language-Action) model
- A general-purpose robot policy that takes camera images and language instructions and directly outputs robot actions. RT-2 and OpenVLA are representative examples.
- High-mix low-volume production (HMLV)
- A mode of production with many product types and a small volume for each type. Production changeovers are frequent, so conventional fixed automation carries a heavy reprogramming burden.
- Execution window
- The limited time interval in which an object coming down a conveyor can be picked up. If the task is not finished within it, the opportunity is lost.
- Partial completion rate (PCR)
- A metric that measures transfer progress instead of a binary success/failure. PCR_T covers only the transfer of the target object, while PCR_S also reflects whether existing objects were disturbed.
Original
Authors: Ming Shin Cindy Sung, Duan Leijiahao, Haoxiang Wang, Boning Tan, Liangna Fu, Chao Tan, Yafu Li, Yulong Chen, et al. · Posted 2026-10-12 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.