Deep dives · 2026-11-13 · Advanced · By bombong
Remote Labor Index (RLI): How much real work do AI agents actually finish?
The Remote Labor Index (RLI) is a benchmark that measures how much remote work AI can automate by handing AI agents real, paid projects from multiple sectors to complete from start to finish. In the first published results, even the best-performing agent reached an automation rate of only 2.5%, putting a number on the gap between test scores and real work ability.
Contents
- Why it was needed: the gap between test scores and real work
- Core idea: real delivery instead of a written exam
- How it works: hand over the whole project and judge the deliverable
- Reading the results: what a 2.5% automation rate tells us
- Not confusing it with a different index of the same name
- Limitations, open issues and practical criteria
- Further study
- Instead of knowledge and reasoning questions, RLI uses real projects with economic value as tasks to measure an AI agent's ability to see work through to the end.
- Because the unit of evaluation is an entire project rather than a single question, the score directly reflects a key trait of real work: a mistake in just one intermediate step can make the whole deliverable useless.
- In the first published results, the best-performing agent's automation rate was 2.5%. This means the agents stayed near the floor.
- RLI aims to ground discussions of AI automation in measurement rather than speculation, and to serve as a common yardstick for tracking change over time against the same standard.
- A 2020 economics study also has an index with the same name, but it is a different concept that measures how feasible working from home is for each occupation.
Why it was needed: the gap between test scores and real work
AI models have been rapidly raising their scores on research benchmarks (test sets used to compare performance) that probe knowledge and reasoning. However, the RLI paper paper points out that it is unclear how much this progress translates into economic value or the automation of real work. Being good at math problems does not guarantee being able to take a job a client has assigned all the way to delivery.
This gap is especially problematic for policy and business decisions. Discussions such as "how many jobs will AI replace" have mostly relied on estimates based on benchmark scores, demo videos or job descriptions. When what is being measured is not real work, both optimistic and pessimistic arguments rest on weak grounds.
So RLI changes the question. Instead of asking "how smart is the model," it asks "if you hand it work that people are actually paid to do, does it finish the job?" The paper states that its goal is to put discussions of AI automation on an empirical footing and to create a common standard that stakeholders can use to prepare for labor automation in advance paper.
Core idea: real delivery instead of a written exam
As an analogy, existing benchmarks are like a written exam for a new hire, while RLI is like assigning a real client project during the probation period. A written exam breaks knowledge into small pieces and gives partial credit for each question. A real project requires understanding the requirements, handling materials, using several tools and, in the end, producing a deliverable the client will accept. If any one of these is missing, the delivery fails.
The RLI paper paper summarizes this design in three characteristics. First, the tasks are "real-world, economically valuable" projects. Second, they are not concentrated in a particular field but span multiple sectors (broadly multi-sector). Third, the benchmark evaluates agent performance from start to finish (end-to-end) in a practical setting.
This choice comes at a cost. Real projects do not have a single correct answer, which makes automatic grading difficult, and preparing and evaluating each task is expensive. What it gains in return is validity. A higher score can be read as an increase in "work that can actually be handed off." If question-based benchmarks chose ease of measurement and reproducibility, RLI chose to reduce the distance from reality.
Research benchmarks vs. RLI
Research benchmarksWritten exam
- Measures knowledge and reasoning with questions
- Easy to auto-grade and reproduce
- Partial credit adds up
- Unclear link to economic value
RLIReal delivery
- Real paid projects as tasks
- Spans multiple sectors
- Evaluated start to finish
- High cost to prepare and judge
공통 Both are yardsticks for comparing AI capability in numbers
How it works: hand over the whole project and judge the deliverable
Conceptually, the RLI evaluation flow can be understood in four steps. First, remote work projects that are actually commissioned across multiple sectors are collected as tasks. Next, each project is handed over in its entirety to an AI agent (an AI system that uses tools and carries out multiple steps on its own). The agent reads the instructions, does the necessary work and produces a deliverable, much as a person working remotely would.
Third, the deliverable is judged on whether it would be acceptable by real work standards. Finally, these judgments are combined into a single number called the "automation rate." Read literally, it refers to the share of assigned work that AI completed in place of a person. For the specific grading procedure and task composition, it is best to check the original paper.
What matters in this structure is that partial credit does not come easily. On a question-based test, getting 9 out of 10 steps right earns close to 90 points, but in a project, getting the last step wrong can make the entire deliverable useless. In an agentic workflow that goes through many steps, small mistakes tend to accumulate into total failure, and RLI exposes exactly that cumulative effect in its score.
Reading the results: what a 2.5% automation rate tells us
According to the RLI paper paper, the AI agents it evaluated performed near the floor, and the best-performing agent's automation rate was 2.5%. This contrasts with the near-human scores commonly reported on research benchmarks.
Reading this number as "AI is useless at that level" would be wrong. What RLI measures is the completion rate when an entire project is handed over without human involvement. That is a different question from productivity gains in collaborative setups where people split up the work, review it and fix it. Even with a low automation rate, AI may already be quite useful for some steps.
Conversely, RLI serves as counterevidence to the claim that "high benchmark scores mean AI will soon replace people." The 2.5% figure shows that turning knowledge and reasoning ability into completed real work requires other factors, such as the ability to carry long tasks through consistently and the ability to check the quality of one's own output. This is also why the paper presents RLI as "a common standard for tracking AI's impact." Measuring repeatedly with the same yardstick makes it possible to see when and how much this number moves.
First published RLI results
Not confusing it with a different index of the same name
The name "Remote Labor Index" also appears in a 2020 economics study. A study analyzing the supply and demand shocks of the COVID-19 pandemic paper built an index measuring how feasible working from home is for each occupation and gave it the same name. Based on this, the study estimated shocks by industry and occupation, and reported that high-wage occupations were relatively resilient to the shocks while low-wage occupations were far more vulnerable.
The two indexes share only a name; they measure different things. The 2020 index measures "whether people can work from home," while RLI, the AI benchmark paper, measures "whether AI agents can complete remote work projects from start to finish." There is no evidence in the source materials that the former led to the latter, so the two should not be treated as related. When searching for or citing sources, it is best to check the year and field to tell them apart.
Limitations, open issues and practical criteria
First, there is the issue of representativeness. Even though it spans multiple sectors, the task set is a sample. The automation rate is a number on that sample, so it should not be restated as "X% of all remote labor will be automated." You should first check which sectors and what kinds of projects are included.
Second, the judging criteria shape the results. Deliverables from real projects do not have a single correct answer, so the automation rate can vary depending on who decides what is "acceptable" and by what standard. Third, there is the issue of timing. The 2.5% figure is the result for the agents evaluated when the paper was written. As models and harnesses (Agent Harness, the framework that wraps a model and provides tools and an execution environment) change, the number can change too, so when citing it, state both the time of evaluation and what was evaluated.
In practice, here is a good way to use it. When considering adopting agents in-house, use results like RLI less as evidence of "what AI can do" and more as grounds for deciding "where a person must be involved." If you build your own evaluation, include, alongside question-based Evals (evaluation sets), at least a small number of project-based evaluations that hand over one real piece of work from start to finish. Common mistakes are setting the scope of automation based on a single research benchmark score, and giving up on step-by-step assistance just because the completion rate is low.
When citing or using RLI results
Interpretation
- Did you state the evaluation time and the agents evaluated?
- Did you check the sector mix of the task sample?
- Did you check the judging criteria?
- Did you distinguish it from the 2020 work-from-home index?
Application
- Did you use it to decide where a person must be involved?
- Did you also include project-based evaluation?
- Did you separately consider step-by-step assistance?
Further study
The best place to start is the original RLI paper paper. After confirming the problem framed in the abstract (the gap between benchmark progress and economic value), look in turn at how the tasks were collected, by what criteria the deliverables were judged and how the automation rate was calculated. This will let you assess the limitations described above for yourself.
The 2020 economics study with the same name paper is a different concept from the AI benchmark, but it shows how to quantify labor shocks at the level of occupations and industries. Readers who want to connect RLI results to labor market discussions can use it as reading material for comparing "what unit is being measured." Related concepts worth exploring include AI agents, computer use, harnesses and data contamination in benchmarks.
Related terms
AI agent Agentic workflow Benchmark Evals Computer use Harness Human-in-the-loop Leaderboard
References
Foundational papers
Recent research
Written by AI from the listed references and reviewed. Please check the original sources for details.