AI Trend Research

← AI glossary

AI glossary · Advanced · AI security

Model extraction

Model extraction is an attack in which an attacker sends a large number of queries to an AI model, collects the answers, and builds a copy model that behaves similarly to the original.

About this term
Level
Advanced · Specialist terms that need the inner workings
Added
2026-10-09
In plain words It is like ordering a famous restaurant's dishes over and over, tasting them, and recreating a similar recipe. Real attacks, however, place far more orders automatically than any person could taste, and the goal is to produce 'almost the same taste' rather than steal the original recipe outright.

How it works

The attacker uses an access point anyone can query, such as an API or web service. They automatically generate a wide variety of queries, send them in bulk, and collect the model's answers (sometimes including probability scores) into a dataset. Training their own model on this data produces a copy that imitates the original model's behavior fairly closely. The principle is the same as Distillation, where a small model is taught from a large model's answers, except that it is done without permission.

How a model is copied through queries and answersAutomatic query generation → Public API(Queries); Public API → Original model; Original model → Query–answer dataset(Collect answers); Query–answer dataset → Copy model training; Copy model training → Copy modelAutomatic querygenerationMany diverse queriesPublic APIOriginal modelQuery–answerdatasetSometimes includesprobability scoresCopy modeltrainingCopy modelBehaves similarlyQueriesCollect answers
How a model is copied through queries and answers The attacker never sees inside the original model; they collect only queries and answers to train a copy model.

Why it matters

Building a good model takes a lot of data and compute cost, so the model itself is a core company asset. If it is copied, others effectively take that investment cheaply. An attacker who obtains a copy model can also experiment on it freely to discover follow-up attacks in advance, such as jailbreaks or adversarial examples. For this reason, model extraction is treated as both an intellectual property issue and a security issue.

Good to know

Defenses include usage limits, detecting unusually high query volumes or odd query patterns, hiding details such as probability scores, and terms of service that forbid use for copying. There are also attempts to embed watermarks in outputs so copying can be verified later. However, it is hard to perfectly distinguish normal users from attackers, so there is a trade-off: blocking too aggressively degrades service quality. It is distinct from hacking that directly steals model weight files; the term usually refers to imitation through queries and answers alone.

Model extraction defense checklist

Access control

  • Are there usage limits?
  • Are odd query patterns detected?
  • Are details such as probability scores hidden?

After-the-fact response

  • Do the terms include a no-copying clause?
  • Can copying be verified via output watermarks?
The main defenses described in the text, grouped as checklist items.

Example

Companies that offer commercial LLM APIs often include a clause in their terms of service saying 'do not use outputs to train competing models'. They also detect accounts that send large volumes of similarly formatted queries in a short time and slow them down or block them. These measures aim to prevent model extraction, in which someone collects API answers and builds their own model through Distillation.

Related terms

Distillation AI watermark Adversarial example Jailbreak Membership inference attack Training data extraction