Small language models (SLMs) and on-device AI: AI that runs in your pocket

Why small models were needed

Large language models (LLMs) usually run in huge data centers, and users send questions over the internet and receive answers. They perform well, but at a cost. The study on Menta paper, a mental health prediction model, pointed out that the size and compute requirements of LLMs stand in the way of real deployment. The study on Quecto-V1 paper, an Indian legal model, noted that most current legal AI relies on 7 billion (7B) or more parameters and cloud inference, which makes it hard to use in resource-constrained settings and creates data sovereignty risks.

On-device AI does this computation inside the user's device instead of on a server. A case study that put SLMs into a mobile game paper summarized the benefit as 'fully offline, private AI that does not rely on the cloud and where data never leaves the device.' The study on LiteLMGuard paper, an on-device guardrail, also cited stronger privacy, lower latency and serverless operation as advantages.

The problem is that a phone's memory, battery and chip performance are nowhere near those of a data center. So models small enough to fit on a device, that is, small language models (SLMs), became necessary. There is no fixed definition of 'small.' The studies covered in this article range from around 120 million parameters to several billion, and even include a 30-billion-parameter model that runs on a consumer GPU. It is more useful to treat 'does it run on the target device?' as the criterion rather than size.

Core idea: a dedicated task assistant instead of an all-purpose encyclopedia

As an analogy, a cloud LLM is a large general hospital with every department, while an on-device SLM is a pocket-sized assistant dedicated to one field. The dedicated assistant cannot answer every question in the world, but it handles its assigned work quickly and quietly. It also never carries the documents outside.

The key point is that 'if you narrow the scope, a small model can do well enough.' Menta paper was trained jointly on six classification tasks from social media posts, such as depression, stress and suicide risk. As a result, it improved by 15.2% on average across tasks over the best SLM without fine-tuning. On depression and stress classification, it was more accurate than a 13-billion-parameter (13B) LLM about 3.25 times its size. Quecto-V1 paper chose to maximize 'lexical density' in the legal domain instead of broad world knowledge, and outperformed general-purpose SLMs on exact-match tasks for statutory provisions.

The trade-offs are also clear. Broad general knowledge, long and complex multi-step reasoning, and the stability to follow formats precisely are weaker than in large models. So on-device AI design is less about 'how big should the model be' and more about deciding 'what to hand to the model and what not to.'

How it works

1) Narrow the task and choose a model. You can take an existing model or build one from scratch. Quecto-V1 paper built a 124-million-parameter model with the GPT-2 architecture and trained it from scratch only on Indian statutory materials.

2) Train it for the task. A common method is LoRA. It leaves most of the original model's weights unchanged and trains only small additional matrices, customizing the model at low cost. Menta paper used a strategy of mixing multiple datasets in a LoRA-based framework and trained with a loss function that accounts for class imbalance. TinyLLM paper, which covered agent tasks such as function calling, compared SFT (supervised fine-tuning), parameter-efficient fine-tuning, reinforcement learning, DPO (direct preference optimization), and hybrid methods that combine them.

3) Compress with quantization. Quantization is a technique that represents weight values with fewer bits to reduce file size and memory. Quecto-V1 applied 8-bit quantization after training and packaged the model as a GGUF file. The model shrank by 74% to under 150MB and runs fully offline on an ordinary CPU.

4) Run within the device's memory and power. Menta showed real-time operation on an iPhone 15 Pro Max using about 3GB of RAM. Power has to be handled separately. DVFSLM paper redesigned DVFS (dynamic voltage and frequency scaling), which adjusts a chip's voltage and clock to the situation, for SLMs. In autoregressive generation, where tokens are produced one at a time, frequency, power and latency are intertwined in complex ways. How much the GPU, CPU and memory controller (EMC) each contribute to power and latency is also not visible from the outside. The study explained that, for these reasons, existing approaches wasted energy or slowed responses.

5) Put safety measures outside the model. At the input stage, add a guardrail that filters dangerous questions; at the output stage, add format checks and a fallback on failure. This is covered in detail in the limitations section below.

How it developed: from proving usefulness to operation and safety

All of the starting-point studies came out in 2025–2026. So rather than a long historical lineage, it is more accurate to see this as three directions being filled in at the same time.

The first is proof that 'small can still be useful.' Menta paper and Quecto-V1 paper showed the value of small models in the narrow domains of mental health and law. TinyLLM paper extended the scope to agent tasks such as tool and API calls. When TinyAgent, TinyLlama, Qwen, xLAM and others were evaluated on the Berkeley Function Calling Leaderboard (BFCL), models with 1–3 billion (1–3B) parameters were clearly better than ultra-small models under 1 billion. With hybrid optimization, overall accuracy reached up to 65.74% and multi-turn accuracy 55.62%.

The second is 'the problem of actually running them.' DVFSLM paper dealt with energy. The mobile game case study paper put Gemma 4 E2B (2.6B) and Qwen3 0.6B into Palabrita, an Android word game, accumulating 204 commits over five days (about 90 of them AI-related). Along the way, the initial design was substantially revised.

The third is 'safety and privacy.' LiteLMGuard paper proposed a lightweight question filter placed in front of a quantized SLM. The personal data substitution study paper handled, entirely on the device, the process of replacing personal data with fake values of the same type instead of erasing it with placeholders like [PERSON]. A 1.5-billion-parameter MoE classifier finds personal data spans, the 1-bit Bonsai-1.7B proposes substitute values for names, addresses and dates, and a rule-based generator handles items with fixed formats.

Where things are heading now

One trend is that the bar for 'small' is rising along with device performance. Muse Glimmer paper quantized a 30-billion-parameter (30B) model to about 4 bits so that it runs inference on a single GPU. It takes text and image input and packs multi-step reasoning, tool use and failure recovery into one model, released as an agent model that runs on consumer devices without the cloud. It was released under the Apache 2.0 license. 1–3B models for phones and tens-of-billions-parameter models for desktops belong to the same 'local execution' trend.

The other trend is a shift in emphasis from model architecture toward designing training data and evaluation. The call screening study paper addressed a scenario where a phone answers calls on behalf of its owner when the owner is unavailable, without sending call content off the device. Existing small models raised needless warnings even for ordinary callers, and in summaries they left out important conditions or recorded the caller's claims as facts. Because real calls have no reference responses, the researchers created synthetic calls with separated information boundaries. The writer playing the caller sees only a private scenario, the response writer sees only the conversation up to the current turn, and the summary writer sees only the full finished conversation. In this way they produced 3,395 calls and 15,102 training targets. Training a single LoRA adapter on six 0.6–4B models raised the response pass rate from 21–54% to 94–97% and the summary pass rate from 0–6% to 43–72%.

Putting the two trends together: hardware sets the upper limit on the size of model that can run on a device. Within that limit, what determines real quality is how you build data suited to the task and what you decide counts as a 'pass.'

Limitations and open issues

Quantization can change safety as well. LiteLMGuard paper warned that fairness, ethics and privacy risks arise along with compression, and that a quantized SLM may answer harmful questions directly even without attacks such as jailbreaks. The study's response is to filter questions in front of the model. An ELECTRA-based classifier judges 'is this a question that can be answered?' with 97.75% accuracy. It filters on the device in about 135ms on average, and its defense rate against harmful questions exceeded 85%.

Small models often violate formats and instructions. The mobile game case paper grouped failures into five types: output format violations, constraint violations, degraded context quality, latency mismatches and unstable model selection. At first, the model was asked to generate the entire JSON puzzle containing the word, category, difficulty and five hints. In the final structure, the word is taken from a pre-selected list, the model generates only three short hints, and on failure it returns to a predefined result. The techniques used together were multiple layers of defensive parsing, retries that report what failed, session replacement, progressively stricter prompts and reducing the model's responsibility.

Models also copy examples verbatim. In the personal data substitution study paper, when given three fixed examples, the 1-bit SLM output the examples' answers regardless of the input. It failed the same way with a 1.58-bit model, so quantization was not the cause. When examples were drawn per input from an example pool matched to the language region, all 482 calls succeeded. However, the study noted that a tendency to copy values remained with small example pools.

Efficiency figures differ by language. According to the tokenizer coverage study paper, the TinyLlama-1.1B tokenizer splits a Latin-script word into 1.34 tokens on average, but splits a Kannada word into 22.60. Because the script is barely present in the vocabulary, it falls back to byte-level splitting, yet no 'unknown token' marker appears, so the problem is not visible on the surface. As a result, an English user can fit 1,528 words into a 2048-token context window, while a Kannada user fits only 91 words. The researchers concluded that vocabulary, not parameter count, is the real constraint.

What to consider when choosing in practice

On-device SLMs fit these cases: when data must not leave the device (mental health, call content, personal data), when the system must run without a network (legal field work, resource-constrained environments), and when the task is narrow and repetitive (classification, short generation, predefined tool calls). Conversely, if open-ended Q&A requiring broad knowledge or long, complex planning is central, using a large model or offloading part of the work to the cloud is more realistic. For agent tasks, as TinyLLM paper found, models under 1 billion parameters can perform much worse, so it is safer to start evaluating from 1–3B.

There are four common mistakes. First, handing the entire result to the model from the start. As in the case study paper, it is more stable to have code and data handle the fixed parts and let the model generate only small pieces. Second, deciding on a multilingual service based only on memory and speed figures measured in English. Measure tokens per word in the target language first. Third, not re-running safety evaluation after quantization. Fourth, leaving fixed few-shot examples as they are. Check whether outputs copy the examples.

Battery and heat should also be treated as functional requirements. As DVFSLM paper pointed out, even small models use a lot of energy, which affects app quality and device reliability. It is worth getting into the habit of measuring not just accuracy but also the time and power each response takes.

Further study

If you are new to this, start with Menta paper and Quecto-V1 paper. Through short case studies, they show the basic pipeline of 'LoRA fine-tuning → quantization → on-device execution' and how small models compete with large ones on narrow tasks.

If you plan to build a product, the mobile game case study paper is the most practical. It lays out five failure types and countermeasures, making it a good design checklist. If you are considering agent features, use TinyLLM paper to see how results differ by model size and training method.

If safety and multilingual support concern you, read LiteLMGuard paper, the personal data substitution study paper and the tokenizer coverage study paper in sequence. Muse Glimmer paper and the call screening study paper show recent directions well. If you are interested in energy, see DVFSLM paper.