Paper review · November 17, 2026
Mobile AI Factory: automatically building specialized language models for your own smartphone
Mobile AI Factory: Building Specialized Language Models for Your Mobile Device
An agent searched over data, hyperparameters, model size and quantization, and used measurements from real phones to automatically build small language models suited to each task and device.
Problem
Small language models (SLMs) that run directly on a device are good for privacy and offline use. But experts have to repeatedly tune task-specific fine-tuning and pick model size, quantization and runtime for each device. Even the same configuration performs very differently from phone to phone, so performance is hard to predict from spec sheets alone.
Method
An orchestrator LLM (Claude Sonnet 5) picks one intervention per iteration: adjusting hyperparameters, collecting more real data, or generating synthetic data targeted at specific failure types with a teacher model (Qwen3.6-35B-A3B). If accuracy drops, it rolls back according to fixed rules. The 27 combinations of 9 base models (135M–4B) × 3 precisions (4/8/16-bit) are grouped into 5 buckets by size; the search starts from the smallest bucket and moves to the next when improvement stalls. The accuracy target is set to the teacher's 5-shot score, with a floor of 0.8. If the target is reached easily, it is raised. The phone profiler measures each phone only once and reuses the results for all tasks. It measures prefill and decode speed, memory and energy for each combination of model, quantization, runtime (llama.cpp/MNN) and backend (CPU/GPU). Accuracy is scored on the quantized artifact that will actually be deployed.
Results
- On held-out test sets, the on-device models built by the Factory beat 5-shot Claude Sonnet 5 and DeepSeek v4 Flash by 0.024–0.342 on 5 of 10 tasks and were comparable on 3. On biomedical entity extraction, a 517MB model scored 0.828, higher than Claude (0.652).
- On biomedical entity extraction, the bucket search came within 0.03 of the result from training all 27 combinations, using one-nineteenth of the GPU time.
- Given only full hardware and model specifications without measurements, Claude Opus 5.5 chose configurations that failed the app's requirements in 25% of easy and medium cases and 60% of hard cases.
- In phone measurements, llama.cpp's GPU prefill was 5.4× faster than CPU on the OnePlus 15 but 7.4× slower on the OnePlus 8 Pro. Decode speed for the same model differed by up to 42× across phones.
- With the same weights, MNN's default 4-bit conversion lowered a fine-tuned SmolLM2-360M's 151-way intent classification score from 0.767 (llama.cpp Q4_K_M) to 0.659.
Limitations
The evaluation is limited to 4 Android phones and 10 tasks. It covers only llama.cpp and MNN as runtimes and CPU and GPU as backends, not NPUs. At 4-bit, speed profiles were measured with Q4_0 and MNN's default conversion, while accuracy was scored with Q4_K_M and HQQ, so the formats used for measurement and scoring differ. GPU decode speed varies by 10–20% between sessions. Training runs on two L40S GPUs on a server.
Why it matters
The work points toward automating per-task, per-device SLM building in a mobile environment with heavy device fragmentation. In particular, two lessons apply directly to on-device AI deployment teams: you should not hand an LLM only spec sheets and let it decide, because real measurements are needed, and quantization accuracy depends on the runtime. The code is public.
Key terms
- SLM(Small Language Model)
- A small language model with hundreds of millions to a few billion parameters. It can run directly on devices such as smartphones.
- Quantization
- A technique that represents model weights at lower precision, such as 4-bit or 8-bit, to reduce model size and computation. The accuracy loss depends on the conversion method.
- Prefill / Decode
- Prefill is the stage that processes the input prompt all at once, and decode is the stage that generates tokens one at a time. The faster backend can differ from device to device.
- LoRA
- An efficient fine-tuning technique that keeps the original weights frozen and trains only small low-rank matrices.
Original
Authors: Evan Li, Krishna Deshpande, Wen Cheng, Shyamnath Gollakota · Posted 2026-11-10 · License CC BY-SA 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.