← Key papers

Paper review · November 17, 2026

Mobile AI Factory: automatically building specialized language models for your own smartphone

Mobile AI Factory: Building Specialized Language Models for Your Mobile Device

Efficiency & infraAgentsLanguage models Intermediate Upvotes 17 cs.CL University of Washington

An agent searched over data, hyperparameters, model size and quantization, and used measurements from real phones to automatically build small language models suited to each task and device.

Problem

Small language models (SLMs) that run directly on a device are good for privacy and offline use. But experts have to repeatedly tune task-specific fine-tuning and pick model size, quantization and runtime for each device. Even the same configuration performs very differently from phone to phone, so performance is hard to predict from spec sheets alone.

Method

An orchestrator LLM (Claude Sonnet 5) picks one intervention per iteration: adjusting hyperparameters, collecting more real data, or generating synthetic data targeted at specific failure types with a teacher model (Qwen3.6-35B-A3B). If accuracy drops, it rolls back according to fixed rules. The 27 combinations of 9 base models (135M–4B) × 3 precisions (4/8/16-bit) are grouped into 5 buckets by size; the search starts from the smallest bucket and moves to the next when improvement stalls. The accuracy target is set to the teacher's 5-shot score, with a floor of 0.8. If the target is reached easily, it is raised. The phone profiler measures each phone only once and reuses the results for all tasks. It measures prefill and decode speed, memory and energy for each combination of model, quantization, runtime (llama.cpp/MNN) and backend (CPU/GPU). Accuracy is scored on the quantized artifact that will actually be deployed.

Results

Limitations

The evaluation is limited to 4 Android phones and 10 tasks. It covers only llama.cpp and MNN as runtimes and CPU and GPU as backends, not NPUs. At 4-bit, speed profiles were measured with Q4_0 and MNN's default conversion, while accuracy was scored with Q4_K_M and HQQ, so the formats used for measurement and scoring differ. GPU decode speed varies by 10–20% between sessions. Training runs on two L40S GPUs on a server.

Why it matters

The work points toward automating per-task, per-device SLM building in a mobile environment with heavy device fragmentation. In particular, two lessons apply directly to on-device AI deployment teams: you should not hand an LLM only spec sheets and let it decide, because real measurements are needed, and quantization accuracy depends on the runtime. The code is public.

Key terms

SLM(Small Language Model)
A small language model with hundreds of millions to a few billion parameters. It can run directly on devices such as smartphones.
Quantization
A technique that represents model weights at lower precision, such as 4-bit or 8-bit, to reduce model size and computation. The accuracy loss depends on the conversion method.
Prefill / Decode
Prefill is the stage that processes the input prompt all at once, and decode is the stage that generates tokens one at a time. The faster backend can differ from device to device.
LoRA
An efficient fine-tuning technique that keeps the original weights frozen and trains only small low-rank matrices.

Original

Authors: Evan Li, Krishna Deshpande, Wen Cheng, Shyamnath Gollakota · Posted 2026-11-10 · License CC BY-SA 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.