Paper review · November 8, 2026
Compression through the lens of context: evaluating domain adaptation, quantization and distillation for Arabic legal LLMs
Compression Through a Context Lens: Evaluating Adaptation, Quantization, and Distillation in Arabic Legal LLMs
Evaluating on a Saudi legal benchmark both with and without the legal passage provided, the authors found that 4-bit quantization cut memory by 64–71% while keeping accuracy changes within ±3.2 percentage points.
Problem
In legal settings where resources are limited and confidentiality matters, deploying a compressed LLM locally is a practical choice. However, the effects of compression have mostly been evaluated only on English and general-purpose benchmarks. In addition, evaluations that provide the legal passage may measure the model's ability to find answers in the passage rather than the knowledge it has learned. This issue has received little attention for Arabic legal tasks.
Method
The authors evaluated on 500 multiple-choice (MCQ) questions from ArabLegalEval (based on documents from the Saudi Ministry of Justice and others) and 200 open-ended NajizQA questions. Every stage compared two protocols: With-context, which provides the legal passage, and No-context, which omits it. The models were FANAR-9B, JAIS-8B and Qwen2.5-7B. Stage 1 compared continued pretraining (CPT), SFT and CPT→SFT using QLoRA. Stage 2 applied INT8, INT4 NF4 and INT4 FP4 post-training quantization to the CPT→SFT models. Stage 3 used the SFT models as teachers and applied sequence-level hard-label distillation to smaller student models, training them on answers the teacher generated in the No-context setting. Distillation across different model families was also included.
Results
- Removing the passage sharply reduced base model accuracy. FANAR dropped 27.0 percentage points, from 97.8% to 70.8%; Qwen2.5-7B dropped 21.4 points; and JAIS-8B dropped 7.6 points.
- On No-context MCQ, CPT→SFT raised FANAR from 70.8% to 79.2% (+8.4 points) and Qwen2.5-7B from 76.0% to 79.8%. JAIS-8B fell from 71.6% to 61.2% with CPT alone, and SFT alone (76.4%) worked best.
- 4-bit quantization cut memory by 64.3–71.1% relative to BF16. No-context accuracy changes ranged from -3.2 points (Qwen, FP4) to +1.6 points (JAIS, NF4). For all three models, NF4 was more accurate than FP4 (JAIS 77.4% vs 74.2%).
- With-context accuracy stayed near the ceiling at 97.4–99.2% after quantization, so it barely distinguished between quantization methods. The authors reported that No-context evaluation is more informative.
- Distillation results varied widely, and accuracy generally fell as the student got smaller. A Qwen2.5-7B student distilled from FANAR as the teacher reached 76.6% No-context, close to the teacher (77.2%).
Limitations
Evaluation used only one benchmark based on Saudi law, and only a fixed subset of 500 MCQ questions from it. Quantization started from the CPT→SFT models and distillation from the SFT models, so the authors state that the absolute numbers of the two compression methods cannot be compared directly. Only hard-label distillation was tested; logit-level distillation was left for future work. MCQ gains did not carry over to open-ended generation (JAIS had a ROUGE-L of nearly 0 after SFT). The authors only checked the proportion of answers whose wording appears verbatim in the passage (54%) and did not verify whether models actually rely on this overlap. The same training settings were applied to all models, so effects of model-specific optimization may be mixed in.
Why it matters
The paper shows with numbers that, when compressing and deploying LLMs locally in sensitive fields such as law, evaluations that provide the passage alone make it hard to detect whether compression has damaged knowledge. The finding that 4-bit NF4 quantization greatly reduced memory without major loss on Arabic legal MCQ is useful for deployment decisions in low-resource settings. It also offers a practical guideline: use No-context evaluation as well when evaluating compressed models.
Key terms
- With-context / No-context evaluation
- An open-book evaluation that gives the legal passage along with the question, and a closed-book evaluation that gives only the question and answer options without the passage. The gap between the two indicates whether a model finds answers in the passage or answers from the knowledge it has learned.
- NF4 quantization
- A 4-bit number format (NormalFloat) designed for weights that follow a normal distribution. In this paper's experiments it was consistently more accurate than FP4, which is also 4-bit.
- Sequence-level hard-label distillation
- A distillation (knowledge distillation) method in which the student learns directly from the answer text generated by the teacher model, rather than from the teacher's probability distribution. It works even when the tokenizers differ, so it allows transfer between different model families.
- CPT (continued pretraining)
- A method that continues pretraining an already trained model with a language modeling objective by feeding it additional domain text. It is distinct from SFT, which trains on instruction-formatted data.
Original
Authors: Md. Rafiul Biswas, George Mikros, Wajdi Zaghouani · Posted 2026-11-05 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.