Paper review · November 3, 2026
Beyond CLIP: Calibrated Few-shot Defect Detection with System One Decision Models
Beyond CLIP: Calibrated (Few-Shot) Defect Detection With System One Decision Models
By combining structured decision models (Jev, Clef-Flash) with two reference images per class, agent-refined class descriptions and Platt calibration, the authors achieved higher accuracy in manufacturing defect inspection than CLIP using the same reference images.
Problem
Training models for manufacturing visual inspection usually requires many images. CLIP-based methods reduced this to 50–100 images per class, but in high-yield or low-volume processes even 50 defect images are hard to collect. In addition, CLIP's defect probabilities cluster around 0.5, making them unreliable as a basis for pass/fail decisions.
Method
It is a four-stage framework. (a) The inspection task is defined by a multiple-choice question, plain descriptions of the normal and defect classes, and k reference images per class, and the defect probability is taken from the decision model's Choice output. (b) A Claude Code agent computes AUC on development images, examines misclassified images and revises the class descriptions. A new description is adopted only if AUC improves by 0.02 or more, with up to six candidates. (c) Platt scaling is fitted on a separate calibration set to calibrate the probabilities. (d) New images are inspected with the model, descriptions, reference images and calibration function all fixed. The model is not retrained.
Results
- With two reference images per class, test accuracy was 0.75–1.00 for Jev and 0.71–0.99 for Clef-Flash, higher than CLIP using the same reference images (0.67–0.84) (five manufacturing cases).
- Refining descriptions with the agent significantly improved accuracy in 6 of 8 model-case combinations.
- Platt calibration lowered the Brier score in 13 of 15 model-case configurations.
- The authors released reproduction code, the R package 'decisions' and a web app that can be used without code.
Limitations
Evaluation is limited to five cases used in prior work, and the test images were also already used in that work (the agent was blocked from accessing them). In the metal pan case, there were too few images to refine descriptions, and there is no independent calibration set. Calibration sets are small, at 17–25 images per class. Jev is an API-based proprietary model, so its internal workings are unknown.
Why it matters
For high-mix, low-volume production sites where defect samples are rare, it shows a practical procedure for starting inspection with only a few images and text descriptions, without retraining the model. Because it provides calibrated probabilities, it can also support pass, fail and re-inspection decisions.
Key terms
- System One model (structured decision model)
- A model that produces only structured judgments, such as a probability for each predefined option, instead of free text. Jev and Clef-Flash are examples.
- k-shot learning
- A classification approach in which only k labeled examples per class are provided together with the input at inference time. The model weights are not changed.
- Platt scaling
- A method that fits a logistic regression to the logits of model scores to calibrate them into probabilities that match actual frequencies of occurrence.
- Brier score
- The average of the squared differences between predicted probabilities and actual outcomes (0 or 1). Lower values mean the probabilities are more trustworthy.
Original
Authors: Fadel M. Megahed, Ying-Ju Chen · Posted 2026-10-30 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.