A fine-tuned model that replaced an expensive frontier API on the line
A frontier vision API classified fabric defects accurately and cost more than the QA salaries it supplemented. We fine-tuned a small open model on the client's own inspection archive and put it on the line.
- Organization
- Textile manufacturer
- Industrial textiles · 2 production lines · high-volume in-line inspection
The shape of the system
What this demonstrates
Inference now runs on a single on-premise GPU box; no image leaves the factory network
The model is trained with class weighting that reflects real scrap cost, not raw class frequency, so rare-but-expensive defects get proportional attention
Evaluated head-to-head against the frontier API on the same held-out set before switching, on cost-weighted recall, latency and cost per thousand inspections
A shadow-deployment path runs any new model alongside the incumbent on live traffic before promotion, with one-command rollback
01/The problem
The client had proven the concept with a frontier vision API: photograph the fabric, classify the defect, route the roll. It worked well enough to be trusted, and at the plant's inspection volume the monthly bill had grown larger than the QA headcount it was meant to supplement.
Latency was the harder constraint. Round-tripping to an external API took over two seconds per inspection, which forced a buffering step in the line and capped throughput below what the machinery could otherwise sustain.
There was also a quality problem hiding underneath the cost problem. The general model was strong on common defects and noticeably weaker on rare classes that carried the highest scrap cost — precisely the ones worth catching.
02/Approach
We began by trying not to train anything. Better prompting and a retrieval-augmented reference set of example images got the frontier model partway on the rare classes, but not far enough, and did nothing about cost or latency. That took two weeks and was worth every day — it is the check that tells you fine-tuning is genuinely the right tool rather than the interesting one.
The client's archive was the deciding asset: hundreds of thousands of inspection images accumulated over years, each already labelled by a QA operator as part of normal work. Very few clients have this, and it is usually the difference between a fine-tuning project that works and one that spends its budget on labelling.
Label quality got a hard look before training. A meaningful share of archive labels were wrong or inconsistent, concentrated — predictably — in exactly the rare classes we cared most about. Two QA leads re-adjudicated a stratified sample, and cleaning those labels improved final recall more than any architecture change we tried afterwards.
03/What we built
- 01
A cleaned, deduplicated and stratified dataset from the image archive, with a held-out test set drawn from later production runs rather than a random split, so the evaluation could not flatter itself by seeing the same rolls.
- 02
A fine-tuned open vision model, trained with class weighting that reflects real scrap cost rather than class frequency — the rare defects are rare precisely because they are expensive, so treating them as equal to common ones optimises the wrong thing.
- 03
Head-to-head evaluation against the frontier API on the same held-out set, measured on cost-weighted recall, latency and cost per thousand inspections, so the decision to switch rested on numbers rather than preference.
- 04
Quantised export to TensorRT running on a single on-premise GPU box, cutting per-inspection latency dramatically and removing the line's buffering step altogether.
- 05
A retraining pipeline triggered by both schedule and drift detection, fed by ongoing QA operator corrections.
- 06
A shadow-deployment path where a new model runs alongside the incumbent on live traffic for a period before promotion, with one-command rollback.
04/Outcome
Moving inference on-premise removed the round-trip to an external API and its cost structure entirely, and let the line run at full mechanical speed for the first time since inspection was automated.
Recall on the highest-cost defect classes improved over the frontier model — the outcome that mattered most and, going in, the one we were least confident of achieving.
The client owns the weights, the dataset and the training pipeline outright. There is no ongoing licence to us, which was an explicit condition of the engagement and one we think more studios should offer.
05/What went wrong
Our first fine-tune scored better on aggregate accuracy and worse on the rare classes — we had optimised the metric that was easy to compute rather than the one the client cared about. Rebuilding the loss around cost-weighted recall fixed it, but we had burned real time getting a genuinely worse model. Agreeing the evaluation metric before training now happens in week one, in writing.
Every engagement has one of these. A case study without it is a brochure.
06/Detail
Stack and services drawn on
Built with
- PyTorch
- LoRA
- TensorRT
- Weights & Biases
- MLflow
- Docker
Recognise this problem?
We start every engagement by measuring what the manual process actually costs. That first week tells you whether the rest is worth doing.