Vision models at the edge: the latency–accuracy trade-off

There is a wide distance between a detection model scoring well on a test set and running for hours without failure on a production line. Much of that distance comes from hardware: a model that runs on a server GPU in the lab has to run in the field on a board the size of a palm, within a fixed latency budget.

This post is about how we decide which accuracy loss is acceptable during that transfer.

Budget first, model second

The common order is: train the best model, then try to make it fit. That order is wrong. We write the latency budget first.

The budget comes from the line itself. How many units pass per second, at which point the camera triggers, by which moment the decision must be made. One number falls out of this: milliseconds allowed per frame. That number is known before the model is chosen and is not negotiable — because when it is exceeded the model does not give a wrong answer, it gives no answer at all.

Accuracy is not a single number

The phrase "2% accuracy loss" means nothing on its own. Which accuracy?

In production line inspection the two error types do not cost the same. A missed defect means a faulty unit reaching the field. A false alarm means the line stopping unnecessarily. For most clients the first is far more expensive than the second, but for some it is the reverse — where the cost of stopping the line is very high, false alarms become unacceptable.

So we measure not at a single threshold but across the threshold curve, and choose the operating point together with the client. What we examine after distillation and quantisation is not average accuracy but the missed-defect rate at the chosen operating point.

Distillation

Transferring a large model's knowledge to a small one performs markedly better than training the small one from scratch. A few points that work in practice:

The student's architecture is chosen from operations the target hardware supports. A model that is small on paper but uses a layer the hardware cannot accelerate can run slower than a larger but compatible one.

Unlabelled field imagery is used as distillation data. The teacher model labels it and the student trains on those labels. This lowers labelling cost and trains the student on exactly the distribution it will encounter.

Quantisation

Moving from FP32 to INT8 usually gives the largest gain. There are two things we watch.

Calibration data is drawn from the real field distribution. Calibration data taken from the test set gives an optimistic result, and that optimism disappears in the field.

Sensitivity is measured layer by layer. Some layers are far more sensitive to quantisation than others; leaving those in FP16 can give a better latency–accuracy point than pushing the whole model to INT8.

Measurement happens on real hardware

We write this point separately because it is the one most often skipped. Latency measured on a development machine does not predict latency in the field. We measure on the target board, at the target temperature, under sustained load.

Thermal throttling is decisive here. A system that meets its target for the first five minutes can exceed the budget once the board heats up. Acceptance testing is therefore done not on a short sample but on a run lasting hours.

Conclusion

Moving to the edge is not a compression problem but a budget problem. When the latency budget is written up front and accuracy is measured with the right metric, acceptable loss becomes a negotiable number. When the budget is written afterwards, the model usually gets rebuilt.