Why My Car Damage Detection Model Only Hits 72% Accuracy

And what the confusion matrix taught me that the accuracy score didn't

By Imran Rafique · June 2026 · 10 min read

Most project write-ups lead with the win. I want to lead with the number that I almost didn't publish: my car damage classifier's best validation accuracy is 72.2%, on a genuinely hard, three-way distinction. That's not a number I'd put on a billboard — but it's a number I can fully explain, and that turned out to matter more than the number itself.

The task: given a photo of a damaged car, classify the severity of the damage as minor, moderate, or severe. It's a real use case — insurance claim triage, marketplace listings, repair-shop intake — and it's the kind of problem that looks deceptively simple until you actually try to draw a hard line between "minor" and "moderate" damage on a dented door panel.

This post is about what the accuracy number hides, what the confusion matrix reveals instead, and why the fix that actually worked wasn't a bigger model — it was a more constrained one.

The Setup

I built this on EfficientNet-B0, pretrained on ImageNet, with a custom classification head: dropout, a 256-unit dense layer with ReLU, more dropout, then a final linear layer over three classes. Training followed a two-phase strategy — first freezing the entire EfficientNet feature extractor and training only the new head for 30 epochs, then unfreezing just the last three feature blocks and fine-tuning at a lower learning rate for up to 20 more epochs, with early stopping (patience of 10) watching validation accuracy.

The dataset was drawn from 1,613 training images and 478 validation images, using only three of the four available classes — minor, moderate, and severe damage — excluding the "no damage" class from this iteration, since grading damage severity is the harder and more interesting problem. To handle class imbalance, training used a weighted random sampler that oversamples less frequent classes so every batch sees a roughly even mix, plus a training-time augmentation pipeline (random crops, flips, rotation, color jitter, and small affine shifts).

The Honest Number

Here's a training curve from one of my earlier runs on this same architecture, which I'm including precisely because it failed instructively. Training accuracy climbs steadily to around 78%. Validation accuracy does something very different — it plateaus almost immediately around 35%, and barely moves for the rest of training, even past the point where fine-tuning begins.

Figure 1 — A training run on an earlier configuration, showing the classic overfitting signature: the gap between train and validation curves widens steadily.

This is a textbook overfitting curve, and I want to be upfront about it rather than crop it out of the post. A gap this wide, this early, with validation loss climbing rather than falling after fine-tuning starts, means the model is memorizing the training set faster than it's learning anything that generalizes — unsurprising with roughly 1,400 training images and a network with millions of parameters.

The fixes that closed most of that gap in later runs were narrower, not bigger: freeze everything except the last few feature blocks during fine-tuning instead of unfreezing the whole network, add a weighted sampler so the model can't shortcut its way to a good loss by favoring the majority class, add early stopping so training halts at the best validation checkpoint instead of running to a fixed epoch count, and add real augmentation so the model can't simply memorize a few hundred exact images per class. That combination is what produced the 72.2% best validation accuracy and the confusion matrix below — a meaningfully smaller train/validation gap than the run shown in Figure 1.

What the Confusion Matrix Told Me That Accuracy Didn't

A single accuracy number flattens everything into one digit, and in this case that digit undersells what the model actually learned. The confusion matrix tells a much more specific, and more useful, story.

Figure 2 — Confusion matrix on the validation set (Minor, Moderate, Severe damage)

Look at where the errors actually land. Minor damage gets confused with Moderate damage 17 times, and almost never with Severe (just 2 times). Severe gets confused with Moderate 11 times, and almost never with Minor (just 1 time). The model's mistakes are concentrated almost entirely between neighboring classes — it essentially never confuses the two extremes.

That pattern matters, because it isn't random noise. It's the same kind of disagreement a human would have looking at the same photos. A dented door panel that's borderline between "minor" and "moderate" is a genuinely ambiguous case, not a model failure — and the sample images make that obvious at a glance.

Figure 3 — Sample images from each of the original four classes. Several "Minor" and "Moderate" examples are genuinely close calls even to a human eye.

Why This Matters More Than the Accuracy Score

Treating damage severity as unrelated categories — the way a standard softmax classifier does — throws away the one piece of structure that actually makes this problem tractable: the classes are ordered. "Severe" is not just "different" from "Minor," it's further along the same scale. A categorical cross-entropy loss penalizes a Minor-predicted-as-Severe mistake exactly as much as a Minor-predicted-as-Moderate mistake, even though the second is a far more reasonable error to make. The rebuilt model almost never makes the first kind of mistake — which tells me it implicitly learned the ordering anyway, just without being told it exists or being rewarded for respecting it.

The next highest-leverage change I haven't made yet wouldn't be a bigger backbone or more aggressive augmentation — it would be reframing this as an ordinal regression problem outright, where the loss function itself penalizes distant misclassifications more than adjacent ones. That one framing change would directly target the exact failure pattern still visible in the confusion matrix above.

Other Changes I'd Still Make

1. More data, especially for the smaller classes. At roughly 450–470 images per damage class, the weighted sampler helps balance what the model sees per batch, but it can't manufacture visual variety that simply isn't in the dataset.
2. Push the ordinal-regression framing through, rather than relying on the model to discover the ordering implicitly through cross-entropy alone.
3. A confidence threshold with human review for borderline predictions, rather than forcing a hard decision on every image — for an insurance or claims use case, an honest "uncertain, escalate to a human" is far more valuable than a confident wrong answer.

The Deployment Side

Separately from the modeling story, the deployment was its own useful lesson. The currently live model is packaged in a Docker container and deployed on AWS Lambda behind API Gateway, rather than a continuously running server — a sensible choice for a low-traffic inference endpoint where paying for an always-on EC2 instance doesn't make sense.

The interesting constraint was Lambda's request/response model: API Gateway has trouble with binary payloads, so the frontend sends the uploaded image as a base64-encoded string inside a JSON body instead of a raw file upload. The Lambda function loads the TensorFlow model lazily on the first request rather than at container startup, which keeps cold starts from blocking on model loading — though the very first request after a period of inactivity still takes 30–45 seconds while a fresh container spins up and loads the model weights into memory. Once a container is warm, subsequent requests return in 2–3 seconds.

What I'd Tell Someone Starting a Similar Project

1. Look at the confusion matrix before you judge a model by its accuracy score. A 72% accuracy that's wrong in a structured, explainable way is more useful — and more fixable — than an 85% accuracy you can't explain.
2. If your classes have a natural order, treat them as ordered. Standard categorical classification throws that signal away for free, and the confusion matrix will quietly show you the cost.
3. A training/validation gap that widens steadily across epochs is a dataset-size and freezing problem before it's an architecture problem. Partial unfreezing, early stopping, and a balanced sampler closed most of that gap here without changing the backbone at all.
4. Ambiguous ground truth is real. Some of these images would get different labels from different human annotators — a model can't beat a ceiling that the labels themselves don't clear.
5. Be honest about what's actually deployed versus what's just been trained. It's tempting to write up the better run as if it were live — the more useful post is the one that says which is which.

Try It Yourself

The live demo, full training code, the confusion matrix and training curves shown above, and the AWS Lambda deployment guide are all on GitHub — including the parts that didn't go perfectly, because I think that's the more useful version of this project to share.

If you need a computer vision system, image classifier, or AWS Lambda deployment built for your business — see my Computer Vision & Cloud services →

🔴 Try the Live Demo  |  View Source on GitHub