WorkEmail classification service
Email classification service
Nine models compared on 92,700 labelled emails with matter-level splits.
0.83
macro F1, held-out, on 4-class triage
The numbers
9
models compared
92,700
labelled emails
4
triage classes
Notes
This service sorts incoming email into four triage classes. I compared nine models on 92,700 labelled emails, and the best reached 0.83 macro F1 on held-out data.
Matter-level splits
The data was split by matter, not by email, so every email from one matter sits on the same side of the split. The held-out score therefore comes from matters the model never saw in training.
Imbalance and model choice
The data has severe class imbalance, so the classifier is two-stage. The nine models include a fine-tuned transformer.
Calibration and a second opinion
Predicted probabilities are calibrated. When a prediction is low-confidence, an LLM verification step (Gemma on vLLM) takes a second look, and a rule-based fallback is there if that step cannot answer.
Extraction
The same service uses an LLM to extract 24 case fields, each with a reference back to its source. OCR handles PDF, image and Office files. It ships as a containerised FastAPI service.