Text Classification for Support Ticket Routing

Key facts, with sources

• Financial fraud losses reached about 54.2 billion US dollars in 2024 and are projected to exceed 68.7 billion by 2026, driving adoption of ML classification for fraud detection. (HyperVerge)
• A 2025 study in Frontiers in Artificial Intelligence showed Random Forest models combined with class imbalance mitigation achieved accuracy above 99.95% on credit card fraud detection while keeping false positives low enough for real-world operations. (Frontiers in Artificial Intelligence)
• A 2025 Scientific Reports telecom churn study reported a Random Forest classifier reaching 95.13% accuracy with an AUC of 0.89 after applying SMOTE and class weighting to a dataset where only 14.6% of customers churned. (Scientific Reports (Nature))
• A 2024 adaptive ensemble learning approach achieved 99.28% accuracy on a large telecom churn classification dataset, illustrating how ensembling and imbalance handling push benchmark performance. (arXiv)
• A 2024 study in Engineering, Technology and Applied Science Research comparing Random Forest, LightGBM, XGBoost, logistic regression, decision trees, and a custom ANN found an ensemble averaging method reached 0.79 accuracy and 0.72 recall on telecom churn test data, showing realistic performance on harder real-world datasets. (Engineering, Technology and Applied Science Research)

The baseline that's hard to beat

Classic machine learning is unreasonably effective on ticket routing because support tickets are full of strong lexical signals — product names, error strings, billing vocabulary. A TF-IDF vectorizer feeding a linear model trains in seconds on a laptop, serves in single-digit milliseconds, costs nothing per prediction, and is fully explainable: you can print exactly which words drove a routing decision.

Build this first even if you intend to use LLMs, because it becomes your benchmark. In my experience, teams that skip the baseline can't answer the only question that matters later: is the expensive model actually better than the cheap one on our data?

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(
        ngram_range=(1, 2),   # unigrams + bigrams: "refund request", "login error"
        min_df=3,
        sublinear_tf=True,
    )),
    ("clf", LogisticRegression(max_iter=1000, class_weight="balanced")),
])

X_train, X_test, y_train, y_test = train_test_split(
    tickets, labels, test_size=0.2, stratify=labels, random_state=42
)
pipeline.fit(X_train, y_train)
print(classification_report(y_test, pipeline.predict(X_test)))

The confidence threshold is the real product decision

No router should route everything. predict_proba gives you a confidence per ticket; the operating policy is: auto-route above the threshold, human queue below it. That threshold trades automation rate against misroute rate, and it's a business decision — a misrouted enterprise ticket costs more than a triager's thirty seconds.

Set it empirically: on the test set, plot automation rate against accuracy at each threshold and let the support lead pick the point. Publishing this as an explicit dial ('we auto-route 78% of tickets at 96% accuracy') builds more organizational trust than any accuracy headline, because it's honest about what happens to the hard 22%.

import numpy as np

def route(ticket_text: str, threshold: float = 0.80) -> dict:
    proba = pipeline.predict_proba([ticket_text])[0]
    best = int(np.argmax(proba))
    if proba[best] >= threshold:
        return {"queue": pipeline.classes_[best], "auto": True,
                "confidence": round(float(proba[best]), 3)}
    return {"queue": "triage", "auto": False,
            "confidence": round(float(proba[best]), 3)}

Where an LLM actually earns its cost

Three places, in priority order. Cold start: with under a few hundred labeled tickets, a zero-shot LLM prompt with your category definitions outperforms anything you can train — use it on day one while the baseline's training data accumulates from its decisions. Low-confidence fallback: instead of sending every below-threshold ticket to humans, let the LLM take a second pass with category descriptions in the prompt; it rescues a meaningful share. Rare categories: classes with a handful of examples are where linear models fail and few-shot prompting shines.

What I don't recommend: LLM-routing every ticket. At real support volume you'd be paying per-token prices and hundreds of milliseconds of latency for tickets the baseline routes correctly for free.

The feedback loop that keeps it accurate

Routing accuracy decays: products ship, new failure modes appear, category definitions drift. The loop that fixes this is already in your data — when a support agent re-assigns a misrouted ticket, that reassignment is a fresh training label. Capture it, and retrain on a schedule (weekly or monthly) with evaluation against a held-out set before promoting the new model.

Monitor two numbers in production: the reassignment rate (your live error rate, no labeling required) and the class distribution of incoming tickets (a sudden shift means something changed upstream — a new product launch, an incident — and the model is now partially blind). Alert on both; they're leading indicators of decay.

When to hire senior help

Senior help matters most when the classifier's errors carry asymmetric costs, such as fraud, credit, or medical triage decisions, because threshold tuning, leakage detection, and imbalance handling are exactly where self-taught implementations quietly fail. A short expert review of the evaluation setup before launch is much cheaper than discovering leakage or an untuned threshold after customer-facing decisions have been made. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships ML — Classification Models projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Reporting plain accuracy on imbalanced data, where predicting the majority class every time already scores 99%, instead of precision, recall, and AUC-PR
• ✕Leaking the label into features, such as including fields that are only populated after the outcome occurs, which inflates offline metrics and collapses in production
• ✕Leaving the decision threshold at the 0.5 default rather than tuning it to the asymmetric business cost of false positives versus false negatives
• ✕Evaluating with random train-test splits instead of time-based splits, which hides temporal drift and overstates real-world performance

Frequently asked questions

Sources

• HyperVerge
• Frontiers in Artificial Intelligence
• Scientific Reports (Nature)
• arXiv
• Engineering, Technology and Applied Science Research