Measure AI Support-Ticket Classification Accuracy with a Confusion Matrix

01Why measure AI classification instead of trusting examples

An AI chat assistant can assign categories to support tickets, emails, comments, or short notes, but a few convincing examples do not show how accurate the classifier is overall. The useful question is not whether individual predictions look plausible. It is how often the AI agrees with a reference set that has already been labeled by a person.

A small hand-labeled test set gives you a fixed answer key. You can compare each AI prediction with that key, calculate accuracy, and inspect a confusion matrix to see which classes are being mixed up. Accuracy gives one overall number, while the confusion matrix shows the pattern behind the mistakes.

02Create a tiny synthetic labeled set

The following eight support tickets are synthetic examples created only for this tutorial. Use three labels: billing for payment or invoice issues, login for access problems, and bug for software behavior that appears broken. The hand_label column is the reference label assigned before asking the AI.

Because the dataset is small, every expected label can be reviewed manually. That makes it suitable for checking the evaluation workflow itself before applying the same method to a larger labeled sample.

03Ask the AI to return one controlled label per ticket

Classification prompts are easier to evaluate when the allowed labels and output format are fixed. Do not ask for free-form explanations unless you need them separately, because extra wording makes comparison harder.

1. Define the only allowed labels: billing, login, and bug.
2. Give short definitions for each label.
3. Require exactly one label per ticket.
4. Require the ticket ID to remain unchanged.
5. Tell the AI not to invent new categories when a ticket is ambiguous.

04Compare the AI output with the reference labels

Suppose the AI produces the following synthetic predictions. Six are correct. T7 is predicted as login even though the reference label is billing, and T8 is predicted as login even though the reference label is bug.

The overall accuracy is therefore 6 correct predictions out of 8 tickets. Calculated by hand, 6 ÷ 8 = 0.75, so the accuracy is 75%.

05Read the confusion matrix instead of accuracy alone

A confusion matrix counts reference labels by predicted labels. For this example, use rows for the hand labels and columns for the AI labels. The diagonal cells are correct predictions; off-diagonal cells are errors.

The matrix explains the two errors immediately. One billing ticket was classified as login, and one bug ticket was also classified as login. The AI did not incorrectly predict billing or bug for any login ticket in this small example. That does not prove login is generally the strongest class; it only describes these eight synthetic cases.

06Calculate accuracy and the matrix with Python

Save the comparison data in ticket_labels.csv with the columns id, hand_label, and ai_label. The script below validates the labels, counts correct predictions, builds the confusion matrix, and writes a report to outputs/classification_report.txt. It uses only the Python 3.12 standard library, does not change the input file, and stops if the output file already exists.

import csv
from collections import Counter
from pathlib import Path

INPUT_FILE = Path("ticket_labels.csv")
OUTPUT_DIR = Path("outputs")
OUTPUT_FILE = OUTPUT_DIR / "classification_report.txt"
LABELS = ["billing", "login", "bug"]


def main():
    if not INPUT_FILE.is_file():
        raise SystemExit(f"Input file not found: {INPUT_FILE}")

    if OUTPUT_FILE.exists():
        raise SystemExit(f"Output already exists: {OUTPUT_FILE}")

    rows = []
    with INPUT_FILE.open("r", encoding="utf-8", newline="") as file:
        reader = csv.DictReader(file)
        required = {"id", "hand_label", "ai_label"}
        if reader.fieldnames is None or not required.issubset(reader.fieldnames):
            raise SystemExit("ticket_labels.csv is missing required columns.")

        for row_number, row in enumerate(reader, start=2):
            hand = row["hand_label"].strip()
            predicted = row["ai_label"].strip()

            if hand not in LABELS:
                raise SystemExit(f"Invalid hand_label on row {row_number}: {hand}")
            if predicted not in LABELS:
                raise SystemExit(f"Invalid ai_label on row {row_number}: {predicted}")

            rows.append((row["id"].strip(), hand, predicted))

    if not rows:
        raise SystemExit("No labeled rows found.")

    correct = sum(1 for _, hand, predicted in rows if hand == predicted)
    accuracy = correct / len(rows)

    counts = Counter((hand, predicted) for _, hand, predicted in rows)

    report = []
    report.append(f"tickets={len(rows)}")
    report.append(f"correct={correct}")
    report.append(f"accuracy={accuracy:.3f}")
    report.append("")
    report.append("confusion_matrix")
    report.append("hand\\predicted\t" + "\t".join(LABELS))

    for hand in LABELS:
        values = [str(counts[(hand, predicted)]) for predicted in LABELS]
        report.append(hand + "\t" + "\t".join(values))

    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
    OUTPUT_FILE.write_text("\n".join(report) + "\n", encoding="utf-8")
    print(f"Wrote: {OUTPUT_FILE}")


if __name__ == "__main__":
    main()

07Review the mistakes before using the classifier

The two incorrect tickets deserve more attention than the 75% score alone. T7 mentions a card, so a weak prompt might associate it with account access even though the defined reference category is billing. T8 contains the phrase after login, but the actual reported problem is a blank dashboard, which the reference scheme treats as a bug. These cases show why category definitions should describe the underlying issue rather than rely on isolated keywords.

• Check every disagreement manually before changing the prompt or label definitions.
• Keep the same label definitions when comparing different prompts or AI runs.
• Do not mix training examples into the test set if you want the test to remain an independent evaluation.
• Use more than overall accuracy when classes are uneven or some mistakes matter more than others.
• Increase the hand-labeled sample before drawing conclusions about production performance.

A confusion matrix is descriptive evidence for the tested sample, not proof of future performance. A small balanced example is useful for learning the workflow, but a real evaluation should include enough hand-labeled tickets to represent actual wording, ambiguous cases, and class frequencies.

Execution and verification record

2026-09-21 · hand-checked example · Python 3.12

• All eight support tickets and labels in this article are synthetic examples.
• The example contains 8 tickets and 6 correct AI predictions.
• Accuracy was checked by hand: 6 ÷ 8 = 0.75, or 75%.
• The billing row of the confusion matrix was checked by hand: 2 predicted billing, 1 predicted login, 0 predicted bug.
• The login row was checked by hand: 0 predicted billing, 2 predicted login, 0 predicted bug.
• The bug row was checked by hand: 0 predicted billing, 1 predicted login, 2 predicted bug.
• The confusion matrix totals 8 cases, matching the number of example tickets.
• The shown Python logic uses only csv, collections, and pathlib from the Python standard library and protects an existing output file from overwrite.

Site-wide writing and verification principles

References

The explanations and examples were written for this site. See the official sources below for the related behavior and concepts.

• Python csv documentation ↗ (New tab)
• Python collections documentation ↗ (New tab)
• Python pathlib documentation ↗ (New tab)