Test an AI-written regular expression with positive and negative examples

01Write the requirement before testing the regex

Suppose an internal work ID must have the form WK-YYYY-NNNN. The prefix must be uppercase WK, the year must be between 2000 and 2099, the separators must be hyphens, and the final part must contain exactly four ASCII digits.

Imagine an AI assistant proposes the pattern WK-\d{4}-\d{4}. It looks plausible, but it only requires four digits for the year. It does not enforce the stated 2000-2099 range. The safest next step is not to use it immediately, but to test it against examples with known answers.

02Create a synthetic set of test cases

The following dataset is synthetic and was written specifically for this article. Save it as regex_cases.csv. It contains three strings that should match and seven that should not.

case_id,value,should_match
C001,WK-2026-0001,yes
C002,WK-2000-0042,yes
C003,WK-2099-9999,yes
C004,WK-1999-0001,no
C005,wk-2026-0001,no
C006,WK-2026-001,no
C007,WK-2026-00001,no
C008,WK-2026_0001,no
C009," WK-2026-0001",no
C010,WK-20A6-0001,no

The cases cover the allowed year boundaries, an out-of-range year, lowercase prefix, short and long sequence fields, wrong separator, leading whitespace, and a non-digit character in the year. These are small enough to classify manually before running any regex.

03Predict what the AI pattern will get wrong

The AI pattern WK-\d{4}-\d{4} should accept C001, C002, and C003. It should reject C005 through C010 for structural reasons. However, it also accepts C004 because 1999 still consists of four digits.

The revised pattern is WK-20[0-9]{2}-[0-9]{4}. The 20 prefix restricts the year to 2000 through 2099, and [0-9] makes the intended digit set explicit. The script uses fullmatch so the entire input string must satisfy the pattern.

04Compare expected and actual matches automatically

Save the following script as ai_regex_check.py. It tests both the AI proposal and the revised pattern, writes every case to a CSV report, and refuses to reuse an existing output folder.

import csv
import re
from pathlib import Path

SOURCE = Path("regex_cases.csv")
OUTPUT_DIR = Path("outputs") / "regex_check_result"
REPORT = OUTPUT_DIR / "regex_check.csv"

AI_PATTERN = re.compile(r"WK-\d{4}-\d{4}")
REVISED_PATTERN = re.compile(r"WK-20[0-9]{2}-[0-9]{4}")


def parse_expected(value: str) -> bool:
    normalized = value.strip().lower()
    if normalized == "yes":
        return True
    if normalized == "no":
        return False
    raise ValueError(f"Expected yes or no, got: {value!r}")


def label(value: bool) -> str:
    return "MATCH" if value else "NO MATCH"


def main() -> None:
    if not SOURCE.is_file():
        raise FileNotFoundError(f"Source CSV not found: {SOURCE}")
    if OUTPUT_DIR.exists():
        raise FileExistsError(f"Output folder already exists: {OUTPUT_DIR}")

    results = []
    ai_failures = 0
    revised_failures = 0

    with SOURCE.open("r", encoding="utf-8-sig", newline="") as stream:
        reader = csv.DictReader(stream)
        required = {"case_id", "value", "should_match"}
        if reader.fieldnames is None or not required.issubset(reader.fieldnames):
            raise ValueError("CSV is missing a required column.")

        for row_number, row in enumerate(reader, start=2):
            expected = parse_expected(row["should_match"])
            text = row["value"]
            ai_actual = AI_PATTERN.fullmatch(text) is not None
            revised_actual = REVISED_PATTERN.fullmatch(text) is not None
            ai_pass = ai_actual == expected
            revised_pass = revised_actual == expected

            if not ai_pass:
                ai_failures += 1
            if not revised_pass:
                revised_failures += 1

            results.append({
                "case_id": row["case_id"],
                "value": text,
                "expected": label(expected),
                "ai_actual": label(ai_actual),
                "ai_test": "PASS" if ai_pass else "FAIL",
                "revised_actual": label(revised_actual),
                "revised_test": "PASS" if revised_pass else "FAIL",
            })

    OUTPUT_DIR.parent.mkdir(parents=True, exist_ok=True)
    OUTPUT_DIR.mkdir()
    fields = [
        "case_id", "value", "expected", "ai_actual", "ai_test",
        "revised_actual", "revised_test"
    ]
    with REPORT.open("x", encoding="utf-8", newline="") as stream:
        writer = csv.DictWriter(stream, fieldnames=fields)
        writer.writeheader()
        writer.writerows(results)

    print(f"Cases: {len(results)}.")
    print(f"AI pattern failures: {ai_failures}.")
    print(f"Revised pattern failures: {revised_failures}.")
    print(f"Report: {REPORT.as_posix()}")

    if revised_failures:
        raise RuntimeError("Revised pattern still fails one or more tests.")


if __name__ == "__main__":
    main()

python ai_regex_check.py

05Check the expected report

There are 10 test cases. The AI pattern should fail exactly one case, C004. The revised pattern should pass all 10 cases.

The expected console output below was derived manually from the test table and script. It is not a captured execution log.

Cases: 10.
AI pattern failures: 1.
Revised pattern failures: 0.
Report: outputs/regex_check_result/regex_check.csv

06Add boundary cases and recognize common mistakes

• Include valid minimum and maximum boundaries, such as 2000 and 2099 in this example.
• Include at least one value just outside a boundary, such as 1999.
• Test incorrect case, separators, field lengths, whitespace, and invalid characters.
• Use fullmatch when the requirement says the entire string must conform to the format.
• Run the script again without changing OUTPUT_DIR. It should stop with FileExistsError instead of overwriting the previous report.

07Understand what regex testing does not prove

Passing these 10 tests shows only that the revised pattern behaves correctly for these 10 synthetic examples. It does not mathematically prove that every possible input is handled correctly. Add cases whenever a new boundary or failure mode is discovered.

A regex also validates syntax, not business truth. WK-2026-1234 can match perfectly even if that ID does not exist in your database. Existence, uniqueness, authorization, and relationships with other fields require separate checks.

This small example does not evaluate performance on very long or adversarial input. More complicated AI-generated patterns should be reviewed for both correctness and runtime behavior before being placed in services that process untrusted text.

Execution and verification record

2026-09-20 · hand-checked example · target: Python 3.12 · standard library: csv, pathlib, re · no execution

• Manually classified 3 synthetic values as expected matches and 7 as expected non-matches.
• Manually checked that the AI pattern accepts C004 because 1999 satisfies the four-digit year portion.
• Manually determined that the AI pattern has 1 failing test out of 10.
• Manually checked that the revised 20[0-9]{2} year portion accepts 2000 through 2099 and rejects the listed 1999 example.
• Inspected the script for fullmatch use, expected-versus-actual comparison, output collision protection, and report generation.
• Derived the expected failure counts and console output by hand.

Site-wide writing and verification principles

References

The explanations and examples were written for this site. See the official sources below for the related behavior and concepts.

• Official Python 3.12 documentation · re ↗ (New tab)
• Official Python 3.12 documentation · re.fullmatch ↗ (New tab)
• Official Python 3.12 documentation · csv ↗ (New tab)