Test an AI-written regular expression with positive and negative examples

01Write the requirement before testing the regex

Suppose an internal work ID must have the form WK-YYYY-NNNN. The prefix must be uppercase WK, the year must be between 2000 and 2099, the separators must be hyphens, and the final part must contain exactly four ASCII digits.

Imagine an AI assistant proposes the pattern WK-\d{4}-\d{4}. It looks plausible, but it only requires four digits for the year. It does not enforce the stated 2000-2099 range. The safest next step is not to use it immediately, but to test it against examples with known answers.

02Create a synthetic set of test cases

The following dataset is synthetic and was written specifically for this article. Save it as regex_cases.csv. It contains three strings that should match and seven that should not.

case_id,value,should_match
C001,WK-2026-0001,yes
C002,WK-2000-0042,yes
C003,WK-2099-9999,yes
C004,WK-1999-0001,no
C005,wk-2026-0001,no
C006,WK-2026-001,no
C007,WK-2026-00001,no
C008,WK-2026_0001,no
C009," WK-2026-0001",no
C010,WK-20A6-0001,no

The cases cover the allowed year boundaries, an out-of-range year, lowercase prefix, short and long sequence fields, wrong separator, leading whitespace, and a non-digit character in the year. These are small enough to classify manually before running any regex.

03Predict what the AI pattern will get wrong

The AI pattern WK-\d{4}-\d{4} should accept C001, C002, and C003. It should reject C005 through C010 for structural reasons. However, it also accepts C004 because 1999 still consists of four digits.

The revised pattern is WK-20[0-9]{2}-[0-9]{4}. The 20 prefix restricts the year to 2000 through 2099, and [0-9] makes the intended digit set explicit. The script uses fullmatch so the entire input string must satisfy the pattern.

04Compare expected and actual matches automatically

Save the following script as ai_regex_check.py. It tests both the AI proposal and the revised pattern, writes every case to a CSV report, and refuses to reuse an existing output folder.

import csv
import re
from pathlib import Path

SOURCE = Path("regex_cases.csv")
OUTPUT_DIR = Path("outputs") / "regex_check_result"
REPORT = OUTPUT_DIR / "regex_check.csv"

AI_PATTERN = re.compile(r"WK-\d{4}-\d{4}")
REVISED_PATTERN = re.compile(r"WK-20[0-9]{2}-[0-9]{4}")


def parse_expected(value: str) -> bool:
    normalized = value.strip().lower()
    if normalized == "yes":
        return True
    if normalized == "no":
        return False
    raise ValueError(f"Expected yes or no, got: {value!r}")


def label(value: bool) -> str:
    return "MATCH" if value else "NO MATCH"


def main() -> None:
    if not SOURCE.is_file():
        raise FileNotFoundError(f"Source CSV not found: {SOURCE}")
    if OUTPUT_DIR.exists():
        raise FileExistsError(f"Output folder already exists: {OUTPUT_DIR}")

    results = []
    ai_failures = 0
    revised_failures = 0

    with SOURCE.open("r", encoding="utf-8-sig", newline="") as stream:
        reader = csv.DictReader(stream)
        required = {"case_id", "value", "should_match"}
        if reader.fieldnames is None or not required.issubset(reader.fieldnames):
            raise ValueError("CSV is missing a required column.")

        for row_number, row in enumerate(reader, start=2):
            expected = parse_expected(row["should_match"])
            text = row["value"]
            ai_actual = AI_PATTERN.fullmatch(text) is not None
            revised_actual = REVISED_PATTERN.fullmatch(text) is not None
            ai_pass = ai_actual == expected
            revised_pass = revised_actual == expected

            if not ai_pass:
                ai_failures += 1
            if not revised_pass:
                revised_failures += 1

            results.append({
                "case_id": row["case_id"],
                "value": text,
                "expected": label(expected),
                "ai_actual": label(ai_actual),
                "ai_test": "PASS" if ai_pass else "FAIL",
                "revised_actual": label(revised_actual),
                "revised_test": "PASS" if revised_pass else "FAIL",
            })

    OUTPUT_DIR.parent.mkdir(parents=True, exist_ok=True)
    OUTPUT_DIR.mkdir()
    fields = [
        "case_id", "value", "expected", "ai_actual", "ai_test",
        "revised_actual", "revised_test"
    ]
    with REPORT.open("x", encoding="utf-8", newline="") as stream:
        writer = csv.DictWriter(stream, fieldnames=fields)
        writer.writeheader()
        writer.writerows(results)

    print(f"Cases: {len(results)}.")
    print(f"AI pattern failures: {ai_failures}.")
    print(f"Revised pattern failures: {revised_failures}.")
    print(f"Report: {REPORT.as_posix()}")

    if revised_failures:
        raise RuntimeError("Revised pattern still fails one or more tests.")


if __name__ == "__main__":
    main()

python ai_regex_check.py

05Check the expected report

There are 10 test cases. The AI pattern should fail exactly one case, C004. The revised pattern should pass all 10 cases.

The expected console output below was derived manually from the test table and script. It is not a captured execution log.

Cases: 10.
AI pattern failures: 1.
Revised pattern failures: 0.
Report: outputs/regex_check_result/regex_check.csv

06Add boundary cases and recognize common mistakes

• Include valid minimum and maximum boundaries, such as 2000 and 2099 in this example.
• Include at least one value just outside a boundary, such as 1999.
• Test incorrect case, separators, field lengths, whitespace, and invalid characters.
• Use fullmatch when the requirement says the entire string must conform to the format.
• Run the script again without changing OUTPUT_DIR. It should stop with FileExistsError instead of overwriting the previous report.

07Understand what regex testing does not prove

Passing these 10 tests shows only that the revised pattern behaves correctly for these 10 synthetic examples. It does not mathematically prove that every possible input is handled correctly. Add cases whenever a new boundary or failure mode is discovered.

A regex also validates syntax, not business truth. WK-2026-1234 can match perfectly even if that ID does not exist in your database. Existence, uniqueness, authorization, and relationships with other fields require separate checks.

This small example does not evaluate performance on very long or adversarial input. More complicated AI-generated patterns should be reviewed for both correctness and runtime behavior before being placed in services that process untrusted text.

Execution and verification record

ChatGPT Python tool · Python 3.13.5 · standard library csv/pathlib/re

• The main run printed exactly: 'Cases: 10.', 'AI pattern failures: 1.', 'Revised pattern failures: 0.', and 'Report: outputs/regex_check_result/regex_check.csv'.
• regex_check.csv contained exactly 10 data rows plus its header.
• The AI pattern WK-\d{4}-\d{4} failed exactly C004, accepting WK-1999-0001 even though the expected result was NO MATCH.
• All other AI-pattern cases matched their expected classifications.
• The revised pattern WK-20[0-9]{2}-[0-9]{4} passed all 10 synthetic cases.
• The revised result therefore had 0 failures, exactly matching the article's expected table and console output.
• The second execution stopped with FileExistsError because outputs/regex_check_result already existed, as required.

References

The explanations and examples were written for this site. See the official sources below for the related behavior and concepts.

• Official Python 3.12 documentation · re ↗ (New tab)
• Official Python 3.12 documentation · re.fullmatch ↗ (New tab)
• Official Python 3.12 documentation · csv ↗ (New tab)