Check Whether AI-Cited References Actually Exist

01Why check AI-generated references?

A reference that looks professional is not automatically reliable. An AI response may contain a real URL with an inaccurate description, a URL that no longer exists, or a completely fabricated reference. Verification therefore has two separate questions: does the reference exist, and does it actually say something that supports the AI's claim?

• Existence check: can the URL be reached and does it return a normal web response?
• Content check: does the retrieved page contain information relevant to the claimed point?
• Source check: is the page really from the organization, publisher, or documentation site the AI named?
• Human review: does the page support the full claim rather than merely sharing a few keywords?

02Use a tiny synthetic example

The following example is synthetic. Imagine asking an AI chat assistant which Python standard-library pages explain opening URLs and working with JSON. You want to verify the answer before putting the references into a work note.

Prompt to the AI: Give me official Python references for opening URLs and reading or writing JSON. For each reference, state one specific claim that the page supports and provide the exact URL.

The third row is deliberately fake for this synthetic exercise. The goal is to make the verification process detect it rather than trusting the formatting of the citation.

03Turn the AI answer into explicit checks

Do not ask Python to decide whether an entire natural-language claim is true. Instead, record the URL, the claim, and a few terms that should reasonably appear if the page is relevant. The terms are only a screening aid.

1. Copy each URL exactly as the AI returned it.
2. Write the claim beside the URL instead of checking the URL alone.
3. Choose only a few distinctive screening terms.
4. After the automated check, open every reference you intend to cite and read the relevant passage yourself.

04Check the URLs and page text with Python

This standard-library script requests each page, records the final URL and HTTP status, extracts basic visible text, and checks the selected terms. It writes a report to outputs/reference_check.json. To avoid overwriting a previous review, it stops if the outputs directory already exists.

from pathlib import Path
from urllib.request import Request, urlopen
from urllib.error import HTTPError, URLError
import html
import json
import re

checks = [
    {
        "reference": "urllib.request",
        "url": "https://docs.python.org/3/library/urllib.request.html",
        "claim": "Python provides urllib.request for opening URLs.",
        "terms": ["urllib.request", "URL"],
    },
    {
        "reference": "json",
        "url": "https://docs.python.org/3/library/json.html",
        "claim": "Python's json module provides JSON encoding and decoding.",
        "terms": ["json", "encoding", "decoding"],
    },
    {
        "reference": "Synthetic fake reference",
        "url": "https://example.invalid/fake-reference",
        "claim": "This page documents an AI-specific reference verifier.",
        "terms": ["reference", "verifier"],
    },
]

output_dir = Path("outputs")
if output_dir.exists():
    raise SystemExit("outputs already exists; remove or rename it before running again")
output_dir.mkdir()

results = []

for item in checks:
    result = {
        "reference": item["reference"],
        "url": item["url"],
        "claim": item["claim"],
        "reachable": False,
        "status": None,
        "final_url": None,
        "term_matches": {},
        "error": None,
    }

    try:
        request = Request(
            item["url"],
            headers={"User-Agent": "Worknote reference checker/1.0"},
        )
        with urlopen(request, timeout=10) as response:
            result["status"] = response.status
            result["final_url"] = response.geturl()
            raw = response.read(500_000).decode("utf-8", errors="replace")

        text = re.sub(r"<script.*?</script>", " ", raw, flags=re.I | re.S)
        text = re.sub(r"<style.*?</style>", " ", text, flags=re.I | re.S)
        text = re.sub(r"<[^>]+>", " ", text)
        text = html.unescape(text)
        text = re.sub(r"\s+", " ", text)

        result["reachable"] = True
        lowered = text.casefold()
        result["term_matches"] = {
            term: term.casefold() in lowered for term in item["terms"]
        }

    except HTTPError as exc:
        result["status"] = exc.code
        result["error"] = f"HTTPError: {exc}"
    except URLError as exc:
        result["error"] = f"URLError: {exc.reason}"
    except Exception as exc:
        result["error"] = f"{type(exc).__name__}: {exc}"

    results.append(result)

report_path = output_dir / "reference_check.json"
report_path.write_text(
    json.dumps(results, indent=2, ensure_ascii=False),
    encoding="utf-8",
)

for result in results:
    print(result["reference"])
    print("  reachable:", result["reachable"])
    print("  status:", result["status"])
    print("  final URL:", result["final_url"])
    print("  term matches:", result["term_matches"])
    print("  error:", result["error"])

print(f"Saved: {report_path}")

05Interpret the result conservatively

A reachable page with matching terms should be labeled for manual review, not automatically marked as a verified claim. Keyword matching cannot determine context, scope, negation, or whether the AI exaggerated what the source says.

06Perform the final manual check

1. Confirm the page title and organization match what the AI named.
2. Search the page for the exact concept used in the claim.
3. Read enough surrounding text to understand qualifications and exceptions.
4. Check whether the AI changed a narrow statement into a broader conclusion.
5. Record the relevant section heading or passage for later review.
6. Reject or correct any citation whose source cannot be located or does not support the stated claim.

07Common mistakes and limits

• Assuming a familiar domain means every URL on that domain is genuine.
• Accepting a page because one keyword appears somewhere in it.
• Ignoring redirects and checking only the originally supplied URL.
• Treating a page title, search snippet, abstract, or table of contents as evidence for a detailed claim.
• Using automated text matching as a substitute for reading the cited passage.
• Forgetting that some valid sites block automated requests or require JavaScript, authentication, or other access methods.

For academic work, also verify bibliographic fields such as authors, title, year, journal, DOI, and the actual paper text. A URL checker cannot establish scholarly validity, detect retracted work, or decide whether evidence is methodologically strong.

Execution and verification record

2026-09-21 · hand-checked example · Python 3.12

• Checked by hand that the example contains three references and that the third is explicitly identified as a synthetic fake reference.
• Checked that the two real example URLs use official Python documentation paths for urllib.request and json.
• Checked that the script uses only Python standard-library modules.
• Checked that the script creates outputs/reference_check.json and stops instead of overwriting an existing outputs directory.
• Checked the logic separating URL reachability from simple term matching and manual claim review.

Site-wide writing and verification principles

References

The explanations and examples were written for this site. See the official sources below for the related behavior and concepts.

• Python documentation: urllib.request — Extensible library for opening URLs ↗ (New tab)
• Python documentation: json — JSON encoder and decoder ↗ (New tab)