Ask AI Questions About a Document and Check Line Citations

01Why require line-number citations

When an AI chat assistant answers questions about a document, the main problem is not whether the answer sounds fluent. The important question is whether each claim is supported by the supplied document. Requiring line-number citations gives the reviewer a direct path back to the source instead of asking them to search the entire file manually.

A useful workflow separates two checks. First, software can verify that every citation points to real line numbers and can reproduce those lines for inspection. Second, a person checks whether the cited text actually supports the claim. The automatic check catches broken or impossible references, but it does not prove semantic support by itself.

02Create a tiny synthetic document

Consider this synthetic project note. Save the seven lines exactly as shown in a plain-text file named project_note.txt. Because the example is intentionally small, every answer and citation can be checked manually.

Suppose the question is: What schedule changed, why did it change, and what deadlines follow? The source clearly states that the prototype test moved from 12 Sep to 16 Sep, the reason was a late replacement sensor, draft results are due 18 Sep, and the final review meeting is scheduled for 21 Sep.

03Require a strict citation format in the prompt

Do not merely ask the AI to cite the document. Define the exact format so that a script can parse the answer later. In this example, every factual sentence must end with a citation such as [L2-L3] or [L5].

1. Tell the AI to use only the supplied document.
2. Require citations in the exact form [Lx] or [Lx-Ly].
3. Ask it not to cite lines that do not directly support the preceding claim.
4. Tell it to say that the document does not provide an answer when evidence is missing.
5. Keep the source file unchanged while reviewing so its line numbers remain stable.

04Inspect a typical cited answer

A suitable answer might be: The prototype test moved from 12 Sep to 16 Sep because the replacement sensor arrived late. [L2-L3] Draft results are due 18 Sep, followed by the final review meeting on 21 Sep. [L5-L6]

This answer contains two citation ranges. The first should resolve to lines 2 and 3, and the second should resolve to lines 5 and 6. Before trusting the answer, a checker can confirm that those ranges exist and print the exact source text associated with each citation.

05Check every citation automatically with Python

Save the AI answer in ai_answer.txt. The script below reads project_note.txt and ai_answer.txt, extracts citations with a regular expression, rejects invalid ranges, and writes a review report to outputs/citation_check.txt. It never modifies the original document or AI answer, and it stops if the output report already exists.

import re
from pathlib import Path

DOCUMENT_FILE = Path("project_note.txt")
ANSWER_FILE = Path("ai_answer.txt")
OUTPUT_DIR = Path("outputs")
OUTPUT_FILE = OUTPUT_DIR / "citation_check.txt"

CITATION_RE = re.compile(r"\[L(\d+)(?:-L?(\d+))?\]")


def main():
    if not DOCUMENT_FILE.is_file():
        raise SystemExit(f"Document not found: {DOCUMENT_FILE}")

    if not ANSWER_FILE.is_file():
        raise SystemExit(f"AI answer not found: {ANSWER_FILE}")

    if OUTPUT_FILE.exists():
        raise SystemExit(f"Output already exists: {OUTPUT_FILE}")

    lines = DOCUMENT_FILE.read_text(encoding="utf-8").splitlines()
    answer = ANSWER_FILE.read_text(encoding="utf-8")
    matches = list(CITATION_RE.finditer(answer))

    if not matches:
        raise SystemExit("No citations found in AI answer.")

    report = []
    report.append(f"document_lines={len(lines)}")
    report.append(f"citations_found={len(matches)}")
    report.append("")

    invalid_count = 0

    for index, match in enumerate(matches, start=1):
        start = int(match.group(1))
        end = int(match.group(2)) if match.group(2) else start
        citation = match.group(0)

        valid = 1 <= start <= end <= len(lines)
        report.append(f"citation_{index}={citation}")
        report.append(f"valid_range={valid}")

        if not valid:
            invalid_count += 1
            report.append("source=INVALID LINE RANGE")
            report.append("")
            continue

        for line_number in range(start, end + 1):
            source_text = lines[line_number - 1]
            report.append(f"L{line_number}: {source_text}")

        report.append("")

    report.append(f"invalid_citations={invalid_count}")

    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
    OUTPUT_FILE.write_text("\n".join(report) + "\n", encoding="utf-8")
    print(f"Wrote: {OUTPUT_FILE}")


if __name__ == "__main__":
    main()

06Compare each claim with the extracted lines

For the synthetic answer, the checker should find two citations and zero invalid citation ranges. [L2-L3] maps to the test-date change and the late replacement sensor. [L5-L6] maps to the draft-results deadline and final review meeting.

Now deliberately try a bad citation such as [L8]. Because the synthetic document has only seven lines, the script should mark that range as invalid. This is useful for catching citations that look convincing in prose but cannot refer to the actual source.

07Avoid common mistakes and understand the limits

• Do not edit the source document after generating the AI answer; inserted or removed lines will change the numbering.
• Do not accept citations such as page names or vague phrases if your checker expects [Lx] syntax.
• Do not assume that an existing line proves the claim merely because the citation range is valid.
• Check whether the AI combines two facts from separate parts of a document but cites only one of them.
• Require the AI to state when the document lacks evidence instead of filling gaps from general knowledge.
• For long documents, preserve a stable source copy so later reviewers can reproduce the same line numbering.

This approach works well for plain-text review, but PDFs, spreadsheets, OCR output, and dynamically generated documents may not have stable line boundaries. In those cases, a different citation unit such as page number, paragraph ID, section ID, or table cell may be more reliable.

The workflow therefore has two layers: automatic validation for citation syntax and line ranges, followed by human verification of whether each cited passage actually supports the answer. Used together, they make document Q&A easier to audit than an uncited AI response.

Execution and verification record

2026-09-21 · hand-checked example · Python 3.12

• The synthetic document contains exactly 7 lines as listed in the article.
• The example answer contains exactly 2 citation ranges: [L2-L3] and [L5-L6].
• [L2-L3] was checked by hand against the synthetic source and contains the changed prototype test date and the stated reason.
• [L5-L6] was checked by hand against the synthetic source and contains the draft-results deadline and final review meeting date.
• A citation to [L8] would be outside the 7-line document and should therefore be marked as an invalid range by the shown logic.
• The regular expression accepts the intended forms [L2], [L2-L3], and [L2-3].

Site-wide writing and verification principles

References

The explanations and examples were written for this site. See the official sources below for the related behavior and concepts.

• Python re documentation ↗ (New tab)
• Python pathlib documentation ↗ (New tab)