Summarize Papers into a Comparison Table and Check Every Cell

01Why a paper comparison table needs cell-level checking

A comparison table is useful because it compresses several papers into the same structure: objective, method, data, result, and limitation. The risk is that an AI chat assistant may make the rows look consistent even when the abstracts do not provide the same level of detail. One abstract may name a dataset, another may only describe a sample, and a third may say nothing about limitations. Filling those gaps with plausible wording creates a neat table but weak evidence.

The safest rule is simple: every factual table cell should be supported by text from the supplied abstract. If the abstract does not state something, the cell should say Not stated rather than infer it from common practice or the paper title. This is especially important when the table will later be used for a literature review, research proposal, or method-selection discussion.

02Use three tiny synthetic abstracts

The following abstracts are synthetic examples created only for this tutorial. They are intentionally short so that every extracted table cell can be checked manually.

03Ask the AI for a constrained comparison table

Do not ask only for a summary of the papers. Define the columns and force the AI to use only information explicitly present in the abstracts. Also require a source field for every nonempty cell so that the claims can be checked later.

1. Provide the abstracts with stable IDs such as P1, P2, and P3.
2. Ask for fixed columns: method, data size, inputs or target, reported result, and limitation.
3. Require Not stated when an abstract does not provide the requested information.
4. Require a short supporting quote or source fragment for every factual cell.
5. Tell the AI not to infer experimental details, model settings, or limitations from general knowledge.

04Inspect a typical AI-produced table

A reasonable table would identify linear regression for P1, random forest for P2, and both Gaussian process regression and a neural network for P3. It should report 120 finite-element samples for P1, 500 simulated bracket designs for P2, and 200 simulation cases for P3. However, some columns are intentionally incomplete. For example, P2 does not state a limitation, so that cell should remain Not stated.

This table is concise, but it should not be accepted simply because every row looks complete. Each cell still needs a direct link back to the abstract text.

05Store the AI table with supporting evidence

For automatic checking, save a simplified CSV named paper_table.csv. Give each factual row a paper_id, field, value, and evidence column. The evidence should contain the exact fragment that the AI claims supports the value. A row with Not stated can use an empty evidence field.

06Check every evidence fragment with Python

Save the three abstracts in abstracts.txt using one line per paper in the format P1<TAB>abstract text. The script below reads abstracts.txt and paper_table.csv, checks whether each claimed evidence fragment actually appears in the corresponding abstract, and writes results to outputs/paper_table_check.txt. It does not modify the input files and stops if the output file already exists.

import csv
from pathlib import Path

ABSTRACTS_FILE = Path("abstracts.txt")
TABLE_FILE = Path("paper_table.csv")
OUTPUT_DIR = Path("outputs")
OUTPUT_FILE = OUTPUT_DIR / "paper_table_check.txt"


def load_abstracts(path):
    abstracts = {}
    with path.open("r", encoding="utf-8") as file:
        for line_number, line in enumerate(file, start=1):
            line = line.rstrip("\n")
            if not line:
                continue
            if "\t" not in line:
                raise SystemExit(f"Invalid abstract line {line_number}.")
            paper_id, text = line.split("\t", 1)
            abstracts[paper_id] = text
    return abstracts


def main():
    if not ABSTRACTS_FILE.is_file():
        raise SystemExit(f"Abstracts file not found: {ABSTRACTS_FILE}")
    if not TABLE_FILE.is_file():
        raise SystemExit(f"Table file not found: {TABLE_FILE}")
    if OUTPUT_FILE.exists():
        raise SystemExit(f"Output already exists: {OUTPUT_FILE}")

    abstracts = load_abstracts(ABSTRACTS_FILE)
    report = []
    checked = 0
    unsupported = 0

    with TABLE_FILE.open("r", encoding="utf-8", newline="") as file:
        reader = csv.DictReader(file)
        required = {"paper_id", "field", "value", "evidence"}
        if reader.fieldnames is None or not required.issubset(reader.fieldnames):
            raise SystemExit("paper_table.csv is missing required columns.")

        for row_number, row in enumerate(reader, start=2):
            paper_id = row["paper_id"].strip()
            field = row["field"].strip()
            value = row["value"].strip()
            evidence = row["evidence"].strip()

            if paper_id not in abstracts:
                raise SystemExit(f"Unknown paper_id on row {row_number}: {paper_id}")

            checked += 1
            if value == "Not stated":
                status = "REVIEW_NOT_STATED"
            elif evidence and evidence in abstracts[paper_id]:
                status = "EVIDENCE_FOUND"
            else:
                status = "EVIDENCE_NOT_FOUND"
                unsupported += 1

            report.append(
                f"{paper_id}\t{field}\t{status}\tvalue={value}\tevidence={evidence}"
            )

    report.append("")
    report.append(f"cells_checked={checked}")
    report.append(f"evidence_not_found={unsupported}")

    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
    OUTPUT_FILE.write_text("\n".join(report) + "\n", encoding="utf-8")
    print(f"Wrote: {OUTPUT_FILE}")


if __name__ == "__main__":
    main()

07Then check whether the evidence actually means the same thing

Finding the evidence text is only the first check. The reviewer must still compare the table value with the abstract wording. For example, P2 says prediction error was below 4% for 90% of validation cases. Rewriting that as prediction error was below 4% would be too strong because it removes the 90% condition.

08Know the limits of abstract-only comparison

• Do not treat an abstract as a substitute for the full paper when methods, datasets, or evaluation details matter.
• Do not convert Not stated into No limitation or None; absence from the abstract is not evidence that the limitation does not exist.
• Do not normalize different metrics into a ranking unless they are actually comparable.
• Do not infer sample splits, hyperparameters, statistical significance, or implementation details unless the abstract states them.
• Preserve qualifiers such as validation set, test set, percentage of cases, and comparison baseline.

Abstract-only comparison is useful for screening and organizing papers, but it is not sufficient for detailed methodological evaluation. The automatic script checks whether claimed evidence fragments are present in the correct abstract. It does not determine whether the paraphrase is accurate, whether two metrics are comparable, or whether a paper's full text would change the interpretation.

A reliable workflow is therefore: give each paper a stable ID, require fixed table columns, use Not stated for missing information, attach evidence to every factual cell, check the evidence automatically, and then review the meaning manually. This keeps the convenience of an AI-generated comparison table without treating a polished table as verified evidence.

Execution and verification record

2026-09-21 · hand-checked example · Python 3.12

• All three abstracts in this article are synthetic examples, not real papers.
• P1 was checked by hand: method is linear regression, data size is 120 finite-element samples, MAE is 0.18 mm on a 30-sample test set, and the stated limitation is linear-elastic cases only.
• P2 was checked by hand: method is random forest, data size is 500 simulated bracket designs, inputs are thickness and hole diameter, and error was below 4% for 90% of validation cases.
• P2 does not state a limitation in the synthetic abstract, so the comparison table correctly uses Not stated.
• P3 was checked by hand: it compares Gaussian process regression and a neural network on 200 simulation cases, reports lower RMSE for Gaussian process regression, and states that training time increased as the dataset grew.
• The shown Python logic checks literal evidence presence within the matching abstract and protects the existing output file from overwrite.

Site-wide writing and verification principles

References

The explanations and examples were written for this site. See the official sources below for the related behavior and concepts.

• Python csv documentation ↗ (New tab)
• Python pathlib documentation ↗ (New tab)