Find duplicate files by size and SHA-256 before deleting anything

01Define what the report means

This workflow groups files by byte size first. Only size groups containing multiple paths need hashing: files with different sizes cannot contain exactly the same bytes. Within each size group, matching SHA-256 digests identify duplicate candidates. Matching names are neither required nor sufficient.

A candidate group is not a deletion instruction. Identical content can serve different purposes in different folders, and a digest is not an absolute proof of equality. The report records every matching path without automatically choosing a file to keep.

02Create seven synthetic files

The following dataset is synthetic and was written for this article. Save the code as create_duplicate_demo.py and run it from your working folder. Binary writes make the contents exact: no newline is appended, and the empty files contain zero bytes.

from pathlib import Path

DEMO = Path("outputs") / "duplicate_demo"
FILES = {
    "archive/a_saved.txt": b"abc",
    "empty_a.txt": b"",
    "empty_b.txt": b"",
    "notes/a.txt": b"abc",
    "notes/a_copy.txt": b"abc",
    "notes/other.txt": b"xyz",
    "unique.txt": b"solo",
}

DEMO.parent.mkdir(parents=True, exist_ok=True)
DEMO.mkdir()  # Refuse to reuse an existing destination.
for relative_path, content in FILES.items():
    target = DEMO / relative_path
    target.parent.mkdir(parents=True, exist_ok=True)
    with target.open("xb") as stream:
        stream.write(content)

python create_duplicate_demo.py

The seven files contain 16 bytes altogether. Four have size 3, but notes/other.txt contains different bytes from the three abc files. This deliberately tests why equal size alone is insufficient.

03Keep the report outside the scanned folder

1. Save the next script as find_duplicate_files.py beside the setup script.
2. Leave SOURCE set to outputs/duplicate_demo for this example.
3. Keep outputs/duplicate_review separate from the source. That destination must not already exist.
4. Run the script from the same working folder using the command below.

python find_duplicate_files.py

The script rejects a report destination inside the resolved source folder. This prevents the scan from including its own output. Relative paths still depend on the terminal's working directory.

04Scan files and write the review CSV

The scanner skips symbolic links discovered during traversal and ignores non-regular file entries. Traversal and read errors stop the run. Metadata comparisons around hashing detect some concurrent changes, but they do not create a locked snapshot.

import csv
import hashlib
import os
import stat
from pathlib import Path

SOURCE = Path("outputs") / "duplicate_demo"
OUTPUT_DIR = Path("outputs") / "duplicate_review"
CHUNK_BYTES = 1024 * 1024


def signature(info: os.stat_result) -> tuple[int, ...]:
    return (info.st_dev, info.st_ino, info.st_size, info.st_mtime_ns)


def raise_walk_error(error: OSError) -> None:
    raise error


def file_digest(path: Path, expected: os.stat_result) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as stream:
        before = os.fstat(stream.fileno())
        if not stat.S_ISREG(before.st_mode) or signature(before) != signature(expected):
            raise RuntimeError(f"File changed before hashing: {path}")
        while chunk := stream.read(CHUNK_BYTES):
            digest.update(chunk)
        after = os.fstat(stream.fileno())
    if signature(after) != signature(expected) or signature(path.lstat()) != signature(expected):
        raise RuntimeError(f"File changed during hashing: {path}")
    return digest.hexdigest()


def main() -> None:
    root = SOURCE.resolve(strict=True)
    if not root.is_dir():
        raise ValueError("SOURCE must be a directory.")
    if OUTPUT_DIR.resolve().is_relative_to(root):
        raise ValueError("The report folder must be outside SOURCE.")

    OUTPUT_DIR.parent.mkdir(parents=True, exist_ok=True)
    OUTPUT_DIR.mkdir()  # Stop if the destination already exists.

    by_size = {}
    scanned = 0
    for directory, directories, filenames in os.walk(
        root, followlinks=False, onerror=raise_walk_error
    ):
        base = Path(directory)
        directories[:] = sorted(
            name for name in directories if not (base / name).is_symlink()
        )
        for name in sorted(filenames):
            path = base / name
            info = path.lstat()
            if not stat.S_ISREG(info.st_mode):
                continue
            by_size.setdefault(info.st_size, []).append((path, info))
            scanned += 1

    groups = []
    hashed = 0
    for size, members in sorted(by_size.items()):
        if len(members) < 2:
            continue
        by_hash = {}
        for path, initial in members:
            digest = file_digest(path, initial)
            by_hash.setdefault(digest, []).append(path)
            hashed += 1
        for digest, paths in sorted(by_hash.items()):
            if len(paths) > 1:
                ordered = sorted(paths, key=lambda path: path.as_posix())
                groups.append((size, digest, ordered))

    report = OUTPUT_DIR / "duplicates.csv"
    with report.open("x", encoding="utf-8", newline="") as stream:
        writer = csv.writer(stream)
        writer.writerow([
            "group_id", "size_bytes", "sha256", "relative_path", "review_status"
        ])
        for number, (size, digest, paths) in enumerate(groups, start=1):
            for path in paths:
                writer.writerow([
                    f"G{number:03d}", size, digest,
                    path.relative_to(root).as_posix(), "UNREVIEWED"
                ])

    matched = sum(len(paths) for _, _, paths in groups)
    print(f"Scanned {scanned} regular file paths; hashed {hashed}.")
    print(f"Duplicate candidates: {len(groups)} groups, {matched} paths.")
    print(f"Review CSV: {report.as_posix()}")


if __name__ == "__main__":
    main()

05Compare the expected groups

The expected report has five data rows plus its header. Groups are ordered by size and then digest; paths within each group are sorted. The table below omits the sha256 column because no digests were calculated here. The script fills that column when executed.

Six files require hashing: the two empty files and all four three-byte files. The four-byte file has no size peer and is skipped at the hashing stage. The expected console text is hand-derived, not an execution log.

Scanned 7 regular file paths; hashed 6.
Duplicate candidates: 2 groups, 5 paths.
Review CSV: outputs/duplicate_review/duplicates.csv

06Review meaning, not just matching bytes

1. Check that each reported path still exists and that the source has not changed since scanning.
2. Compare candidate files byte for byte before a destructive decision. Confirm that a separate, usable backup exists.
3. Check folder purpose, ownership, references from other files, and application requirements. Identical contents do not establish interchangeability.
4. Record a proposed action in a separate review copy. Do not interpret the first path in a group as the automatically preferred copy.

Empty files may be intentional placeholders. Their matching content does not make them unnecessary, and removing them would not eliminate any file-content bytes.

07Check the safeguards yourself

• Confirm that both empty files and all three abc files appear, while notes/other.txt and unique.txt do not.
• Check that each group has one shared size and one shared 64-character SHA-256 hexadecimal value.
• Run again without changing OUTPUT_DIR. FileExistsError should stop the run before another report is written.
• Try a separate source containing only unique sizes. Expect a header-only report and zero hashed files.

These are reader checks, not tests executed for this article. A completed report with zero groups is different from a scan that stopped with an exception.

08Recognize common errors

09Understand the limits

Hashing reads content in chunks, but the script stores file metadata, paths, and groups in memory. This is not a constant-memory inventory. It compares complete bytes, not visual similarity or document meaning; matching-looking documents can have different digests.

The report counts paths, not independent physical copies. Hard links can produce multiple reported paths without separate content storage, so reported sizes are not guaranteed disk savings. Windows junctions, mount points, and hostile concurrent changes require additional handling beyond the symbolic-link checks.

Execution and verification record

2026-09-20 · hand-checked example · target: Python 3.12 · standard library: csv, hashlib, os, pathlib, stat · no execution

• Manually counted seven files: two of size 0, four of size 3, and one of size 4, totaling 16 bytes.
• Identified the identical-content groups by inspecting the synthetic byte strings: two empty files and three abc files.
• Traced the size filter to six hashing candidates and the expected review report to two groups containing five paths.
• Inspected exclusive destination creation, report placement, traversal error handling, and the absence of deletion operations.
• Derived the expected console text and report row order by hand.

Site-wide writing and verification principles

References

The explanations and examples were written for this site. See the official sources below for the related behavior and concepts.

• Official Python 3.12 documentation · hashlib ↗ (New tab)
• Official Python 3.12 documentation · os.walk ↗ (New tab)
• Official Python 3.12 documentation · pathlib ↗ (New tab)
• Official Python 3.12 documentation · csv ↗ (New tab)