Filter files by size, compare SHA-256 digests, and write a review-only CSV without changing the source files. Use a seven-file synthetic example to check which paths belong in the report and which do not.
Who this is forThis guide is for people who want to identify potentially redundant files while keeping deletion decisions separate from the scan.
This workflow groups files by byte size first. Only size groups containing multiple paths need hashing: files with different sizes cannot contain exactly the same bytes. Within each size group, matching SHA-256 digests identify duplicate candidates. Matching names are neither required nor sufficient.
A candidate group is not a deletion instruction. Identical content can serve different purposes in different folders, and a digest is not an absolute proof of equality. The report records every matching path without automatically choosing a file to keep.
The following dataset is synthetic and was written for this article. Save the code as create_duplicate_demo.py and run it from your working folder. Binary writes make the contents exact: no newline is appended, and the empty files contain zero bytes.
from pathlib import Path
DEMO = Path("outputs") / "duplicate_demo"
FILES = {
"archive/a_saved.txt": b"abc",
"empty_a.txt": b"",
"empty_b.txt": b"",
"notes/a.txt": b"abc",
"notes/a_copy.txt": b"abc",
"notes/other.txt": b"xyz",
"unique.txt": b"solo",
}
DEMO.parent.mkdir(parents=True, exist_ok=True)
DEMO.mkdir() # Refuse to reuse an existing destination.
for relative_path, content in FILES.items():
target = DEMO / relative_path
target.parent.mkdir(parents=True, exist_ok=True)
with target.open("xb") as stream:
stream.write(content)
python create_duplicate_demo.py| Relative path | Exact text content | Bytes |
|---|---|---|
| archive/a_saved.txt | abc | 3 |
| empty_a.txt | Empty | 0 |
| empty_b.txt | Empty | 0 |
| notes/a.txt | abc | 3 |
| notes/a_copy.txt | abc | 3 |
| notes/other.txt | xyz | 3 |
| unique.txt | solo | 4 |
The seven files contain 16 bytes altogether. Four have size 3, but notes/other.txt contains different bytes from the three abc files. This deliberately tests why equal size alone is insufficient.
python find_duplicate_files.pyThe script rejects a report destination inside the resolved source folder. This prevents the scan from including its own output. Relative paths still depend on the terminal's working directory.
The scanner skips symbolic links discovered during traversal and ignores non-regular file entries. Traversal and read errors stop the run. Metadata comparisons around hashing detect some concurrent changes, but they do not create a locked snapshot.
import csv
import hashlib
import os
import stat
from pathlib import Path
SOURCE = Path("outputs") / "duplicate_demo"
OUTPUT_DIR = Path("outputs") / "duplicate_review"
CHUNK_BYTES = 1024 * 1024
def signature(info: os.stat_result) -> tuple[int, ...]:
return (info.st_dev, info.st_ino, info.st_size, info.st_mtime_ns)
def raise_walk_error(error: OSError) -> None:
raise error
def file_digest(path: Path, expected: os.stat_result) -> str:
digest = hashlib.sha256()
with path.open("rb") as stream:
before = os.fstat(stream.fileno())
if not stat.S_ISREG(before.st_mode) or signature(before) != signature(expected):
raise RuntimeError(f"File changed before hashing: {path}")
while chunk := stream.read(CHUNK_BYTES):
digest.update(chunk)
after = os.fstat(stream.fileno())
if signature(after) != signature(expected) or signature(path.lstat()) != signature(expected):
raise RuntimeError(f"File changed during hashing: {path}")
return digest.hexdigest()
def main() -> None:
root = SOURCE.resolve(strict=True)
if not root.is_dir():
raise ValueError("SOURCE must be a directory.")
if OUTPUT_DIR.resolve().is_relative_to(root):
raise ValueError("The report folder must be outside SOURCE.")
OUTPUT_DIR.parent.mkdir(parents=True, exist_ok=True)
OUTPUT_DIR.mkdir() # Stop if the destination already exists.
by_size = {}
scanned = 0
for directory, directories, filenames in os.walk(
root, followlinks=False, onerror=raise_walk_error
):
base = Path(directory)
directories[:] = sorted(
name for name in directories if not (base / name).is_symlink()
)
for name in sorted(filenames):
path = base / name
info = path.lstat()
if not stat.S_ISREG(info.st_mode):
continue
by_size.setdefault(info.st_size, []).append((path, info))
scanned += 1
groups = []
hashed = 0
for size, members in sorted(by_size.items()):
if len(members) < 2:
continue
by_hash = {}
for path, initial in members:
digest = file_digest(path, initial)
by_hash.setdefault(digest, []).append(path)
hashed += 1
for digest, paths in sorted(by_hash.items()):
if len(paths) > 1:
ordered = sorted(paths, key=lambda path: path.as_posix())
groups.append((size, digest, ordered))
report = OUTPUT_DIR / "duplicates.csv"
with report.open("x", encoding="utf-8", newline="") as stream:
writer = csv.writer(stream)
writer.writerow([
"group_id", "size_bytes", "sha256", "relative_path", "review_status"
])
for number, (size, digest, paths) in enumerate(groups, start=1):
for path in paths:
writer.writerow([
f"G{number:03d}", size, digest,
path.relative_to(root).as_posix(), "UNREVIEWED"
])
matched = sum(len(paths) for _, _, paths in groups)
print(f"Scanned {scanned} regular file paths; hashed {hashed}.")
print(f"Duplicate candidates: {len(groups)} groups, {matched} paths.")
print(f"Review CSV: {report.as_posix()}")
if __name__ == "__main__":
main()
The report has five data rows plus its header. Groups are ordered by size and digest, with sorted paths in each group. The table below omits the SHA-256 column for readability; the generated CSV includes it and the values were checked during execution.
| group_id | size_bytes | relative_path | review_status |
|---|---|---|---|
| G001 | 0 | empty_a.txt | UNREVIEWED |
| G001 | 0 | empty_b.txt | UNREVIEWED |
| G002 | 3 | archive/a_saved.txt | UNREVIEWED |
| G002 | 3 | notes/a.txt | UNREVIEWED |
| G002 | 3 | notes/a_copy.txt | UNREVIEWED |
Six files require hashing: the two empty files and all four three-byte files. The four-byte file has no size peer and is skipped. The execution matched the expected console output.
Scanned 7 regular file paths; hashed 6.
Duplicate candidates: 2 groups, 5 paths.
Review CSV: outputs/duplicate_review/duplicates.csvEmpty files may be intentional placeholders. Their matching content does not make them unnecessary, and removing them would not eliminate any file-content bytes.
These are reader checks, not tests executed for this article. A completed report with zero groups is different from a scan that stopped with an exception.
| Symptom | What to check |
|---|---|
| FileNotFoundError | Run the setup first and check SOURCE against the terminal's working directory. |
| FileExistsError | Review the previous destination and choose a fresh folder beneath outputs. Existing empty destinations are also refused. |
| PermissionError or another read error | Resolve access problems or narrow the source deliberately. Do not treat an interrupted scan as complete. |
| File changed before or during hashing | Stop applications modifying the source, then scan again using a fresh output destination. |
| Unexpected byte sizes | A text editor may have added a newline or changed encoding. Recreate the example with the binary-writing setup script. |
| Empty folder or partial report after failure | An interrupted run does not roll back its destination. Keep that result separate from completed reports. |
Hashing reads content in chunks, but the script stores file metadata, paths, and groups in memory. This is not a constant-memory inventory. It compares complete bytes, not visual similarity or document meaning; matching-looking documents can have different digests.
The report counts paths, not independent physical copies. Hard links can produce multiple reported paths without separate content storage, so reported sizes are not guaranteed disk savings. Windows junctions, mount points, and hostile concurrent changes require additional handling beyond the symbolic-link checks.
ChatGPT Python tool · Python 3.13.5 · standard library csv/hashlib/os/pathlib/stat
The explanations and examples were written for this site. See the official sources below for the related behavior and concepts.