Document what every dataset column means before analysis, including its unit, data type, and allowed values. A small synthetic test dataset shows how the same dictionary can also support simple automated validation.
Who this is forThis guide is for researchers who need a clear column-level description of a dataset that other people can read and check.
A column name is rarely enough to describe research data. speed could mean vehicle speed, wheel speed, or angular speed. accel could mean longitudinal, lateral, or total acceleration. A data dictionary records the intended meaning so later analysis does not depend on guessing.
This tutorial records six pieces of information for each column: column name, meaning, unit, data type, allowed values, and an example. The allowed-values field can describe categories, numeric ranges, uniqueness rules, or simple text patterns.
The following dataset is synthetic and was written specifically for this article. Save it as test_runs.csv. Four rows follow the intended rules. The fifth row deliberately contains four problems so the dictionary can be used as a validation reference.
run_id,speed_kmh,accel_m_s2,road_surface,test_status
R001,30,0.5,dry,complete
R002,50,-1.2,wet,complete
R003,70,0.0,dry,review
R004,40,-0.8,wet,complete
R005,135,fast,snow,done
There are 5 rows and 5 columns. R005 has a speed above the allowed range, a nonnumeric acceleration value, an unsupported road-surface category, and an unsupported test-status value. Its run_id itself is valid.
The dictionary should be understandable without opening the analysis code. Keep the meaning specific enough that another researcher can distinguish the column from similar measurements.
| column_name | meaning | unit | type | allowed_values | example |
|---|---|---|---|---|---|
| run_id | Unique identifier for one test run | string | R followed by exactly 3 digits; unique | R001 | |
| speed_kmh | Vehicle speed at the start of the record | km/h | integer | 0 to 120 inclusive | 50 |
| accel_m_s2 | Longitudinal acceleration | m/s^2 | float | -5.0 to 5.0 inclusive | -1.2 |
| road_surface | Road-surface condition used for the test | category | dry;wet | wet | |
| test_status | Review state of the test record | category | complete;review | complete |
An empty unit is intentional for identifiers and categorical labels because they are dimensionless metadata, not measured physical quantities. The type field describes the intended data representation rather than whatever a spreadsheet happens to infer automatically.
Save the following script as data_dictionary_check.py. The column specification is written once in SPEC. The script exports that specification as data_dictionary.csv and checks each row against the same rules. The original test_runs.csv is read only.
import csv
import re
from pathlib import Path
SOURCE = Path("test_runs.csv")
OUTPUT_DIR = Path("outputs") / "data_dictionary_result"
DICTIONARY = OUTPUT_DIR / "data_dictionary.csv"
ISSUES = OUTPUT_DIR / "validation_issues.csv"
SPEC = [
{
"column_name": "run_id",
"meaning": "Unique identifier for one test run",
"unit": "",
"type": "string",
"allowed_values": "R followed by exactly 3 digits; unique",
"example": "R001",
},
{
"column_name": "speed_kmh",
"meaning": "Vehicle speed at the start of the record",
"unit": "km/h",
"type": "integer",
"allowed_values": "0 to 120 inclusive",
"example": "50",
},
{
"column_name": "accel_m_s2",
"meaning": "Longitudinal acceleration",
"unit": "m/s^2",
"type": "float",
"allowed_values": "-5.0 to 5.0 inclusive",
"example": "-1.2",
},
{
"column_name": "road_surface",
"meaning": "Road-surface condition used for the test",
"unit": "",
"type": "category",
"allowed_values": "dry;wet",
"example": "wet",
},
{
"column_name": "test_status",
"meaning": "Review state of the test record",
"unit": "",
"type": "category",
"allowed_values": "complete;review",
"example": "complete",
},
]
def issue(row_number, column, value, problem):
return {
"row_number": row_number,
"column_name": column,
"value": value,
"issue": problem,
}
def main() -> None:
if not SOURCE.is_file():
raise FileNotFoundError(f"Source CSV not found: {SOURCE}")
if OUTPUT_DIR.exists():
raise FileExistsError(f"Output folder already exists: {OUTPUT_DIR}")
expected_columns = [item["column_name"] for item in SPEC]
issues = []
seen_run_ids = set()
row_count = 0
with SOURCE.open("r", encoding="utf-8-sig", newline="") as stream:
reader = csv.DictReader(stream)
if reader.fieldnames != expected_columns:
raise ValueError(
f"Header mismatch. Expected {expected_columns}, got {reader.fieldnames}."
)
for row_number, row in enumerate(reader, start=2):
row_count += 1
run_id = row["run_id"].strip()
if re.fullmatch(r"R[0-9]{3}", run_id) is None:
issues.append(issue(row_number, "run_id", run_id, "PATTERN_ERROR"))
elif run_id in seen_run_ids:
issues.append(issue(row_number, "run_id", run_id, "DUPLICATE_VALUE"))
seen_run_ids.add(run_id)
try:
speed = int(row["speed_kmh"])
except ValueError:
issues.append(issue(row_number, "speed_kmh", row["speed_kmh"], "TYPE_ERROR"))
else:
if not 0 <= speed <= 120:
issues.append(issue(row_number, "speed_kmh", row["speed_kmh"], "OUT_OF_RANGE"))
try:
accel = float(row["accel_m_s2"])
except ValueError:
issues.append(issue(row_number, "accel_m_s2", row["accel_m_s2"], "TYPE_ERROR"))
else:
if not -5.0 <= accel <= 5.0:
issues.append(issue(row_number, "accel_m_s2", row["accel_m_s2"], "OUT_OF_RANGE"))
if row["road_surface"] not in {"dry", "wet"}:
issues.append(issue(row_number, "road_surface", row["road_surface"], "ALLOWED_VALUE_ERROR"))
if row["test_status"] not in {"complete", "review"}:
issues.append(issue(row_number, "test_status", row["test_status"], "ALLOWED_VALUE_ERROR"))
OUTPUT_DIR.parent.mkdir(parents=True, exist_ok=True)
OUTPUT_DIR.mkdir()
dictionary_fields = [
"column_name", "meaning", "unit", "type", "allowed_values", "example"
]
with DICTIONARY.open("x", encoding="utf-8", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=dictionary_fields)
writer.writeheader()
writer.writerows(SPEC)
issue_fields = ["row_number", "column_name", "value", "issue"]
with ISSUES.open("x", encoding="utf-8", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=issue_fields)
writer.writeheader()
writer.writerows(issues)
print(f"Rows checked: {row_count}.")
print(f"Dictionary columns: {len(SPEC)}.")
print(f"Validation issues: {len(issues)}.")
print(f"Output folder: {OUTPUT_DIR.as_posix()}")
if __name__ == "__main__":
main()
Only R005 should produce issues. Because the CSV header is row 1, R005 is physical CSV row 6. Four issue rows should be written.
row_number,column_name,value,issue
6,speed_kmh,135,OUT_OF_RANGE
6,accel_m_s2,fast,TYPE_ERROR
6,road_surface,snow,ALLOWED_VALUE_ERROR
6,test_status,done,ALLOWED_VALUE_ERROR
The expected console output below was derived by hand from the synthetic dataset and script. It is not a captured execution log.
Rows checked: 5.
Dictionary columns: 5.
Validation issues: 4.
Output folder: outputs/data_dictionary_result| Problem | What to check |
|---|---|
| A new dataset column has no dictionary entry | Update the dictionary before analysis so the meaning and unit are documented. |
| The same concept appears with different units | Use separate columns or standardize units explicitly; do not rely on memory. |
| Category labels drift over time | Define allowed values and document intentional additions such as a new road condition. |
| Spreadsheet changes integer IDs into numbers | Store identifiers as strings when leading zeros or patterns are meaningful. |
| Validation passes but meaning is wrong | Structural checks cannot detect an incorrect scientific definition or mislabeled sensor channel. |
A practical research data dictionary often needs more than the six fields shown here. Depending on the project, add source system, sensor location, coordinate direction, sampling rate, nullable status, precision, calculation method, valid range, missing-value code, and responsible owner.
Allowed values should reflect scientific meaning rather than simply the minimum and maximum currently observed. If speed values happen to range from 30 to 70 km/h in one file, that does not automatically mean 30 to 70 is the valid engineering range.
Version the dictionary together with the dataset or analysis code. If a column definition, unit, sign convention, or category changes, record when the change happened. A data dictionary is most useful when it describes the exact version of the data being analyzed.
ChatGPT Python tool · Python 3.13.5 · standard library: csv, pathlib, re
The explanations and examples were written for this site. See the official sources below for the related behavior and concepts.