Data Pipeline Specification Template
A fill-in specification for a data pipeline — source contract, schedule, idempotency, freshness and volume checks, failure behaviour, ownership and the alert that goes to a human who can act.
Markdown. No sign-up, no email.
The question this document exists to answer: when this pipeline produces nothing, or produces something wrong, who finds out and how? Everything else is detail.
Pipeline: _______________ Owner (person): _______________ Date: _______ Version: _______
1. Purpose#
What decision or product depends on this data: _______________
Who consumes the output, and what they do with it: _______________
What happens downstream if it is a day late: _______________
What happens downstream if it is silently wrong: _______________
Those last two answers are usually very different, and they should drive different alerts. Late is visible. Wrong is not.
2. Source#
| System | |
| Access method (API, file, database, feed) | |
| Owner of the source (person) | |
| Do they know we depend on it | yes / no |
| How we are told about schema changes | |
| Rate limits / quotas | |
| Historical availability, honestly |
🔴 If nobody at the source knows you consume it, your pipeline will break the day they make a routine change. Introduce yourself before that happens.
3. Contract with the source#
| Field | Type | Nullable | Meaning | What we do if it is missing |
|---|---|---|---|---|
Expected volume per run: _____ (min _____ / max _____) Expected freshness: data should be no older than _____ Known quirks: _______________ (timezones, encodings, duplicate keys, retroactive edits)
4. Schedule and idempotency#
| Runs | |
| Why that time (upstream availability?) | |
| Late-arriving data window | |
| Safe to re-run the same period twice | yes / no |
| How re-running is made safe | overwrite partition / upsert on key / other |
| Backfill procedure |
A pipeline you cannot safely re-run cannot be recovered under pressure. If the answer above is "no", fix that before anything else on this page.
5. Transformations#
| Step | What it does | Assumption it relies on |
|---|---|---|
Business rules encoded here that are documented nowhere else: _______________
6. Output#
| Destination | |
| Write mode | append / overwrite / merge |
| Partitioning | |
| Schema owned by | |
| How consumers are told about changes | |
| Retention |
7. Quality checks — the ones that catch silent failure#
Run these on the OUTPUT, on every run, and fail the run when they fail.
| Check | Threshold | Action on failure |
|---|---|---|
| Row count within expected range | fail / warn | |
| Row count not zero | fail | |
| Freshness: max timestamp within ___ of now | fail / warn | |
| Primary key unique | fail | |
| Null rate per critical column below ___ | fail / warn | |
| Values within domain (currencies, statuses, ranges) | fail / warn | |
| Totals reconcile to source within ___ | fail / warn | |
| Distribution shift versus last ___ runs | warn |
🔴 A job that succeeds having written zero rows is the most common silent data failure there is. Exit codes describe whether code ran, not whether data arrived. Check the values.
8. Failure behaviour#
| On source unavailable | retry ___ times, then ____ |
| Does a failed run leave the previous good data intact | yes / no |
| Partial write possible | yes / no — how prevented |
| Alert goes to | |
| Alert channel | |
| Alert fires on state change or every run | |
| Runbook location |
Fail closed: when in doubt, keep the last good output and raise an alarm. Stale data that is labelled stale is recoverable; empty or corrupted output propagates downstream in minutes.
9. Observability#
- [ ] Run start, end, duration and row counts logged
- [ ] Metrics retained long enough to see a trend
- [ ] Someone would notice if this pipeline stopped running entirely
- [ ] Dashboard shows last successful run and freshness
- [ ] Cost per run known
How would we notice if this pipeline stopped running at all? _______________ (Monitoring the runs is not enough — a pipeline that is never triggered produces no failures.)
10. Sign-off#
| Name | Date | |
|---|---|---|
| Built by | ||
| Source owner informed | ||
| Consumer accepted | ||
| In production since |