What a record is

A pipeline reads data from one location, changes its shape, and writes it somewhere else. The mechanics of reading and writing are straightforward. The difficulty comes from the data itself: records represent claims about real-world events, and upstream sources rarely validate entries thoroughly.

A row is a claim

Consider this row from the extract used throughout the course:

1004,ada@example.com,2026-03-02,,29.99,shipped

The record states that order 1004 was placed by ada@example.com on March 2, total 29.99, status shipped, with an empty quantity field.

Something shipped, but the exact quantity is unknown. This is not a formatting issue. It is an incomplete record, and downstream systems calculating quantities will produce incorrect totals.

Defaulting the value to zero allows calculations to execute, but distorts the record: an order with zero items did not ship. Defaulting to one is a common assumption, but inserting an assumed value into a factual column causes downstream consumers to treat an estimate as fact.

The preferred approach is to retain the row, flag the missing value explicitly, and ensure incomplete data is never aggregated without inspection.

Common input formats

Data arrives in several distinct formats with different trade-offs:

A delimited file separates fields with a specific character. CSV files are common and prone to edge cases: fields can contain delimiters when wrapped in quotes, quoted fields can contain escaped quotes, and records can span multiple lines. A simple string split fails on these cases and corrupts fields with complex content.

A document format such as JSON carries primitive types and supports nested structures. This avoids delimiter collisions, but introduces structural questions: a nested structure must be transformed before it can fit into tabular storage.

A database table enforces a schema directly in the storage engine. This provides deterministic type enforcement and constraints.

Data pipelines move records between these representations, and each transformation risks subtle data loss.

Where pipeline failures originate

Volume is rarely the primary failure mode in data engineering. Pipelines more commonly fail when field semantics drift, jobs run multiple times without idempotence, or upstream schema changes modify values without raising errors.

These failures share a common trait: state changes occur silently without triggering validation. A catastrophic error that halts execution immediately is straightforward to identify and resolve. A job that completes successfully while writing incorrect numbers can persist undetected for months, corrupting dependent analytics.

A reliable pipeline treats transformations as verifiable invariants across repeated executions.

Environment requirements

This course uses Python 3.10 or newer with standard library modules: csv, json, sqlite3, and datetime.

The sampledata module generates extracts deterministically for local test execution without external database dependencies or network access.

The following lesson covers parsing raw delimited text. Subsequent lessons define row schemas and enforce data validation across pipeline stages.

Seen in production

Dropbox documented the architecture of a document processing pipeline in Creating a modern OCR pipeline. The relevant takeaway is how early pipeline stages define, normalize, and extract structured records from unstructured receipt images.