Module 04 · Moving data

Quality and contracts

A pipeline that fails loudly is a good afternoon. A pipeline that succeeds while producing wrong numbers can go unnoticed for a quarter. Counting rows, checking freshness, and writing the promise down as a test.

3 lessons · 2 graded exercises·2.5 hours·advanced·Prerequisite: Pipelines

Counting is a test

25 min · exercise

Almost every useful data check is a count and a threshold. The sophistication is in choosing what to count, not in the statistics.

Freshness

30 min · exercise

A pipeline that runs successfully and loads nothing looks exactly like one that runs successfully. Freshness is the check that tells them apart, and it is the one most often missing.

The contract

30 min

Everything in this course was one idea. Write down what you promise about a table, make the promise executable, and you have turned a pipeline into something somebody else can safely build on.