Module 03 · Moving data

Pipelines

Running a job once is easy. Running it again tomorrow, and again over last month, and again after it died halfway, is the job. Idempotence, watermarks, backfills and resumable runs.

4 lessons · 3 graded exercises·3.5 hours·intermediate·Prerequisite: Storage and query

Idempotence

25 min

A job you cannot safely run twice is a job you cannot safely run at all, because sooner or later something will run it twice and it will not be you.

Watermarks

35 min · exercise

An incremental load needs to know where it got to. The obvious answer, the maximum timestamp it has seen, is wrong in a way that quietly loses rows.

Backfills

30 min · exercise

Re-running a pipeline over the past is the same operation as running it over today, and if it is not, one of the two is wrong.

The partial run

30 min · exercise

Everything so far assumed the job finished. Most of the interesting failures are the ones where it did not, and a run that cannot say what it completed cannot be resumed.