Module 01 · Handling data

Cleaning and validation

Every value arrives as a string and something has to decide what it means. Coercion that guesses, a schema that reports instead of throwing, and a quarantine so a bad row is kept where somebody can look at it.

3 lessons · 3 graded exercises·2.5 hours·beginner·Prerequisite: Records and formats

Coercion and what it costs

25 min · exercise

A CSV field is a string. Turning it into a number, a date or a boolean is a guess, and the guess has to be able to say it failed rather than returning something plausible.

A schema that reports

30 min · exercise

Validation that raises tells you about one row. Validation that collects tells you about the extract, which is the thing you need to make a decision about.

Quarantine, not deletion

25 min · exercise

Dropping a bad row makes the load succeed and the number wrong. Keeping it somewhere with the reason attached costs almost nothing and is the difference between a bug you can find and one you cannot.