Governance & PII

Know which columns are personal, restrict who can read them, and be able to delete one person from a lake that was designed to be immutable.

Data Platform in Production · Intermediate · ~5 min

In plain English

Knowing which drawers hold personal files, keeping them locked, and being able to pull one person's file out of a filing system designed never to remove anything.

Why it's worth your time

Erasure in an append-only lake is genuinely hard, and 'we could probably do it' is not an answer an audit accepts.

If you remember three things

Classify into metadata tags, then attach masking policies to the tags
Deleting rows isn't enough — time travel still serves them
Crypto-shredding satisfies erasure without rewriting petabytes

Overview

Governance is three practical capabilities, not a policy document. Classification: you know which columns contain personal data, and the knowledge lives in metadata rather than in someone's head. Access control: who can read those columns is enforced by the platform — column masking, row-level policies, separate schemas — rather than by convention. And erasure: when a person exercises their right to deletion, you can actually find and remove their data, which is genuinely hard in an append-only lake full of Parquet files that were designed never to change.

In an interview

Governance means classifying PII columns in metadata, enforcing access with column masking and row-level policies rather than convention, and being able to delete an individual on request. Erasure is the hard one in an immutable lake: the workable patterns are lakehouse-format deletes, or crypto-shredding — encrypt per subject and destroy the key.

Production defaults

Classification: automated scan, human confirmation, tag-driven policy
Retention: short time-travel window on PII tables, with scheduled snapshot expiry
Non-production: mask or synthesise on the way in, without exception

What breaks

Deleted data still readable — Snapshots weren't expired. Delete plus expire, and shorten the retention window.
PII inventory is out of date — It's maintained by hand. Automate the scan and require confirmation on new tables.

Watch it explained

How to Anonymization and Data Masking for PostgreSQL in Ubuntu 22.04 LTS Server — The Lazy SysAdmin, 8:02