FIELD GUIDE 02
Catalogs and lineage
Trace what users and agents can discover, read and reconstruct.
Separate discovery from permission.
A catalog can tell someone a dataset exists. Access still depends on identity, credentials, policy evaluation and enforcement by the engine serving it. Test allowed and denied reads with each identity that will actually use the data.
Follow the job, run and dataset.
OpenLineage models jobs, runs and datasets. It can describe how an output was produced when the relevant jobs report events and dataset identities line up. Decide who investigates a missing event, stale mapping or broken lineage link.
Rehearse a policy change.
Revoke access for a human and a service account, then retry through an alternate engine and a previously issued credential. Compare when new access stops with when old credentials expire. Save the actual audit records.
Primary references
- StandardOpenLineage project and metadata model
- SnowflakeHorizon Catalog product reference
- DatabricksUnity Catalog and the open lakehouse