FIELD GUIDE 05
Data for enterprise AI
Carry entitlement, provenance and retention from source to retrieval.
Start with the entitlement path.
Trace the user or service identity through catalog discovery, retrieval, tool execution and logs. Record which permissions are evaluated at index build time and which are evaluated when someone makes a request.
Record provenance that can be replayed.
Associate sample responses with source identifiers, dataset versions and retrieval configuration. Include a refusal test, a revoked-source test and a test against a restricted attribute. Keep the results with the intended use and reviewer.
Make retention visible.
Inventory embeddings, prompts, caches and traces as well as source records. For deletion or correction, specify how each derived store is updated and how to prove the new behaviour. Apply oversight to the workflow and the people operating it.
Primary references
- FrameworkNIST AI Risk Management Framework
- StandardOpenLineage project and metadata model
- SnowflakeHorizon Catalog product reference
- DatabricksDevelop agents with Agent Bricks