Google Cloud Introduces Governance Agent for Proactive Data Metadata

The Governance Agent project uses column-level lineage to automatically propagate governance metadata, such as descriptions, business glossary terms, policy tags, and trust scores, across data assets. This approach aims to reduce 'governance debt' by ensuring context travels with data through transformations, eliminating the need for manual re-documentation at each stage.

The agent traces column lineage to propose descriptions from upstream sources. For columns resulting from SQL transformations like SUM() or CASE WHEN, it reads the actual SQL to generate accurate, new descriptions that reflect the transformation, rather than simply copying outdated upstream information.

It maps technical column names to business glossary terms using semantic similarity, ensuring consistent language across the organization. The agent can also extract explicit definitions from unstructured documents like PDFs or design specifications, avoiding guesswork based solely on column names.

The agent traces PII (Personally Identifiable Information) tags from source columns and recommends them downstream, verifying if transformations preserve sensitivity or anonymize data. It also provides summaries of current read access and applicable masking rules, enhancing risk management as data flows.

Instead of treating new views as unknowns, the agent derives a trust score based on Data Quality and Profiling results from its upstream sources. It also identifies and credits transformations that genuinely improve data quality, such as deduplication or null handling.

For data assets without clean upstream lineage, the agent allows users to provide external context through documents like policy PDFs, product specifications, or markdown design docs. Utilizing Vertex AI Search, it can retrieve relevant passages from these documents to ground metadata without inferring sensitive details like PII status.