Many government agencies still operate legacy applications and data environments that were never designed for today’s AI workloads. As pressure to modernize grows, agencies must do more than adopt AI. They need to know what data it can reach and whether that access can be explained.

An auditor asks a simple question: Who can access the applicant data?

The answer should take minutes. Instead, the data team opens cloud IAM policies. The warehouse administrator exports role assignments. Security reviews storage permissions. An application team checks service accounts. Someone else searches old access tickets. Five answers emerge, and no one is certain they describe the same reality.

This is not only an audit problem. It is an architecture problem.

Consider a fictional public benefits agency. Applicant data moves through operational databases, cloud storage, scheduled jobs, a warehouse, notebooks, dashboards, and a new AI assistant. Each system has its own permissions and activity records. More people can use the data, but fewer can explain the full access path.

Faster Spark processing and pressure to adopt AI do not justify a migration by themselves. The case becomes stronger when engineering, analytics, security, and AI operate across disconnected systems with separate access rules and incomplete lineage.

Databricks can bring those workloads onto one platform. Unity Catalog provides the governance structure connecting them.

Databricks can simplify a fragmented data environment, but only if Unity Catalog is designed as the foundation of the migration. Moving pipelines first and “adding governance later” simply recreates the old access problem on a new platform.

Five Systems, Five Answers

At the benefits agency, application records begin in PostgreSQL. Documents and extracts land in Amazon S3. Scheduled jobs clean and join the records. Analysts use a warehouse for reporting. Data scientists maintain notebooks and feature datasets. A new AI assistant will help authorized employees find case information.

The data passes through all of these systems, but its access model does not travel with it.

The database has roles. S3 has bucket policies and IAM permissions. The warehouse has another set of grants. Notebooks may use shared credentials. The AI application can retrieve whatever its service identity is allowed to reach.

When an employee changes teams, someone must find every place where that employee received access. When a sensitive field is copied into a new table, the masking rule must be recreated. When an auditor asks where a Social Security number traveled, the team reconstructs the answer from jobs, SQL, logs, and institutional memory.

Each system can work as designed while the agency still lacks a defensible view of access across them.

How Unity Catalog Changes the Migration

A Databricks migration can consolidate ingestion, transformation, analytics, streaming, and AI workloads. Unity Catalog governs the assets those workloads use.

Tables, views, volumes, functions, and registered models become securable objects. So do connections, storage credentials, service credentials, and external locations. Unity Catalog also supports model services and MCP services as governed AI assets; those service object types are currently Beta. The agency can grant privileges to users, groups, and service principals through one hierarchy.

The benefits agency could begin by federating selected PostgreSQL data for governed, read-only exploration. High-use datasets would move into Delta tables. Catalogs and schemas would define production boundaries and ownership, while account-level groups would represent eligibility analysts, investigators, engineers, and application identities.

Sensitive columns would be identified and tagged before masks or row filters are applied. Lineage would show which supported pipelines, dashboards, and models use the data. Audit records would capture platform activity. Once the new controls were tested, the agency would remove the old access paths.

Databricks FedRAMP High and AWS GovCloud

Databricks on AWS GovCloud has FedRAMP High Authorized status. The compliance security profile is enabled by default on all AWS GovCloud workspaces. It requires a hardened compute image and AWS Nitro instances that provide network encryption between cluster nodes and encryption at rest for local disks. Automatic cluster update and enhanced security monitoring are also enabled. The agency still owns its assigned controls under the applicable responsibility matrix.

Feature parity is a separate issue. Data Classification is not currently listed as available in AWS GovCloud. The benefits agency therefore needs a supported process for identifying and tagging sensitive columns instead of assuming the platform will do it automatically.

Every proposed capability must be checked against the current AWS GovCloud feature-availability table. Databricks also warns that a feature may appear in that table before it is actually released. Verify it in the target environment. This separation between platform authorization and operational evidence also shapes Archetype Core’s public-sector work.

Unity Catalog Access Control in Practice

Suppose the agency creates a curated applicant table:

prod_benefits.curated.applicant_profile

Eligibility analysts need assigned cases. Investigators need broader access. Contact-center staff need contact information but should not see full Social Security numbers. The AI assistant should return only the records its user may see.

At the object level, Unity Catalog privileges establish the baseline access path. A simplified grant might look like this:

GRANT USE CATALOG
  ON CATALOG prod_benefits
  TO `eligibility_analysts`;

GRANT USE SCHEMA
  ON SCHEMA prod_benefits.curated
  TO `eligibility_analysts`;

GRANT SELECT
  ON TABLE prod_benefits.curated.applicant_profile
  TO `eligibility_analysts`;

That grant establishes baseline access. Row filters determine which records members of the group see. Column masks redact values they are not authorized to view. Unity Catalog attribute-based access control for row filters and column masks is generally available. It uses account-level governed tags to extend a policy across matching assets; ungoverned tags do not qualify.

Those policies require serverless compute, standard compute on Databricks Runtime 16.4 or above, or dedicated compute on Runtime 16.4 or above with fine-grained access control filtering enabled. Unsupported compute fails closed rather than bypassing the policy. The agency should confirm the supported configuration in its target GovCloud workspace before making ABAC an architectural dependency.

This is where Unity Catalog becomes more than a catalog of tables. It becomes an enforcement point.

The agency can inspect lineage to see whether the applicant table feeds a report, model, or AI application. It can query system.access.audit to investigate access and administrative events. The audit system table is currently Public Preview with a documented 365-day free retention period. Availability must be checked for the agency’s deployment.

What Unity Catalog Actually Changes

One access model is not automatic

If the AI assistant is deployed as a Databricks App, it receives a dedicated service principal. When the app uses that identity to access data, all users share its permissions. That works for common operations, but it does not enforce a different data boundary for each person.

If the benefits agency wants its AI assistant to respect an employee’s Unity Catalog privileges, the app must use user authorization for those requests. Unity Catalog then evaluates the employee’s identity, including applicable row filters and column masks. The team must design and test the scopes, service-principal privileges, and audit behavior. Otherwise, the AI assistant can become a side door.

Lineage becomes usable

Lineage can trace a field from a source table through transformations into reports and models. That helps with impact analysis and incident investigation. It also answers a question an access list cannot: What could this data influence?

Audit evidence becomes queryable

Databricks audit logs and the audit system table give the agency a common place to investigate platform activity. The 365-day free retention period does not replace the agency’s own retention and review requirements. It does reduce the number of systems the team must search.

Migration can be phased

Query federation provides governed, read-only access to supported external data through foreign catalogs. A federated legacy Hive metastore is the documented exception: its foreign tables can be writable. AWS Glue federation can also expose existing assets through Unity Catalog during a transition. The agency can learn which workloads belong on Databricks before moving everything.

Databricks Migration Tradeoffs

Unity Catalog brings old access and ownership decisions into the open. The migration team has to resolve them.

Identity debt

Importing overlapping groups and broad entitlements only centralizes the confusion. The team must clean up identities, assign ownership, and remove access that no longer has a purpose. This work is often harder than moving the data.

Catalog boundaries

Catalogs and schemas define ownership, privilege inheritance, and isolation. Too many boundaries create administrative burden. Too few produce broad permissions. The design must reflect how the agency actually operates.

Legacy paths

Hive metastore tables, views, jobs, and pipelines may require synchronization, cloning, federation, or recreation. Code using two-part names may need Unity Catalog’s three-level namespace. The team still needs a dependency map, parallel validation, a rollback plan, and proof that the legacy route was retired.

Storage bypass

Databricks recommends managed tables for most workloads. External tables can preserve interoperability with agency-controlled storage, but Unity Catalog cannot control someone who bypasses Databricks and reads the files directly. Broad S3 permissions can undo a careful catalog design.

Federation limits

Federation works well for discovery and staged migration, but write behavior and performance depend on the source. Query federation is read-only except for federated legacy Hive metastore tables, which can be writable. High-volume analytical workloads may still need ingestion.

Platform dependency

Consolidation increases the need for Databricks skills, cost controls, and service availability. Open data formats help with portability, but permissions, policies, and job definitions do not move automatically. The architecture still needs an exit plan.

Governance Alone Is Not the Business Case

For the benefits agency, the case becomes credible when engineering and AI need the same governed data, permissions conflict across systems, and every access review requires manual reconciliation. Incomplete lineage and workloads that have outgrown the current architecture add weight.

The case is weak when an agency has a stable warehouse, predictable workloads, and an access model it can already defend. Unity Catalog may be valuable without justifying a full replatforming.

The agency must fund the operating-model change as well as the platform. It should ask which problems Databricks would remove, which it would relocate, and what new responsibilities it would create.

A Practical Databricks Migration Sequence

For regulated data, begin with a bounded pilot. The benefits agency could use one eligibility domain to determine whether the governance improvement justifies a broader migration. When the current evidence chain is unclear, the AI and Data Readiness Review provides a bounded way to map one use case before deciding what to modernize.

  1. Inventory the current reality. Document the stores, pipelines, identities, permissions, sensitive fields, consumers, and direct storage paths. Design from the current reality.

  2. Choose one meaningful domain. Choose a use case with real value and real governance requirements. For the benefits agency, that might be the applicant profile and its eligibility workflow. A sensitive-data pilot reveals more than a demonstration using public samples.

  3. Design access before moving data. Define what each role can see, who approves access, how exceptions work, and what evidence must remain. Then establish catalog boundaries, ownership, groups, service principals, storage credentials, and external locations.

  4. Decide what to federate and what to move. Use federation where read-only access and source performance are acceptable. Ingest data that needs repeated transformation, lower latency, or Databricks-managed optimization.

  5. Rebuild and test access. Review old permissions before carrying them forward. Grant access through groups, separate people from application identities, and test both allowed and prohibited behavior. Confirm masks, row filters, lineage, audit events, and service-identity behavior.

  6. Remove the old route and measure. Revoke legacy credentials, direct storage access, old clusters, and parallel metastore routes. Test the revocation. Then compare access-review effort, audit response time, reliability, performance, and cost before expanding.

The Auditor Asks Again

The strongest reason to move to Databricks is a shared governance model across engineering, analytics, and AI.

Unity Catalog provides the center. Its value depends on the architecture and operating discipline around it.

At the end of the pilot, the auditor asks again: Who can access the applicant data?

This time, the benefits agency should be able to answer through one governed model instead of reconstructing five partial answers. The migration succeeds when the agency can answer five questions without starting a fire drill:

  1. Who has access?
  2. What exactly can they see or change?
  3. Why was the access granted?
  4. When was it last used or reviewed?
  5. Which downstream data and AI assets can it affect?

If the new architecture cannot answer those questions, the agency has moved its workloads but not solved its governance problem.

Technical References

Back to Insights