Databricks platform administration and governance with Terraform

The problem

Data engineers could not provision new medallion schemas in Unity Catalog through a clear, repeatable process. A new schema meant a ticket, a wait, and manual work in the Databricks UI by another team. Cloud, IAM, and Security teams received tickets with missing context, leaving governance as an afterthought.

The platform ran across four environments, each on its own long-running Git branch, so configuration drifted between them. CI authenticated with static service principal credentials, which made rotation and audit harder than they needed to be.

This request path is one focus of our Databricks platform administration work.

How we worked

1. Assess. We worked with Data Engineering to trace how requests moved between teams. The bottleneck became clear: requesting a new schema, workspace, or service principal required several tickets, with context and approvals split across Data Engineering, Cloud, IAM, and Security.

2. Codify. We brought the existing Databricks workspace configuration and Unity Catalog resources fully under Terraform, creating a version-controlled representation of the live environment and a repeatable way to propose changes. The repository moved to a single branch with one directory per environment. Catalogs, medallion schemas, external locations on ADLS Gen2, cluster policies, and per-schema group grants all became code, and the network was rebuilt with per-workspace virtual networks and managed-identity access to Unity Catalog storage.

3. Operationalize. We defined the governance boundaries for who could request, review, and approve each type of change. Pull requests gave Cloud and Security teams one place to validate changes against the permissions model the business intended to follow. Terraform plans rendered into each pull request, each environment had its own code owners, and applies were locked so two changes could not run at once. GitHub Actions reached Azure through OIDC federated credentials, which removed the static CI secrets.

4. Enable. Identity ran through Microsoft Entra ID. Databricks sign-in used the same Conditional Access policies as other Microsoft 365 workloads, Unity Catalog grants went to Entra ID security groups instead of named users, and SCIM provisioning kept Databricks users and groups in step with Entra ID. A Terraform module wrapped a new schema, its storage location, its group grants, and its cluster-policy bindings, so Data Engineering could add a schema to all four environments through one pull request. Data Engineering received a runbook and an example showing how to make a change. Cloud Engineering received documentation explaining how infrastructure changes are tested, promoted across environments, applied, and recovered when necessary.

The outcome

The migration ran without workload interruption. The Unity Catalog changes in scope now move through pull-request review instead of separate ticket queues. Human access is standardized around team-owned groups, with service principals used for automation. Both are managed through version control, and CI no longer holds long-lived credentials to rotate.