Data

Databricks platform administration and governance with Terraform

Converted an existing Databricks environment into infrastructure as code, so engineers experiment faster and request Databricks resources through pull requests.

The problem

Data engineers could not provision new medallion schemas in Unity Catalog through a clear, repeatable process. Cloud, IAM, and Security teams received tickets with missing context, leaving governance as an afterthought.

This request path is one focus of our Databricks platform administration work.

How we worked

  1. Assess. We worked with Data Engineering to trace how requests moved between teams. The bottleneck became clear: requesting a new schema, workspace, or service principal required several tickets, with context and approvals split across Data Engineering, Cloud, IAM, and Security.

  2. Codify. We imported the existing Databricks workspace configuration and Unity Catalog resources into Terraform, creating a version-controlled representation of the live environment and a repeatable way to propose changes.

  3. Operationalize. We defined the governance boundaries for who could request, review, and approve each type of change. Pull requests gave Cloud and Security teams one place to validate changes against the permissions model the business intended to follow.

  4. Enable. Data Engineering received a runbook and an example showing how to make a change. Cloud Engineering received documentation explaining how infrastructure changes are tested, promoted across environments, applied, and recovered when necessary.

The outcome

The Unity Catalog changes in scope now move through pull-request review instead of separate ticket queues. Human access is standardized around team-owned groups, with service principals used for automation. Both are managed through version control.