Project Summary

Built an Azure Databricks lakehouse for intended HIPAA-regulated workloads from scratch, designed to attach to an existing regional Unity Catalog metastore already used by another regulated, health-data-handling Databricks environment within the organization. An initial standard deployment exposed more network surface than the regulated workload allowed, so the environment was rebuilt around VNet injection, Databricks’ compliance security controls, and ingress restricted to VPN and trusted internal networks only, while still supporting controlled data ingestion from on-premises systems and specific external sources.

This project is also what drove the broader hub-spoke network redesign and private DNS consolidation described elsewhere in this portfolio; the platform-level networking work existed to support this environment, not the other way around. The migration is currently in progress with contracted support; early metrics are promising: one migrated ETL process improved from approximately six hours to about 45 minutes, roughly an 87.5% reduction in elapsed processing time, though a fully quantified cost comparison isn’t in yet.

Problem

The organization needed a new Databricks lakehouse environment built around HIPAA-related security requirements and attached to the existing regional Unity Catalog metastore already supporting another regulated, health-data-handling Databricks environment. This wasn’t a lift-and-shift or a cleanup of something inherited; it was a greenfield build, but one that had to interoperate correctly with infrastructure that already existed.

An initial standard deployment, tested early on, exposed more network surface than the regulated workload requirements allowed. Rather than incrementally harden that deployment, the environment was rebuilt around VNet injection and explicitly controlled ingress, egress, and service connectivity from the start.

On top of the compliance requirement, the environment had two connectivity needs in tension with each other: on-premises systems needed to feed data into storage governed through Unity Catalog, and specific external-to-the-organization sites needed to be reachable to pull additional source data, all while keeping the environment itself restricted to VPN and trusted internal network access only, not broadly open in either direction.

Constraints

• New possible HIPAA-related security requirements drove the design directly; this wasn’t a generic best-practices exercise, the environment handles regulated health data.
• The new environment had to participate in an existing regional Unity Catalog governance model rather than establish an independent metastore.
• Access needed to be restricted to VPN and trusted internal networks only.
• On-premises systems needed a secure path to feed data into storage governed through Unity Catalog.
• Specific external-to-the-organization sites needed to be reachable from within the environment to pull source data, without broadly opening outbound access.
• Azure and on-premises DNS operated as separate resolution domains, requiring an explicit forwarding or resolver layer for hybrid name resolution.
• The environment is being deployed and migrated with contracted third-party support, alongside internal engineering; coordination across both was part of the constraint, not just the technical design.

Architecture

VNet-Injected, Compliance-Configured Databricks

The workspace runs VNet-injected rather than on Databricks’ default managed-network model, giving control over classic compute-plane networking instead of accepting Databricks’ default posture. VNet injection alone doesn’t restrict who can reach the workspace front end, though; that’s governed separately, through IP access restrictions and Azure Private Link limiting workspace ingress to VPN and trusted internal networks. NSGs provide segmentation within the VNet itself, and Databricks’ compliance security controls add the additional safeguards the regulated workload requires.

Built on a Hub-Spoke Foundation

This environment doesn’t have its own isolated network story; it runs on the hub-spoke architecture and consolidated private DNS zones built at the platform level. That platform work wasn’t done in the abstract; this project is the reason it exists. Getting Databricks secured for the regulated workload requirements required a network foundation capable of supporting it, and building that foundation properly became a prerequisite project in its own right.

Unity Catalog Across an Existing Deployment

Unity Catalog here isn’t a standalone catalog; the new workspace attaches to the organization’s existing regional Unity Catalog metastore, already in use by the regulated health-data group, rather than standing up an independent governance island. That gives the new environment its own workspace and network boundaries while still participating in the existing catalog and governance model, with on-premises systems feeding data into storage that Unity Catalog then governs, not into the catalog itself. It also meant the catalog design had to account for data and access patterns that predate this project, not just define a clean structure from a blank slate.

Bridging Serverless Compute Into a Private Network

Databricks serverless compute doesn’t run inside the customer’s own VNet; it runs in Microsoft’s managed environment outside the boundary that VNet injection controls. That’s a gap VNet injection alone doesn’t close: securing the workspace network doesn’t automatically secure the path from serverless compute back into protected Azure resources. Azure Network Security Perimeter addresses that specific gap, authorizing serverless compute to reach the Azure Storage it needs without reverting that storage to broad public-network access.

Controlled External Reach

Keeping the environment VPN-and-trusted-network-only didn’t mean isolating it from everything; specific external-to-the-organization sites still needed to be reachable to pull source data. Rather than treating that as an exception to punch a hole for, outbound access was scoped deliberately to the specific destinations that actually needed it, keeping the rest of the environment’s egress posture unchanged.

Making that egress traffic identifiable to the outside world required Azure NAT Gateway, one per environment, three total across prod, dev, and test, giving each VNet its own stable, known static outbound IP rather than unpredictable egress addresses. That matters in practice: external sites this environment needs to pull data from can allowlist a specific, fixed address instead of an unbounded range, which keeps the “controlled” in controlled external reach concrete rather than aspirational.

CI/CD Through a Self-Hosted Runner

Because the environment is network-isolated by design, standard GitHub-hosted Actions runners had no approved network path into it. A self-hosted runner was placed inside that trusted network instead, giving CI/CD a controlled deployment path to the environment. On top of the runner, GitHub repository and environment configuration was standardized across dev, test, and prod, each with its own service principal, separate from the service principals used by scheduled Databricks jobs, so deployment identity and job identity never overlap.

A Lightweight Hybrid DNS Resolver, Built Ahead of Need

Azure’s DNS and the organization’s on-premises DNS don’t know about each other’s zones, which is a real problem for an environment that needs to reach on-premises systems by name. Azure DNS Private Resolver is the native managed answer to that, but deploying dedicated resolver endpoints and subnets was more infrastructure than this stage of the project justified for what was, at the time, a relatively narrow requirement. A lightweight conditional-forwarding service was built on the trusted-network host that also runs the self-hosted GitHub Actions runner instead: queries for Azure-related namespaces route to Azure DNS, queries for on-premises/institutional namespaces route to the organization’s own DNS infrastructure. Since that host already had to sit inside the trusted network for CI/CD, it provided an economical place to also run resolution across the two domains without standing up another dedicated server. That service is built and validated but remains an available capability rather than a production dependency today, ready for when it’s needed rather than solving a problem that’s already blocking work.

Right-Sizing Compute

Compute node configuration was tuned for a performance-to-cost ratio rather than defaulting to a standard sizing recommendation, matching cluster shape to the workload instead of over-provisioning for headroom that isn’t actually used.

Engineering Challenges

Two Different Kinds of “Locked Down”

The environment needed to be tightly restricted, VPN and trusted networks only, while still reaching specific external sites and on-premises systems. Those aren’t the same problem: inbound access needed to stay closed to everything except trusted origins, while outbound access needed to be selectively open to a small, specific set of destinations. Solving both without loosening the environment’s overall posture meant treating them as two separate, deliberately scoped rule sets rather than one blanket policy.

Serverless Compute Doesn’t Live Where You’d Expect

The instinct with VNet injection is to assume it covers the workspace’s networking story completely. It doesn’t; serverless compute runs outside that boundary entirely, in Microsoft’s own managed environment, which means a securely VNet-injected workspace can still have an ungoverned path between serverless compute and protected storage if nothing is done about it. Recognizing that gap, and scoping an Azure Network Security Perimeter specifically to authorize serverless-to-storage access, meant understanding Databricks’ actual compute architecture deeply enough to know VNet injection wasn’t the whole answer.

CI/CD Can’t Reach What It’s Supposed to Deploy To

The network-isolated environment gave standard GitHub-hosted Actions runners no approved deployment path into the workspace. Building a self-hosted runner inside the trusted network provided a controlled CI/CD path without reopening workspace ingress.

Bridging Two DNS Resolution Domains

Azure and on-premises DNS not knowing about each other’s zones has a native answer, Azure DNS Private Resolver, but standing up dedicated resolver endpoints and subnets was more infrastructure than the requirement justified at this stage, and no one was eager to take that on for what was still a fairly narrow need. Building a lightweight conditional-forwarding service instead, and placing it on the already-necessary self-hosted runner’s host rather than standing up separate dedicated infrastructure for it, kept the solution from adding another isolated piece of infrastructure to maintain, a deliberate scope tradeoff, not a case of Azure having no answer at all.

Learning the Platform Deeply Enough to Secure It

Compared to the Azure networking and identity work this builds on, Databricks itself was newer ground; getting the compliance security profile, Unity Catalog integration, and serverless networking model right meant learning the platform’s actual architecture in enough depth to make informed security decisions, not just follow a generic hardening checklist.

Results

• Rebuilt from an initial deployment that exposed too much network surface into a VNet-injected environment with workspace ingress restricted through IP access controls and Private Link, VNet-level segmentation via NSGs, and Databricks’ compliance security controls on top.
• Attached the new workspace to the organization’s existing regional Unity Catalog metastore rather than standing up an independent governance island, with on-premises systems feeding data into storage governed through Unity Catalog.
• Closed the serverless-compute networking gap that VNet injection alone leaves open, scoping Azure Network Security Perimeter specifically to authorize serverless access to Azure Storage.
• Established scoped, deliberate external connectivity for specific data sources without broadening the environment’s overall network exposure, using a dedicated Azure NAT Gateway per environment (prod, dev, test) to give each VNet a stable, allowlistable static outbound IP.
• Built a self-hosted CI/CD runner and standardized dev/test/prod deployment configuration and service principals, giving a network-isolated environment a working automated deployment path.
• Built (validated, pending production activation) a lightweight conditional-forwarding DNS service, a deliberate scope tradeoff against deploying Azure DNS Private Resolver’s dedicated endpoints and subnets for what was, at this stage, a narrower requirement.
• This project’s networking requirements directly drove the broader hub-spoke and private DNS consolidation work at the platform level.
• Early metrics on migrated workloads are strong: one ETL process improved from roughly six hours to about 45 minutes, an ~87.5% reduction in elapsed processing time. A fully quantified cost comparison isn’t available yet; attributing cost cleanly across databases, storage, Data Factory, and Databricks compute isn’t a solved problem, and the migration itself is still underway with contracted support.

Technologies

Databricks & Data Platform

Azure Databricks, VNet injection, Unity Catalog (regional metastore attachment), Databricks compliance security controls, serverless compute

Networking & Security

Hub-spoke architecture, Azure NAT Gateway, IP access lists, Network Security Groups, Azure Network Security Perimeter, Azure Private Link, Azure Private DNS, Azure DNS Private Resolver (evaluated), VPN-restricted access, scoped external connectivity

Identity

Service principals (jobs and CI/CD, separated per environment)

CI/CD & Automation

GitHub, self-hosted GitHub Actions runner, environment-scoped deployment configuration (dev/test/prod)