Databricks on Azure: enterprise lakehouse architecture

The modern data stack has converged on a single architectural truth: the lakehouse.

Lakehouse Architecture for Petabyte-Scale Analytics

Unity Catalog, Delta Live Tables, ML at Scale & Azure Integration Patterns

Data platform practice  |  Microsoft ecosystem Veratas  |  2025 edition

Classification: Public  |  Version 1.0  |  2025

Executive summary

This white paper addresses the gap between Databricks introductory documentation and the architectural decisions that determine whether a Databricks deployment scales from a proof-of-concept to a petabyte-scale production platform. It is written for data architects, senior data engineers, ML engineers, and technical leaders making platform decisions for organizations processing millions to trillions of records daily.

Strategic insight: Databricks and Microsoft Fabric are not mutually exclusive. The optimal enterprise architecture in 2025 uses both: Databricks for heavy data engineering, ML training, and streaming at scale; Microsoft Fabric for business-facing analytics, Power BI integration, and self-service BI. Unity Catalog bridges both platforms through its Fabric connector, enabling shared governance across the full data estate.

Why Databricks on Azure: Strategic positioning

1.1 The lakehouse market in 2025

The lakehouse market has reached a tipping point. Snowflake, Google BigQuery, and Microsoft Fabric all offer lakehouse-compatible architectures, but Databricks retains three durable advantages that make it the preferred platform for engineering-heavy, ML-intensive, or high-volume data workloads:

Apache Spark ownership: Databricks employs the core Spark committers. Every Databricks Runtime release ships Spark optimizations ahead of open-source availability. For workloads processing terabytes or petabytes per job, this translates to faster execution over commodity Spark environments.

Delta Lake protocol maturity: Delta Lake, created by Databricks, is the most widely adopted open table format. Delta Universal Format (UniForm) enables interoperability with Apache Iceberg and Apache Hudi readers, future-proofing storage investments against format fragmentation.

Unified ML platform: MLflow (also created by Databricks) is the de facto standard for ML experiment tracking and model registry, with integrations in Azure ML, SageMaker, Google Vertex AI, and every major ML framework. No other lakehouse platform matches Databricks’ native integration between data engineering and ML workflows.

1.2 Databricks vs. Microsoft Fabric: Decision framework

The most common architectural question in the Microsoft ecosystem: when to use Databricks, when to use Fabric, and when to use both. The answer is workload-driven:

Workload characteristic	Recommended platform	Rationale
Large-scale Spark ETL	Databricks	Databricks Runtime faster than open-source Spark; better autoscaling for burst workloads
Streaming ingestion at high volume	Databricks (Structured Streaming)	Native Kafka connector, exactly-once semantics, DLT auto-scaling for variable throughput
ML model training (large models)	Databricks	Distributed training (horovod, ray), GPU cluster management, native mLflow
Business analytics & self-service BI	Microsoft Fabric / Power BI	DirectLake mode, Fabric semantic model, M365 integration, non-technical user experience
SQL analytics on small-medium datasets	Microsoft Fabric (warehouse)	Easier SQL authoring, better Power BI integration, lower cost for sQL-only workloads
Enterprise data catalog & governance	Microsoft Purview + Unity Catalog	Purview for estate-wide governance; Unity Catalog for Databricks-specific fine-grained control
Mixed workload (ETL + analytics + ML)	Databricks + Fabric (integrated)	Databricks processes & governs; Fabric serves; Unity Catalog bridges via Fabric connector

1.3 Azure-Native integration advantages

Deploying Databricks on Azure (Azure Databricks) provides integration benefits that are unavailable on other cloud providers:

• Azure Active Directory (Entra ID) as the identity provider, no separate Databricks user management; SSO and conditional access policies apply automatically
• Azure Private Link for Databricks control plane and data plane, full network isolation without public internet exposure
• Unity Catalog integration with Microsoft Purview, bidirectional metadata sync for enterprise-wide governance
• Native connectors to Azure Event Hubs (Kafka protocol), Azure Blob Storage, ADLS Gen2, Azure SQL, Cosmos DB, Azure Synapse, no third-party connector licensing
• Azure Monitor + Log Analytics for unified observability, Databricks cluster logs, job metrics, and SQL warehouse query history in the same observability stack as all Azure resources
• Microsoft Fabric shortcut to Databricks-managed Delta tables, Fabric reports can query Databricks Lakehouse tables directly without data movement

Cost reality check: Azure Databricks pricing combines DBU (Databricks Unit) consumption with underlying Azure VM costs. A common planning mistake is budgeting only DBU costs. Total cost = DBU cost + VM cost + storage cost + networking cost. This white paper’s cost optimization chapter addresses all four components.

CHAPTER 2

Lakehouse architecture: Foundations & design patterns

2.1 The Medallion Architecture

The Medallion Architecture, Bronze, Silver, Gold, is the de facto standard for organizing data in a Databricks Lakehouse. Its value is not the three-tier label but the quality contract each tier enforces, the decoupling it provides between ingestion and consumption, and the lineage clarity it creates for governance teams.

Layer	Data state	Quality contract	Consumers	Storage format
Bronze (raw)	Exact copy of source data; no transformations; append-only	Schema-on-read; no business rules applied; complete audit trail of source data	Data engineers, audit/compliance	Delta (with source schema preserved); partitioned by ingestion date
Silver (curated)	Cleaned, deduplicated, conformed; business rules applied; joins across source systems	Schema enforced; data quality rules pass; sLA-driven freshness; no PII in open columns	Data engineers, data scientists, advanced analysts	Delta; partitioned by business date; z-ordered on high-cardinality query columns
Gold (certified)	Business-ready aggregations, dimensional models, feature stores; domain-specific semantics	Certified by domain owner; SLA guaranteed; documentation complete; sensitivity labelled	Analysts, Power BI / Fabric, ML feature pipelines, APIs	Delta; highly optimized; partitioned and z-ordered for query patterns

Extended medallion: Platinum layer

High-maturity organizations add a Platinum layer: ML feature store output, aggregated serving tables with sub-second query SLAs, or API-serving materialized views. Platinum tables are the most heavily governed, most tightly SLA-bound, and most aggressively optimized assets in the lakehouse.

2.2 Storage organization patterns

The physical organization of the lakehouse in ADLS Gen2 has major implications for performance, security, cost, and governance. Two primary patterns:

Pattern 1: Zone-Isolated storage accounts (Recommended for enterprise)

Separate ADLS Gen2 storage accounts per medallion layer: adls-bronze-prod, adls-silver-prod, adls-gold-prod. Advantages: independent encryption keys per zone, network security group rules per zone, independent soft-delete and versioning policies, clear billing separation. Disadvantage: slightly more complex pipeline configuration.

Pattern 2: Container-Isolated single account

Single ADLS Gen2 account with containers per layer: /bronze, /silver, /gold. Advantages: simpler configuration, single place for RBAC. Disadvantage: shared encryption key, harder to enforce zone-level network policies, difficult to separate storage costs by layer.

Recommendation: Use zone-isolated storage accounts for any deployment handling regulated data (PCI, PHI, PII at scale). The per-key encryption enables different key rotation policies per zone and makes regulatory scoping unambiguous. For development environments, a single account with containers is sufficient.

2.3 Workspace architecture for enterprise

A Databricks workspace is the primary deployment unit, the boundary for compute resources, notebooks, jobs, and most RBAC. Enterprise deployments require deliberate workspace strategy:

Workspace pattern	Use case	Advantages	Disadvantages
Single workspace	Small teams, single project, low sensitivity	Simple administration, low overhead	No environment isolation; production/dev share resources and governance
Environment-separated (dev/test/prod)	Most enterprise deployments	Environment isolation; CI/CD promotion; prod cluster policies enforced separately	Data sharing between envs requires Delta Sharing or data copies
Domain-separated (data mesh)	Large orgs with multiple data domains	Domain autonomy; per-domain billing; independent upgrade cycles	Unity Catalog metastore still shared; requires cross-workspace governance coordination
Hub-and-spoke (recommended)	Enterprise with shared platform team + domain teams	Shared infra (hub) + domain autonomy (spokes); central governance + local engineering	Most complex to set up; requires strong platform team

2.4 Cluster architecture

Cluster selection is the single biggest driver of both performance and cost. Databricks offers three compute types with fundamentally different operational models:

All-Purpose Clusters: Interactive, long-lived clusters for notebook development and ad hoc analysis. Charged continuously while running. Never use all-purpose clusters for production jobs, auto-termination misconfiguration is the single most common source of unexpected Databricks cost overruns. Enforce auto-termination of 30 minutes via cluster policy.

Job Clusters: Ephemeral clusters created at job start and terminated at job end. The correct compute type for all production jobs. Cost = compute time only. Cluster start time is the only overhead. Use cluster pools to pre-warm instances and reduce start latency.

SQL Warehouses (Serverless preferred): Purpose-built for SQL analytics workloads. Classic SQL Warehouses run on customer-managed VMs. Serverless SQL Warehouses run on Databricks-managed compute and typically start in 2 to 6 seconds. Use serverless for all interactive BI and analyst SQL workloads. Classic warehouses only when data residency requirements prohibit serverless.

Cluster type	Start time	Cost model	Best for	Avoid for
All-Purpose	Cold start	Per-second while running	Interactive development, notebooks	Production jobs (cost risk)
Job cluster	Cold start	Per-second for job duration only	All production batch/streaming jobs	Interactive exploration
Job cluster + pool	Faster (warm pool)	Pool idle cost + per-second for job	Latency-sensitive production jobs	Infrequent jobs (pool idle cost not worth it)
Serverless SQL warehouse	2 to 6 sec (typical)	Per-second, serverless pricing	BI analytics, analyst SQL, Power BI directQuery	Large Spark batch processing
Classic SQL warehouse	About 4 min (typical)	Per-second (customer VM)	Data residency-constrained SQL analytics	General use (serverless preferred)

CHAPTER 3

Delta Lake: The storage layer deep dive

3.1 Delta Lake architecture

Delta Lake is not merely a file format, it is a transactional storage layer that adds ACID guarantees, scalable metadata management, and time travel to Parquet files stored in cloud object storage. Understanding its internals is essential for designing high-performance, cost-efficient lakehouse pipelines.

Every Delta table consists of: a data directory (Parquet files organized by partition), a transaction log directory (_delta_log/) containing JSON commit files, and periodically generated Parquet checkpoint files that compact the JSON transaction log for fast metadata reads.

Delta capability	Technical mechanism	Operational significance
ACID transactions	Optimistic concurrency control via transaction log; writers serialize via log entry; readers see consistent snapshots	Multiple concurrent writers without data corruption; safe concurrent reads during writes
Time travel	Transaction log preserves all historical versions; TIMESTAMP AS OF or VERSION AS OF syntax	Audit compliance; rollback from bad pipeline runs; reproducible ML training datasets
Schema enforcement	Writer validates incoming data against table schema; rejects non-conforming writes by default	Prevents silent data corruption from schema drift in upstream sources
Schema evolution	MERGE INTO, ALTER TABLE ADD COLUMN, or mergeSchema option	Safely add new columns without breaking existing readers
Scalable metadata	Parquet checkpoints compact JSON log files; column stats stored in log for data skipping	Tables with very large file counts read metadata quickly
Data skipping	Min/max column statistics per file stored in transaction log; queries skip files outside predicate range	Faster filtered queries without full table scan
Z-ordering	Co-locates related data in same files based on specified columns; improves data skipping effectiveness	Especially effective for high-cardinality columns (customer_id, product_id) used in WHERE clauses

3.2 Delta table optimization: Production patterns

OPTIMIZE and Z-ORDER

— Run after bulk loads or when small files accumulate

— Schedule as a post-load job, not inline in the pipeline

— Basic OPTIMIZE (compacts small files; autotuned target is 256MB to 1GB, by table size)

OPTIMIZE silver.sales_transactions;

— OPTIMIZE with Z-ORDER (co-locates data by query columns)

— Z-ORDER columns: choose a few columns used in WHERE/JOIN predicates (effectiveness drops with each extra column)

— Do NOT Z-ORDER on partition columns (already co-located by partition)

— Do NOT Z-ORDER on columns with low cardinality

OPTIMIZE silver.sales_transactions

ZORDER BY (customer_id, product_id);

— Predictive optimization (DBR 12.2+), let Databricks decide when to OPTIMIZE

— Enable at catalog or schema level (Unity Catalog required):

ALTER CATALOG main SET DBPROPERTIES (delta.enablePredictiveOptimization = TRUE);

— Predictive optimization monitors file stats and triggers OPTIMIZE automatically

— Eliminates need for scheduled OPTIMIZE jobs in most cases

Partitioning strategy

— Partitioning rules:

— Partition only on LOW cardinality columns (date, region, status)

— Target partition size: at least 1GB of data per partition

— NEVER partition on high-cardinality columns (customer_id, order_id)

— High-cardinality partitioning = millions of tiny files = metadata explosion

— Good: partition by date (daily batches ~1-10GB per day)

CREATE TABLE silver.sales_transactions (

transaction_id STRING,

customer_id    STRING,

product_id     STRING,

amount         DECIMAL(18,2),

transaction_ts TIMESTAMP,

transaction_dt DATE

) USING DELTA

PARTITIONED BY (transaction_dt)

LOCATION “abfss://silver@storage.dfs.core.windows.net/sales/transactions/”;

— After partitioning by date, Z-ORDER by customer_id and product_id

— to optimize point lookups within each date partition

OPTIMIZE silver.sales_transactions

WHERE transaction_dt >= current_date() – INTERVAL 7 DAYS

ZORDER BY (customer_id, product_id);

3.3 Change data capture with Delta

Delta’s Change Data Feed (CDF) is the recommended pattern for incremental processing pipelines that need to propagate changes downstream without full table scans. CDF captures insert, update, and delete operations at the row level with change type metadata.

— Enable change data feed on a Delta table

ALTER TABLE silver.customers

SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

— Read changes since a specific version or timestamp

val changes = spark.read

.format(“delta”)

.option(“readChangeFeed”, “true”)

.option(“startingVersion”, lastProcessedVersion)  // store in metadata table

.table(“silver.customers”)

— _change_type values: insert, update_preimage, update_postimage, delete

// Process only inserts and updates (ignore preimage rows)

val upserts = changes.filter(“_change_type IN (‘insert’, ‘update_postimage’)”)

— Apply to downstream gold table using MERGE

deltaTable.alias(“target”)

.merge(upserts.alias(“source”), “target.customer_id = source.customer_id”)

.whenMatchedUpdateAll()

.whenNotMatchedInsertAll()

.execute()

— Store the last processed version for idempotency

spark.sql(s”UPDATE pipeline_metadata.watermarks SET last_version = ${changes.select(max(‘_commit_version’)).collect()(0)(0)} WHERE table_name = ‘silver.customers'”)

3.4 Time travel & data retention

Operation	Syntax	Use case	Retention constraint
Query historical version	SELECT * FROM table VERSION AS OF 42	Debug pipeline issues; compare before/after transformation	Only works within retention window
Query by timestamp	SELECT * FROM table TIMESTAMP AS OF ‘2025-01-15’	Reproduce ML training dataset as of a specific date	Timestamp must be within retention window
Restore table	RESTORE TABLE silver.customers TO VERSION AS OF 38	Rollback bad pipeline run or accidental delete	Restores table state; does not delete newer log entries
View table history	DESCRIBE HISTORY silver.customers	Audit all operations; find version before bad write	Shows full transaction log within retention window

Default retention is 30 days for table history and 7 days for deleted data files, so time travel is only reliable for 7 days. For regulatory environments requiring longer audit trails, set: ALTER TABLE silver.customers SET TBLPROPERTIES (‘delta.logRetentionDuration’ = ‘interval 365 days’, ‘delta.deletedFileRetentionDuration’ = ‘interval 365 days’);

Unity Catalog: Enterprise data governance

4.1 Unity Catalog architecture

Unity Catalog (UC) is Databricks’ unified governance layer, the control plane for data access, audit logging, lineage tracking, and data sharing across all workspaces in an organization. It is the most significant governance advancement in Databricks’ history, replacing the legacy per-workspace Hive metastore with a centrally managed, ANSI-SQL-compatible catalog with fine-grained access control.

The UC object hierarchy determines how permissions propagate and how assets are organized:

Object level	Example	Grants propagate to	Primary governance action
Metastore	Contoso-AUED-Metastore (one per Azure region)	All catalogs, schemas, tables below	Assign metastore admins; attach to workspaces; configure audit log destination
Catalog	main, dev_catalog, finance_catalog	All schemas and tables within catalog	Assign catalog owners; set default privileges; configure storage credentials
Schema (database)	main.silver, finance_catalog.reporting	All tables, views, functions within schema	Assign schema owners; set managed location; grant domain team access
Table / view	main.silver.sales_transactions	Columns (if column mask applied)	Assign owners; grant SELECT/MODIFY; apply row filters; apply column masks
Column	sales_transactions.customer_email	N/A (leaf node)	Apply column masks (e.g., mask PII for non-privileged users)
Function	main.silver.calculate_ltv()	N/A	Control which users can call UDFs and table-valued functions
Storage credential	adls-silver-credential	External locations using this credential	Grant to specific workspaces or service principals; never grant to end users

4.2 Privilege model & RBAC design

Unity Catalog uses a privilege model inspired by ANSI SQL GRANT/REVOKE but extended for cloud-native data access patterns. Understanding privilege inheritance prevents common over-permissioning mistakes.

Privilege inheritance rules

• Privileges granted at a higher level (catalog) are inherited by all objects below (schemas, tables, columns)
• DENY is not supported, UC uses positive permissions only; to restrict access, do not grant at that level
• Table owner automatically has all privileges on that table; schema owner has all privileges on all tables in the schema
• USE CATALOG and USE SCHEMA are “traversal” privileges, required to access any object within, but do not grant read access to data
• SELECT on a table requires: USE CATALOG on the parent catalog AND USE SCHEMA on the parent schema AND SELECT on the table, all three must be explicitly granted

Recommended RBAC pattern

— Unity Catalog RBAC: Principle of least privilege

— Use Entra ID (AAD) groups, not individual users

— Tier 1: Read-only analysts (most common role)

GRANT USE CATALOG ON CATALOG main TO `data-analysts@contoso.com`;

GRANT USE SCHEMA ON SCHEMA main.gold TO `data-analysts@contoso.com`;

GRANT SELECT ON ALL TABLES IN SCHEMA main.gold TO `data-analysts@contoso.com`;

— Note: analysts CAN see gold data but cannot see silver or bronze

— Tier 2: Data engineers (read silver/bronze, write gold)

GRANT USE CATALOG ON CATALOG main TO `data-engineers@contoso.com`;

GRANT USE SCHEMA ON SCHEMA main.bronze TO `data-engineers@contoso.com`;

GRANT USE SCHEMA ON SCHEMA main.silver TO `data-engineers@contoso.com`;

GRANT USE SCHEMA ON SCHEMA main.gold TO `data-engineers@contoso.com`;

GRANT SELECT ON ALL TABLES IN SCHEMA main.bronze TO `data-engineers@contoso.com`;

GRANT SELECT ON ALL TABLES IN SCHEMA main.silver TO `data-engineers@contoso.com`;

GRANT SELECT, MODIFY ON ALL TABLES IN SCHEMA main.gold TO `data-engineers@contoso.com`;

— Tier 3: Service principals for production pipelines

— Use service principals (not groups) for job clusters

GRANT USE CATALOG, USE SCHEMA ON CATALOG main TO `sp-pipeline-prod`;

GRANT SELECT ON SCHEMA main.silver TO `sp-pipeline-prod`;

GRANT SELECT, MODIFY ON SCHEMA main.gold TO `sp-pipeline-prod`;

— Principle: pipelines have exactly the permissions needed for their specific tables

4.3 Column masks & row filters

Column masks and row filters are the most powerful and underutilized Unity Catalog features. They implement attribute-based access control (ABAC) at the data level, the same data asset serves different data consumers with different visibility based on their group membership or attributes, without duplicating data or maintaining separate views.

— Column mask: Mask email addresses for non-privileged users

— Users in “data-scientists-privileged” group see real email

— All other users see masked version

CREATE FUNCTION main.security.mask_email(email STRING)

RETURNS STRING

RETURN CASE

WHEN is_member(‘data-scientists-privileged’) THEN email

ELSE CONCAT(LEFT(email, 2), ‘***@***.com’)

END;

ALTER TABLE main.silver.customers

ALTER COLUMN customer_email

SET MASK main.security.mask_email;

— Row filter: Data scientists only see rows for their assigned region

— Useful for multi-tenant datasets, regional data residency, or cost centre allocation

CREATE FUNCTION main.security.filter_by_region(region STRING)

RETURNS BOOLEAN

RETURN is_member(‘global-data-access’)    — global team sees all

OR is_member(CONCAT(‘region-‘, region)); — regional team sees their region

ALTER TABLE main.silver.sales_transactions

SET ROW FILTER main.security.filter_by_region ON (region);

— Verify mask/filter is working (run as non-privileged user):

SELECT customer_id, customer_email FROM main.silver.customers LIMIT 5;

— Should return: “jo***@***.com” style masked emails

4.4 Automated lineage & data discovery

Unity Catalog automatically captures lineage for all operations executed through Databricks, SQL queries, Spark DataFrame transformations, Python, R, and Scala notebooks, without any instrumentation code. Lineage is captured at the column level and stored in the UC metastore, queryable via the Databricks REST API or viewable in the Catalog Explorer UI.

Lineage from Unity Catalog is also exported to Microsoft Purview via the native connector, creating a unified lineage graph that spans Databricks (engineering/ML) and the broader Microsoft data estate (Fabric, ADF, Power BI). This is the recommended governance architecture for organizations using both platforms.

Lineage source	Granularity	Capture method	Purview export
Spark SQL (notebooks, jobs)	Column-level	Automatic (UC query analysis)	Yes, via Purview-UC connector
Python dataFrame API	Table-level	Automatic (I/O operation capture)	Yes
Delta Live Tables pipelines	Column-level (full DAG)	Automatic (DLT execution graph)	Yes
ML training runs (mLflow)	Dataset-to-model	Automatic (mLflow UC integration)	Yes (model entity in Purview)
dbt on Databricks	Column-level (dbt metadata)	Via dbt-databricks adapter + UC integration	Yes (via dbt-purview connector)

Delta Live Tables: Declarative pipeline engineering

5.1 DLT architecture & value proposition

Delta Live Tables (DLT) is Databricks’ declarative pipeline framework that manages the full lifecycle of data pipelines: execution ordering, dependency resolution, error handling, retry logic, data quality enforcement, autoscaling, and monitoring. Instead of writing orchestration logic, engineers declare what data should look like, DLT determines how to produce it.

The productivity gain is substantial. A pipeline that required many lines of Spark code with custom error handling, dependency management, and retry logic can be expressed in far fewer lines of DLT SQL or Python. More importantly, the operational properties (idempotency, exactly-once semantics, automatic backfill) are guaranteed by the framework, not the engineer.

5.2 DLT pipeline design patterns

Pattern 1: Full medallion DLT pipeline (SQL)

— Bronze: Ingest raw events from Event Hubs (streaming)

CREATE OR REFRESH STREAMING LIVE TABLE bronze_events

COMMENT “raw events from IoT sensors via Event Hubs. schema preserved exactly.”

AS SELECT * FROM cloud_files(

“abfss://landing@storage.dfs.core.windows.net/events/”,

“json”,

map(“cloudFiles.inferColumnTypes”, “true”,

“cloudFiles.schemaLocation”, “/checkpoints/bronze_events/schema”)

);

— Silver: Clean, validate, and enrich (streaming)

CREATE OR REFRESH STREAMING LIVE TABLE silver_events (

CONSTRAINT valid_device_id EXPECT (device_id IS NOT NULL) ON VIOLATION DROP ROW,

CONSTRAINT valid_temperature EXPECT (temperature BETWEEN -50 AND 150) ON VIOLATION QUARANTINE,

CONSTRAINT valid_timestamp EXPECT (event_ts > ‘2020-01-01’) ON VIOLATION FAIL UPDATE

)

COMMENT “validated and enriched sensor events. Quarantined rows in silver_events_quarantine.”

AS SELECT

e.device_id,

e.temperature,

e.humidity,

e.event_ts,

d.device_name,

d.location,

d.manufacturer

FROM STREAM(LIVE.bronze_events) e

LEFT JOIN LIVE.dim_devices d ON e.device_id = d.device_id;

— Gold: Business aggregation (batch, refreshed every 15 minutes)

CREATE OR REFRESH LIVE TABLE gold_device_hourly_stats

COMMENT “hourly aggregated stats per device. Power BI source.”

AS SELECT

device_id,

location,

date_trunc(‘HOUR’, event_ts) AS hour,

avg(temperature) AS avg_temp,

max(temperature) AS max_temp,

min(temperature) AS min_temp,

avg(humidity) AS avg_humidity,

count(*) AS event_count

FROM LIVE.silver_events

GROUP BY device_id, location, date_trunc(‘HOUR’, event_ts);

5.3 Data quality with expectations

DLT Expectations are the single most important feature for production pipeline reliability. They formalize data quality rules as executable constraints, replacing ad hoc validation notebooks and undocumented “it looks wrong” discoveries.

Expectation violation mode	Behaviour	Use when	Monitoring
ON VIOLATION DROP ROW	Invalid rows are silently dropped; valid rows continue processing	Noise in source data is expected and acceptable; business logic tolerates missing records	Metrics track dropped row count; alert if drop rate exceeds threshold
ON VIOLATION QUARANTINE	Invalid rows routed to _quarantine table; valid rows continue	Invalid rows need investigation without blocking pipeline; compliance requires audit of rejected data	Query _quarantine table; alert on non-zero counts for critical constraints
ON VIOLATION FAIL UPDATE	Pipeline fails if constraint violated; no partial writes	Zero tolerance for invalid data; downstream consumers cannot handle bad data	Pipeline failure triggers alert; requires manual investigation before rerun

5.4 DLT enhanced autoscaling

DLT Enhanced Autoscaling is the recommended compute mode for streaming pipelines with variable throughput. Unlike standard Databricks autoscaling (which scales based on task queue depth), DLT Enhanced Autoscaling uses pipeline-level metrics, lag behind source, processing rate, memory pressure, to make scaling decisions optimized for streaming throughput.

// DLT pipeline configuration (JSON) for enhanced autoscaling

{

“name”: “iot-sensor-pipeline”,

“clusters”: [{

“label”: “default”,

“autoscale”: {

“min_workers”: 2,

“max_workers”: 20,

“mode”: “ENHANCED”  // key: use ENHANCED not LEGACY

},

“node_type_id”: “Standard_DS4_v2”,

“spark_conf”: {

“spark.databricks.delta.optimizeWrite.enabled”: “true”,

“spark.databricks.delta.autoCompact.enabled”: “true”

}

}],

“continuous”: true,   // for streaming: always-on pipeline

// “development”: false, production mode: no interactive restarts

“channel”: “CURRENT”, // production: always use CURRENT, not PREVIEW

“edition”: “ADVANCED” // required for enhanced autoscaling + expectations

}

Machine learning at scale: MLflow & model serving

6.1 The Databricks ML platform

Databricks’ ML platform integrates three components into a unified workflow: the Databricks Runtime for ML (pre-installed with TensorFlow, PyTorch, scikit-learn, XGBoost, and many other ML libraries), MLflow for experiment tracking and model lifecycle management, and Model Serving for production model deployment. Together, they shorten the time from experiment to production.

6.2 MLflow: Experiment tracking & model registry

import mlflow

import mlflow.sklearn

from sklearn.ensemble import gradientBoostingClassifier

from sklearn.metrics import roc_auc_score

# Set experiment (creates if not exists; tracked in Unity Catalog)

mlflow.set_experiment(“/Shared/churn-prediction/experiments/gbm-v2”)

with mlflow.start_run(run_name=”gbm-depth6-lr0.05″) as run:

# Log hyperparameters

params = {“n_estimators”: 500, “max_depth”: 6, “learning_rate”: 0.05}

mlflow.log_params(params)

# Train model

model = gradientBoostingClassifier(**params)

model.fit(X_train, y_train)

# Log metrics

auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])

mlflow.log_metric(“test_auc”, auc)

mlflow.log_metric(“train_auc”, roc_auc_score(y_train, model.predict_proba(X_train)[:, 1]))

# Log model with input example and signature (required for model serving)

signature = mlflow.models.infer_signature(X_train, model.predict(X_train))

mlflow.sklearn.log_model(

model, “model”,

signature=signature,

input_example=X_train[:5],

registered_model_name=”main.ml_models.churn_prediction”  # uC-registered

)

print(f”Run ID: {run.info.run_id} | AUC: {auc:.4f}”)

6.3 Model registry & deployment lifecycle

Stage	Registry status	Who transitions	Automated gates
Development	None (experiment runs)	Data scientist	N/A, free experimentation
Candidate	Registered (no alias)	Data scientist	Auto: unit tests, schema validation, bias metrics check
Staging	@staging alias	ML engineer	Auto: integration test, shadow mode evaluation against production traffic, data drift check
Production	@production alias	ML lead (approval gate)	Auto: A/B test gate, latency SLA check, manual sign-off
Archived	@archived alias	Automated (on new production)	Retain for 90 days for rollback capability; auto-delete after retention window

6.4 Model serving: Production deployment

Databricks Model Serving provides serverless, auto-scaling REST endpoints for ML models registered in Unity Catalog. The endpoint handles traffic routing, A/B testing, canary deployments, and feature lookup, eliminating the need for a separate model serving infrastructure.

# Deploy model to Databricks model serving via REST API

# (also configurable via UI or Terraform)

import requests

import json

token = dbutils.secrets.get(scope=”ml-secrets”, key=”databricks-token”)

workspace_url = “https://adb-<workspace-id>.azuredatabricks.net”

endpoint_config = {

“name”: “churn-prediction-v2”,

“config”: {

“served_models”: [

{

“name”: “churn-gbm-production”,

“model_name”: “main.ml_models.churn_prediction”,

“model_version”: “3”,

“workload_size”: “Small”,   # small/Medium/Large (CPU); GPU options available

“scale_to_zero_enabled”: True,  # cost saving for non-critical endpoints

}

],

“traffic_config”: {

“routes”: [{“served_model_name”: “churn-gbm-production”, “traffic_percentage”: 100}]

}

}

}

response = requests.post(

f”{workspace_url}/api/2.0/serving-endpoints”,

headers={“Authorization”: f”Bearer {token}”},

json=endpoint_config

)

6.5 Feature store

The Databricks Feature Store (now Unity Catalog Feature Store) solves the training-serving skew problem, the most common source of ML model degradation in production. Features computed during training are stored in Unity Catalog, and the same feature values are retrieved at inference time, guaranteeing consistency.

Key capability: Point-in-time lookups. The Feature Store retrieves the feature value that was valid at the timestamp of each training record, not the current value. This is critical for time-series problems (churn prediction, fraud detection, demand forecasting) where using future feature values in training creates data leakage.

CHAPTER 7

Azure integration architecture

7.1 Network architecture

Production Databricks deployments on Azure require deliberate network design. The default “no VNet injection” configuration routes all cluster traffic through public endpoints, acceptable for development, unacceptable for regulated data workloads.

VNet injection (Required for production)

VNet injection deploys Databricks cluster VMs into a customer-managed Azure Virtual Network. Both the control plane (Databricks-managed) and data plane (cluster VMs) communicate through private endpoints. This architecture enables:

• Databricks clusters that cannot be reached from the public internet
• Outbound traffic routed through customer-managed Azure Firewall or NVA with full packet inspection
• Private endpoint connectivity from clusters to ADLS Gen2, Azure Key Vault, Azure SQL, Event Hubs, and all other Azure PaaS services
• NSG rules that restrict lateral movement between cluster nodes and to/from non-approved destinations

Network component	Configuration	Purpose
VNet	Dedicated /16 or /17 (e.g., 10.10.0.0/16)	Container for all Databricks network resources
Public subnet	/17 or /18; NSG with Databricks service tag rules	Cluster driver nodes; required by Databricks (even in “no public IP” mode)
Private subnet	/17 or /18; NSG with Databricks service tag rules	Cluster worker nodes; no public IP; data plane traffic
Azure private endpoints	One per PaaS service (ADLS, Key Vault, ACR, etc.)	Private IP access to Azure services; eliminates data exfiltration via public endpoints
Azure Firewall / NVA	Custom route table (0.0.0.0/0 → firewall)	Inspect and log all outbound traffic; allow only approved destinations
Private DNS zones	One per private endpoint service type	Resolve Azure service hostnames to private IPs within vNet

7.2 Identity integration with Azure active directory

Azure Databricks uses Azure Active Directory (Entra ID) as the exclusive identity provider when Unity Catalog is enabled. This eliminates Databricks-local user management and enables:

• Single Sign-On via Entra ID SAML 2.0, users authenticate with their corporate credentials
• Conditional Access Policies, MFA requirements, device compliance, IP location restrictions apply to Databricks access
• Group synchronization via SCIM, Entra ID security groups sync to Databricks automatically; deprovisioning in Entra ID immediately revokes Databricks access
• Service principal authentication, pipelines and automation use managed identities or service principals with federated credentials; no secrets in code

7.3 Key Azure integration patterns

Azure service	Integration pattern	Authentication	Use case
ADLS Gen2	Direct mount via Unity Catalog external location; no storage credentials in notebooks	Managed identity on cluster; or service principal via Unity Catalog storage credential	All Bronze/Silver/Gold storage; primary lakehouse storage layer
Azure Event Hubs	Kafka-compatible connector (eventHubsConf); or Azure Event Hubs Spark connector	SAS token via Key Vault secret; or managed identity (recommended)	Streaming ingestion into Bronze tables via DLT or Structured Streaming
Azure Key Vault	Databricks secret scope backed by Azure Key Vault	Managed identity; no AKV credentials stored in Databricks	All secrets (connection strings, API keys, tokens); never hardcode credentials
Azure SQL Database	JDBC connector; or recommended: Data Factory staging to Delta	Service principal with SQL login; or managed identity (SQL MI)	Incremental extraction from operational databases into Bronze
Azure Data Factory	ADF Databricks notebook/Jar/Python activities	Service principal with contributor role on workspace	Orchestrate Databricks jobs from ADF parent pipelines; cross-system dependencies
Microsoft Fabric	Fabric shortcut to Delta tables in ADLS Gen2; or Delta Sharing protocol	Entra ID passthrough; or Delta Sharing credential	Serve Databricks gold tables in Fabric/Power BI without data movement
Azure Monitor	Cluster log delivery to Log Analytics via diagnostic settings	Workspace MSI; no configuration required beyond enabling	Unified observability; query Databricks logs alongside all Azure resources

7.4 Secrets management

# NEVER do this:

# connection_string = “server=myserver.database.windows.net;User=admin;Password=P@ssw0rd!”

# ALWAYS do this: Key Vault-backed secret scope

# Step 1: Create Key Vault-backed secret scope (one-time, via Databricks CLI)

# databricks secrets create-scope \

#   –scope “production-secrets” \

#   –scope-backend-type AZURE_KEYVAULT \

#   –resource-id “/subscriptions/…/vaults/contoso-kv-prod” \

#   –dns-name “https://contoso-kv-prod.vault.azure.net/”

# Step 2: Read secret in notebook/job (secret never exposed in plain text)

jdbc_password = dbutils.secrets.get(scope=”production-secrets”, key=”sql-db-password”)

# The value of jdbc_password is REDACTED in all Databricks logs

# print(jdbc_password) → [REDACTED]

# Step 3: Use in connection

df = spark.read.format(“jdbc”) \

.option(“url”, f”jdbc:sqlserver://server.database.windows.net”) \

.option(“dbtable”, “dbo.orders”) \

.option(“user”, dbutils.secrets.get(“production-secrets”, “sql-db-user”)) \

.option(“password”, jdbc_password) \

.load()

Security architecture & zero-trust design

8.1 Security layers

A production Databricks security architecture operates across five layers. Each layer is necessary; none is sufficient alone:

Security layer	Controls	Databricks mechanism
Identity & authentication	SSO, MFA, conditional access, SCIM provisioning, service principal governance	Entra ID integration, Unity Catalog identity management
Network	VNet injection, private endpoints, NSG rules, firewall egress control, no-public-iP clusters	Azure vNet injection, private link, Azure Firewall integration
Data access	Table/column/row-level access control, column masks, row filters, data masking	Unity Catalog RBAC, column masks, row filters
Compute	Cluster policies, single-user access mode, no SSH access, immutable cluster configuration	Cluster access control, cluster policies, admin restrictions
Audit & monitoring	All data access logged, admin actions logged, anomaly detection, retention compliance	Unity Catalog audit logs, Azure Monitor, Databricks audit log delivery

8.2 Cluster access modes

Cluster access mode is one of the most critical security decisions in Databricks. The access mode determines who can use the cluster and what data they can access, a misconfigured access mode is the most common source of unintended data exposure in Databricks deployments.

Access mode		Who can use	Unity Catalog support	Use case	Recommended
Single user		One user or service principal only	Full support (all features)	Production jobs, sensitive ML workloads, privileged analyst access	Yes, for all production jobs
Shared		Multiple users, process isolation via kernel namespacing	Full support	Shared interactive clusters for teams with different permissions	Yes, for multi-user interactive
No isolation shared (legacy)		Multiple users, NO process isolation	Limited (no row filters, column masks)	Legacy workloads only	No, avoid for any new deployment
Custom (legacy)		Databricks-local users/groups only	Not supported	Pre-UC legacy workloads	No, migrate to Unity Catalog
	Security anti-pattern: Many organizations set cluster access mode to “no isolation shared” because it was the historical default and “things just work.” this mode provides zero process isolation between users, a notebook run by user A can access the memory, temp files, and Spark state of user B running concurrently on the same cluster. Never use no isolation shared mode in production environments handling sensitive data.

8.3 Cluster policies

Cluster policies enforce governance controls on cluster creation, preventing cost overruns, security misconfigurations, and unapproved software. In enterprise deployments, cluster policies are mandatory: without them, any user with CREATE CLUSTER permission can launch arbitrarily large clusters with any configuration.

// Cluster policy JSON: Production job clusters

// Enforce: spot instances, auto-termination, approved node types, no SSH

{

“cluster_type”: {“type”: “fixed”, “value”: “job”},  // jobs only, no all-purpose

“autotermination_minutes”: {

“type”: “range”, “minValue”: 10, “maxValue”: 30, “defaultValue”: 20

},

“node_type_id”: {

“type”: “allowlist”,

“values”: [“Standard_DS4_v2”, “standard_DS5_v2”, “standard_E8s_v3”, “standard_E16s_v3”]

},

“azure_attributes.availability”: {

“type”: “fixed”, “value”: “SPOT_WITH_FALLBACK_AZURE”  // spot instances required

},

“spark_conf.spark.databricks.cluster.profile”: {

“type”: “fixed”, “value”: “singleNode”,

“hidden”: true  // users cannot override

},

“ssh_public_keys”: {“type”: “fixed”, “value”: []},  // no SSH access

“data_security_mode”: {

“type”: “fixed”, “value”: “SINGLE_USER”  // enforce single-user access mode

},

“runtime_engine”: {“type”: “fixed”, “value”: “PHOTON”}  // require Photon for performance

}

Performance engineering & cost optimization

9.1 Photon Engine

Photon is Databricks’ C++ vectorized query engine that replaces the JVM-based Spark execution engine for SQL and DataFrame operations. Photon delivers query speedups for analytical workloads with no code changes required, it is drop-in compatible with any Spark SQL or DataFrame API code.

Workload type	Recommendation
Aggregations (GROUP BY, COUNT, SUM)	Enable Photon on all analytics clusters; cost of DBU premium pays back immediately
Hash joins on large tables	Critical for gold layer transformations with multi-table joins
Sorting and ordering	Especially impactful for OPTIMIZE Z-ORDER operations
Data scans with filters (WHERE clauses)	Combined with Delta data skipping, Photon makes scans of large tables feel like point lookups
Streaming (Structured Streaming)	Enable on DLT clusters; reduces streaming lag during throughput spikes
Complex Python UDFs	Python UDFs bypass Photon; rewrite as SQL expressions or pandas UDFs for Photon benefit

9.2 Cost optimization: The four levers

Lever 1: Spot instances

Spot (preemptible) instances offer significant VM cost savings compared to on-demand pricing. Databricks handles spot instance interruptions transparently: if a worker is preempted, the task is automatically retried on a replacement worker. The job runs longer but at a lower compute cost.

Use SPOT_WITH_FALLBACK_AZURE in cluster policies: spot instances preferred, falls back to on-demand if no spot capacity.

Lever 2: Cluster right-sizing

// Finding the right cluster size: use ganglia metrics or Databricks compute UI

//

// Symptoms of over-provisioned cluster:

// – Executor CPU utilization low during active processing

// – Memory utilization low (driver and executors)

// – Shuffle spill to disk = 0 (could handle smaller workers)

//

// Symptoms of under-provisioned cluster:

// – Executor CPU utilization high and sustained

// – Shuffle spill to disk > 0 (increase worker count or use memory-optimized VMs)

// – Task time variance high (some tasks far slower = data skew; fix in code, not hardware)

//

// For memory-intensive operations (ML, wide joins): use e-series (memory-optimized)

// For compute-intensive operations (Photon SQL, streaming): use d-series

//

// Enable cluster autoscaling for variable workloads:

// min_workers = minimum needed for SLA at lowest throughput

// max_workers = maximum needed at peak throughput

// autoscale.mode = “ENHANCED” for DLT, “LEGACY” for general clusters

Lever 3: Serverless compute

Databricks Serverless SQL Warehouses and Serverless Job Compute eliminate the cluster start time and idle cost problem entirely. With serverless, SQL warehouses typically start in 2 to 6 seconds, compute scales to zero when not in use, and charges only for active query time. For workloads with unpredictable or bursty patterns, serverless is almost always the lowest total cost option.

Lever 4: Storage optimization

Storage optimization	Implementation	Cost impact
ADLS Gen2 tiering	Move Bronze data older than 90 days to cool tier; archive after 1 year	Lower Bronze storage cost (higher access cost)
Delta VACUUM	Run VACUUM weekly on all Delta tables, with delta.deletedFileRetentionDuration set to 30 days	Removes old Parquet files from deleted/updated rows; reduces storage on high-churn tables
Compression tuning	Use snappy for hot data (faster decompress); zstd for cold data (better ratio)	Storage reduction with no query performance impact for aDLS-backed tables
Predictive optimization	Enable UC predictive optimization to auto-manage OPTIMIZE and VACUUM	Reduces over-optimizing small tables; reduces VACUUM not being run; saves engineering time

9.3 Handling data skew

Data skew, where some partition keys have vastly more rows than others, is the most common cause of mysterious Spark job slowdowns that more hardware cannot fix. Identifying and resolving skew is a core Databricks engineering competency.

# Identify skew: look for tasks taking far longer than the median task

# In Spark UI: Stage detail → task metrics → sort by duration descending

# Solution 1: Salting (for joins with skewed keys)

# Add random salt to skewed key to distribute across multiple partitions

from pyspark.sql import functions as F

SALT_FACTOR = 20  # tune based on skew severity; more salt = more parallelism

# Salt the large (skewed) table

large_df_salted = large_df.withColumn(

“customer_id_salted”,

F.concat(F.col(“customer_id”), F.lit(“_”), (F.rand() * SALT_FACTOR).cast(“int”))

)

# Explode the small (lookup) table to match salt values

small_df_exploded = small_df.crossJoin(

spark.range(SALT_FACTOR).select(F.col(“id”).cast(“string”).alias(“salt”))

).withColumn(

“customer_id_salted”,

F.concat(F.col(“customer_id”), F.lit(“_”), F.col(“salt”))

)

# Join on salted key

result = large_df_salted.join(small_df_exploded, “customer_id_salted”)

# Solution 2: Adaptive query execution (AQE), enable for automatic skew handling

spark.conf.set(“spark.sql.adaptive.enabled”, “true”)

spark.conf.set(“spark.sql.adaptive.skewJoin.enabled”, “true”)

# AQE automatically splits skewed partitions at runtime (enabled by default)

# Enable first; only implement manual salting if AQE is insufficient

Operations, monitoring & SRE practices

10.1 Observability stack

A production Databricks deployment requires three observability layers: job-level monitoring (did the job succeed? how long did it take?), cluster-level monitoring (CPU, memory, network utilization), and data-level monitoring (did the data arrive? is it fresh? does it pass quality checks?).

Observability layer	Data source	Tooling	Key alerts
Job & pipeline	Databricks jobs API, DLT pipeline events, mLflow run metrics	Azure Monitor (Log Analytics), Databricks workflow emails	Job failure, duration > SLA, DLT expectation violation rate spike
Cluster & compute	Databricks cluster metrics to Azure Monitor, ganglia (legacy)	Azure Monitor dashboards, custom KQL queries	Spot interruption rate above threshold, memory OOM events, driver failure
Data quality	DLT expectations metrics, custom great expectations/Soda suites	Databricks SQL dashboards, Power BI, pagerDuty	Null rate spike, row count anomaly, freshness breach (> SLA hours since last update)
Cost	Databricks cost API, Azure cost management	Azure cost management dashboards + budget alerts	Daily DBU spend > budget threshold, unexpected long-running cluster

10.2 CI/CD for Databricks

Production Databricks deployments require a CI/CD pipeline that tests notebooks and jobs before deployment, promotes code through environments (dev → test → prod), and maintains configuration as code for all workspace resources.

# Azure DevOps pipeline: Databricks CI/CD (YAML)

# Stages: Test → deploy to test → integration test → deploy to prod

trigger:

branches:

include: [main, release/*]

stages:

– stage: Test

jobs:

– job: UnitTests

steps:

– script: pip install databricks-connect pytest pytest-mock

– script: pytest tests/unit/ -v –junitxml=test-results.xml

– task: PublishTestResults@2

inputs:

testResultsFiles: test-results.xml

– stage: DeployTest

dependsOn: Test

jobs:

– job: DeployToTest

steps:

– script: |

pip install databricks-cli

databricks workspace import_dir ./notebooks /shared/pipelines/test –overwrite

databricks jobs reset –job-id $TEST_JOB_ID –json @job_configs/test/ingest_job.json

env:

DATABRICKS_HOST: $(DATABRICKS_TEST_HOST)

DATABRICKS_TOKEN: $(DATABRICKS_TEST_TOKEN)

– stage: DeployProd

dependsOn: IntegrationTest

condition: and(succeeded(), eq(variables[‘Build.SourceBranch’], ‘refs/heads/main’))

jobs:

– deployment: DeployToProd

environment: production  # requires manual approval gate

strategy:

runOnce:

deploy:

steps:

– script: databricks bundles deploy –target prod

# Databricks asset bundles: recommended for production deployments

10.3 Databricks asset bundles

Databricks Asset Bundles (DABs) are the recommended approach for managing Databricks resources as code. DABs define jobs, pipelines, notebooks, and cluster configurations in a YAML manifest that can be version-controlled, deployed consistently across environments, and integrated with CI/CD pipelines.

# databricks.yml, asset bundle definition

bundle:

name: analytics-platform

variables:

catalog: {default: dev_catalog}

environment: {default: dev}

targets:

dev:

mode: development

workspace:

host: https://adb-dev.azuredatabricks.net

variables:

catalog: dev_catalog

environment: dev

prod:

mode: production

workspace:

host: https://adb-prod.azuredatabricks.net

variables:

catalog: main

environment: prod

resources:

pipelines:

iot_sensor_pipeline:

name: “IoT sensor pipeline (${var.environment})”

catalog: ${var.catalog}

schema: silver

libraries:

– notebook: {path: ./src/pipelines/iot_sensor_pipeline.py}

continuous: true

channel: CURRENT

# Deploy: databricks bundle deploy –target prod

# Destroy: databricks bundle destroy –target dev

Migration patterns: On-Premises & cloud to Databricks

11.1 Migration assessment framework

Before migrating any workload to Databricks, a structured assessment determines which workloads benefit most, which require significant re-engineering, and which should remain on existing platforms.

Workload type	Migration complexity	Migration value	Recommended approach
SQL Server SSIS packages	Medium	High	Rewrite as DLT pipelines or ADF; use SSIS → Databricks migration accelerator tools
Informatica powerCenter mappings	High	High	Use Databricks professional services migration toolkit; expect partial automation, with manual rewrite for the rest
Teradata SQL workloads	Medium-High	Very high (cost reduction)	Use Databricks Teradata migration utilities; SQL dialect conversion tools; validate results
On-premises Hadoop (HDFS + hive)	Low-Medium	Very high	Lift-and-shift HDFS to ADLS Gen2; convert hive tables to Delta; Spark code works with minimal changes
Azure Synapse Analytics (dedicated pool)	Low	Medium	Delta Sharing from Databricks to Synapse; or migrate workloads preferring Databricks compute to Databricks
Snowflake	Medium	High (for mL-heavy orgs)	Use Delta Sharing to mirror key tables; migrate engineering workloads; evaluate co-existence vs. full migration

11.2 Hadoop migration pattern

Hadoop-to-Databricks is the most common migration pattern and also the most straightforward. HDFS and Hive are architecturally similar to ADLS Gen2 and Delta Lake, making it possible to migrate many Hadoop workloads with minimal code changes.

1. Storage migration: Use Azure Data Box for initial bulk transfer of HDFS data to ADLS Gen2 (for large datasets). Use AzCopy for incremental transfer during cutover. Target: Bronze layer in ADLS Gen2.
2. Hive table conversion: Run Databricks CONVERT TO DELTA on all Parquet-backed Hive tables. For ORC-backed tables, read as Spark DataFrame and write as Delta. Register in Unity Catalog metastore.
3. Spark code migration: Existing PySpark and Scala Spark code often runs on Databricks Runtime without modification. Common issues: Hadoop-specific configuration properties, HDFS path references, and deprecated Spark 2.x APIs that were removed in Spark 3.x.
4. Hive metastore migration: Use Databricks Unity Catalog migration tool to migrate existing Hive metastore to Unity Catalog, preserving table definitions, partition metadata, and (where possible) access control information.
5. Scheduler migration: Replace Apache Oozie or Airflow (on Hadoop YARN) with Databricks Workflows or Azure Data Factory. Export Oozie workflow XML and use the Databricks Oozie migration converter for automated translation.

11.3 Teradata migration pattern

Teradata migrations require the most careful SQL dialect conversion and performance validation.

Migration phase	Activities	Key risk
Discovery	Inventory all Teradata objects (tables, views, stored procedures, macros); profile query patterns; identify BTEQ and TPT scripts	Undocumented dependencies; legacy macros with no documentation
Schema conversion	Convert Teradata DDL to Delta Lake DDL; map data types; handle SET tables (implicit dedup) vs. MULTISET (allow duplication)	SET table semantics differ from Delta; requires explicit deduplication logic
SQL conversion	Convert BTEQ → pySpark/SQL; convert Teradata-specific functions (QUALIFY, SAMPLE, periods) to Spark equivalents	QUALIFY (row numbering within groups) needs careful WINDOW function translation
Performance validation	Run parallel execution: Teradata vs. Databricks; validate row counts, aggregations, and business metrics match	Floating point rounding differences; NULL handling differences
Cutover	Final data sync; redirect pipelines; decommission Teradata	Cutover window; rollback plan if validation fails

CHAPTER 12

90-Day implementation roadmap

The following roadmap’s sequencing is optimized for: early value demonstration (quick wins by Day 30), platform stability before scale (foundation before onboarding all workloads), and sustainable operations (CI/CD and monitoring before production traffic).

Phase 1: Foundation (Days 1-30)

Week	Activity	Owner	Success criteria
1	Azure Databricks workspace deployment with vNet injection; Unity Catalog metastore creation; Entra ID SCIM sync configuration; Key Vault-backed secret scope setup	Data platform engineer + Azure architect	Workspace live in private network; Unity Catalog active; first Entra ID group synced; Key Vault scope working
1-2	ADLS Gen2 storage account creation (bronze/silver/gold); Unity Catalog external locations and storage credentials; cluster policies defined and published	Data platform engineer	Storage accounts with hierarchical namespace; external locations in Unity Catalog; cluster policies active
2	Catalog and schema design in Unity Catalog; RBAC model design; data domain owner identification; Entra ID group structure for Databricks access	Data governance lead + data architect	Catalog hierarchy documented and deployed; RBAC matrix approved; all data domain owners identified and briefed
2-3	First pipeline: select highest-value Bronze ingestion (one source); build as DLT pipeline with expectations; deploy to dev; validate against source	Senior data engineer	First DLT pipeline ingesting into Bronze Delta table; expectations defined; data validated against source system
3-4	CI/CD pipeline setup (Azure DevOps); Databricks asset bundles configured; dev/prod workspace deployment automation; first automated unit tests	Data engineer + DevOps	Code deployment automated; dev→prod promotion via pipeline; unit tests running on every PR
4	Azure Monitor integration; Log Analytics workspace linked; first cost alert configured; Databricks audit log delivery enabled	Data platform engineer	Cluster metrics in Log Analytics; cost alert active; audit logs flowing

Phase 2: Activation (Days 31-60)

Week	Activity	Owner	Success criteria
5-6	Onboard tier 1 workloads: build Silver transformation pipelines with DLT; define data quality expectations; deploy to production	Data engineering team	Tier 1 Silver tables populated; DLT expectations capturing quality metrics; pipeline SLAs defined
6	Quick win: first Gold table serving Power BI / Fabric; first analyst using Databricks SQL for self-service query; governance maturity demo to leadership	Data engineering + analytics lead	First Gold table live; at least 3 analysts running queries; leadership demo delivered
6-7	Unity Catalog lineage verification; column masks and row filters for PII data; Purview integration configured (if in scope)	Data engineer + data governance	Lineage visible in catalog explorer for tier 1 tables; PII columns masked; Purview sync active
7-8	MLflow setup (if ML in scope): first experiment tracked; model registered in Unity Catalog; model serving endpoint deployed for at least one model	ML engineer	MLflow experiments running; first model in registry with champion/Challenger staging; serving endpoint live
7-8	Streaming pipeline deployment (if real-time in scope): Event Hubs connector; DLT streaming with enhanced autoscaling; latency SLA validation	Senior data engineer	Streaming pipeline ingesting at production volume; end-to-end latency within SLA; autoscaling tested under load
8	Cost review: spot instance utilization report; cluster right-sizing audit; predictive optimization enabled on UC catalogs; budget alert refinement	Data platform engineer + finOps	Identified and actioned top 3 cost reduction opportunities

Phase 3: Scale & optimise (Days 61-90)

Week		Activity	Owner	Success criteria
9-10		Onboard remaining workloads: complete Bronze ingestion for all tier 2 sources; Silver and Gold for priority domains; DLT for all new pipelines (no raw Spark jobs)	Data engineering team	Most planned data sources ingested; all new pipelines using DLT; data quality SLAs documented
10-11		Performance engineering: identify top 10 slowest jobs; apply Z-ORDER, OPTIMIZE, and Photon; validate speedup; document optimization patterns for team	Senior data engineer	Top 10 jobs optimized; performance tuning runbook created
11		Operations runbook: incident response procedures; on-call rotation; DLT pipeline restart procedures; cost escalation process; SLA breach response	Data platform lead + SRE	Runbook published; on-call rotation active; first gameday exercise completed
12		Platform review: measure KPIs vs. day 1 baseline; business case revalidation; 90-180 day roadmap (next workloads, ML roadmap, Fabric integration plan)	CDO + data platform lead	KPI improvement documented; business case validated; 90-180 day plan approved by leadership
	Critical success factor: The most common Databricks deployment failure mode is “platform without pipelines”, the infrastructure is deployed correctly but business workloads are never migrated to it. Ensure at least one business-critical pipeline is live in production by day 45. early production usage drives adoption, surfaces real-world issues before scale, and maintains organizational momentum.

APPENDIX

Appendix: Reference materials

A. essential PySpark patterns

A1. efficient incremental load with watermarks

# Production pattern: Incremental load with high-water mark

# Stores last processed timestamp in a metadata Delta table

from delta.tables import deltaTable

from pyspark.sql import functions as F

# Read watermark

watermark = spark.table(“main.metadata.pipeline_watermarks”) \

.filter(“pipeline_name = ‘orders_ingestion'”) \

.select(“last_processed_ts”) \

.collect()[0][0]

# Extract only new/changed records

new_records = spark.read.format(“jdbc”) \

.option(“url”, jdbc_url) \

.option(“query”, f”SELECT * FROM orders WHERE updated_at > ‘{watermark}'”) \

.load()

# Upsert to Bronze Delta table

bronze_table = deltaTable.forName(spark, “main.bronze.orders”)

bronze_table.alias(“target”) \

.merge(new_records.alias(“source”), “target.order_id = source.order_id”) \

.whenMatchedUpdateAll() \

.whenNotMatchedInsertAll() \

.execute()

# Update watermark

new_watermark = new_records.agg(F.max(“updated_at”)).collect()[0][0]

spark.sql(f”””

UPDATE main.metadata.pipeline_watermarks

SET last_processed_ts = ‘{new_watermark}’

WHERE pipeline_name = ‘orders_ingestion’

“””)

A2. dynamic partition overwrite pattern

# Safe overwrite of specific partitions without touching others

# Use case: reprocess a date range after source data correction

spark.conf.set(“spark.sql.sources.partitionOverwriteMode”, “dynamic”)

# This overwrites ONLY the partitions present in df, leaves others untouched

df.write \

.format(“delta”) \

.mode(“overwrite”) \

.option(“replaceWhere”, “transaction_dt >= ‘2025-01-01’ AND transaction_dt <= ‘2025-01-07′”) \

.saveAsTable(“main.silver.sales_transactions”)

# Note: replaceWhere is safer than dynamic partition overwrite for Delta

# It uses Delta transaction semantics; atomic; rolls back on failure

B. Databricks REST API quick reference

Operation	Method	Endpoint	Common use
List jobs	GET	/api/2.1/jobs/list	Audit all jobs; generate cost attribution reports
Run job now	POST	/api/2.1/jobs/run-now	Trigger job from external orchestrator (ADF, logic apps)
Get run status	GET	/api/2.1/jobs/runs/get?run_id={id}	Poll completion from external orchestrator
List DLT pipelines	GET	/api/2.0/pipelines	Audit pipelines; governance inventory
Get pipeline events	GET	/api/2.0/pipelines/{id}/events	Extract DLT expectation metrics for monitoring
List Unity Catalog tables	GET	/api/2.1/unity-catalog/tables	Data catalog integration; asset inventory
Get table lineage	GET	/api/2.0/lineage-tracking/table-lineage	Extract lineage for Purview integration
Create cluster policy	POST	/api/2.0/policies/clusters/create	Infrastructure-as-code cluster governance
List mLflow experiments	GET	/api/2.0/mlflow/experiments/list	ML governance; experiment inventory
Get model versions	GET	/api/2.0/mlflow/model-versions/search	Model registry governance; deployment audit

C. Databricks runtime version policy

Runtime channel	Description	Recommendation
CURRENT	Latest stable DBR; receives bug fixes and security patches	Use for all production workloads; update quarterly
PREVIEW	Next DBR version; not yet GA; may contain breaking changes	Use in development/test only; never in production
LTS (long-term support)	Extended support cycle (3 years from GA); conservative updates	Use for regulated workloads with strict change control; accept performance tradeoff vs. CURRENT
Custom (ML runtime)	Includes tensorFlow, pyTorch, scikit-learn, xGBoost pre-installed	Use for all ML workloads; eliminates library installation time on cluster start

D. glossary

Term	Definition
Adaptive query execution (AQE)	Spark feature that re-optimizes query plans at runtime based on actual data statistics; automatically handles skew, coalesces partitions, and selects join strategies
Asset bundle (DAB)	Databricks resource configuration framework that defines jobs, pipelines, and clusters as YAML manifests for version-controlled, environment-aware deployment
Change data feed (CDF)	Delta Lake feature that captures row-level changes (inserts, updates, deletes) with change type metadata, enabling efficient incremental downstream processing
Delta Lake	Open-source storage layer providing ACID transactions, scalable metadata, time travel, and schema enforcement on top of Parquet files in cloud object storage
Delta Live Tables (DLT)	Databricks declarative pipeline framework; engineers declare data transformations and quality expectations; DLT manages execution, retry, monitoring, and autoscaling
Lakehouse	Architecture combining data lake storage economics with data warehouse reliability and performance, using open table formats (Delta Lake, iceberg, hudi) as the unifying layer
Medallion Architecture	Data organization pattern (Bronze/Silver/Gold) representing progressive data quality levels, from raw ingestion to business-certified analytics-ready assets
MLflow	Open-source platform for ML lifecycle management: experiment tracking, model registry, model serving, and evaluation; created by Databricks; integrated into Unity Catalog
Photon	Databricks C++ vectorized query engine replacing JVM Spark execution for analytical workloads; delivers performance improvements with no code changes
Unity Catalog	Databricks unified governance layer providing centralized access control, lineage tracking, data sharing, and audit across all workspaces in an organization
Z-Ordering	Delta Lake data layout optimization that co-locates related data in the same files based on specified columns, improving query performance by enabling more aggressive data skipping

Secure, well-architected Azure landing zones.

Scalable, governed analytics foundations on Azure and Fabric.

Unify the data estate on one governed analytics platform.

Talk to the team behind these articles. Start with a no-obligation discovery call with a senior Veratas architect.