Interactive Learning Guide

Azure Databricks Architecture
& Design Fundamentals

A practical guide to Databricks' core concepts on Azure — from the Lakehouse model and Delta Lake to Unity Catalog, MLflow, and enterprise security. Expand each section for deep dives and test your understanding with mini-quizzes.

0 of 12 sections
0%
01

The Lakehouse — One Platform for Everything

Databricks invented the Lakehouse pattern to collapse the traditional two-tier architecture (data lake + data warehouse) into a single layer. All data lives in open-format files on cloud storage; Delta Lake adds warehouse-grade reliability on top.

Cloud Storage — Azure Data Lake Storage Gen2 (ADLS Gen2)
Delta Parquet files
_delta_log/ (transaction log)
Raw / Bronze zone
Silver & Gold zones
↕ open format — no proprietary lock-in
Delta Lake — Reliability Layer (ACID, Schema, Time Travel)
ACID Transactions
Schema Enforcement
Time Travel
Z-Order Clustering
Liquid Clustering
↕ unified query engine
Databricks Runtime — Apache Spark + Photon
Spark SQL
PySpark / Scala
Photon (C++ vectorised)
Structured Streaming
↕ serve all workloads
Workload Surfaces
Data Engineering
Databricks SQL / BI
Machine Learning
Streaming Analytics

Lakehouse vs Data Warehouse (Snowflake): Both separate compute from storage, but the Lakehouse keeps data in open Parquet/Delta files. This means your data scientists can read the same files with Python/pandas directly from ADLS Gen2, without copying data to a separate analytics store. There is no "export to data lake" step — it is the lake.

🔍 Deep Dive: Why Not a Traditional Data Lake?

+

Classic data lake problems: Raw files on ADLS Gen2 give you cheap storage and any-format flexibility, but with no ACID guarantees. A failing ETL job leaves partial writes visible to readers. No schema enforcement means a bad upstream schema change silently corrupts downstream pipelines. No updates — you must rewrite entire partitions to fix one row.

Delta Lake solves all of this: It wraps your Parquet files in a transaction log (_delta_log/). Every INSERT, UPDATE, MERGE, and DELETE is recorded as an atomic commit. Readers always see a consistent snapshot. Schema is enforced at write time. Old versions are retained so you can query yesterday's data or roll back bad writes.

Lakehouse vs Data Warehouse trade-off: The Lakehouse excels when your team writes Python/Spark, runs ML workloads, and wants one storage layer across structured and semi-structured data. A pure Data Warehouse (Synapse Analytics, Snowflake) wins when your team is primarily SQL analysts who need best-in-class concurrency scaling and fine-grained per-query cost control without cluster management.

✅ Check Your Understanding

Q1: A junior engineer asks why Databricks stores data as Parquet files instead of a proprietary format. What's the best answer?

Q2: An ETL job fails halfway through writing 10M rows. Without Delta Lake, what happens to downstream readers?


02

Workspace & Azure Infrastructure Setup

A Databricks Workspace is the top-level Azure resource — it maps to one Azure region and one Azure subscription. Understanding the network and identity setup is essential before writing a single notebook.

Your Azure Subscription
Resource Group (customer-managed)
ADLS Gen2 Storage Account
Azure Key Vault
VNet (optional but recommended)
↕ Databricks manages compute in a separate managed resource group
Databricks Managed Resource Group (auto-created)
Cluster VMs (Worker + Driver)
Managed Disks (shuffle storage)
NAT Gateway (egress)
↕ Databricks Control Plane (SaaS, Databricks-managed)
Databricks Control Plane — databricks.com
Web UI / API
Job Scheduler
Cluster Manager
Unity Catalog Metastore

🏗️ VNet Injection

Deploy cluster VMs into your own VNet instead of a Databricks-managed VNet. Required for: private endpoint access to ADLS Gen2, on-premises connectivity via ExpressRoute/VPN, NSG-level traffic control. Must be configured at workspace creation — cannot change later.

🔒 Private Link

Route traffic between notebooks/clusters and the Databricks control plane over Azure Private Link. Prevents all data and control traffic from crossing the public internet. Recommended for enterprise environments with strict network compliance requirements.

👤 Managed Identity

Assign a User-Assigned Managed Identity to the Databricks workspace to access ADLS Gen2 and Key Vault without secrets. The cluster authenticates using the MI token automatically — no credentials stored in notebooks or config files.

💰 SKU Tiers

Standard tier: core Spark + notebooks + basic jobs. Premium tier: Unity Catalog, cluster policies, enhanced audit logs, SCIM provisioning, IP access lists, credential passthrough. Enterprise customers almost always need Premium.

🔍 Deep Dive: Managed Resource Group Explained

+

When you create a Databricks workspace, Azure automatically creates a second resource group (e.g. databricks-rg-myworkspace-xxxx) that Databricks manages on your behalf. This is where all cluster VMs, managed disks, and network infrastructure (load balancers, public IPs) live while clusters are running.

Important: You cannot manually delete resources in the managed resource group — Databricks manages their lifecycle. When a cluster terminates, the VMs are deleted (you stop paying). When the cluster starts, new VMs are provisioned fresh. This is why Databricks clusters have a ~1–3 minute cold start time.

Cost implication: VM costs accrue to your Azure subscription under the managed resource group. Databricks DBU (Databricks Unit) fees are billed separately through the Azure Marketplace. Your total cost = Azure VM cost + Databricks DBU cost.

🔍 Deep Dive: ADLS Gen2 Setup for Databricks

+

Storage account requirements: Enable Hierarchical Namespace (HNS) on the storage account — this turns it into ADLS Gen2. Without HNS, you get plain Blob Storage which is slower for directory-heavy operations. HNS cannot be enabled after creation.

Container layout (recommended):

# One container per environment, one sub-path per Lakehouse zone adls-account/ landing/ # raw source files (immutable) bronze/ # Delta tables — raw ingested silver/ # Delta tables — cleansed, conformed gold/ # Delta tables — aggregated, BI-ready checkpoints/ # Structured Streaming checkpoints mlflow/ # MLflow artifact store

Access method priority: (1) Unity Catalog storage credentials using Managed Identity — no secrets, no rotation headaches. (2) Service Principal with client secret stored in Key Vault and mounted via Databricks Secret Scope. (3) Storage Account Access Key — avoid in production (too broad, not auditable).


03

Clusters & Compute — Types, Sizing & Policies

Every computation in Databricks runs on a cluster — a set of Azure VMs coordinated by Apache Spark. Choosing the right cluster type and size is the single biggest lever for controlling cost and performance.

🔬 All-Purpose Cluster

Interactive clusters you start manually and share with your team. Used in notebooks for exploration, ad-hoc analysis, and development. Billed at the All-Purpose DBU rate (most expensive). Auto-terminates after N minutes of inactivity. Never use for production jobs — use Job clusters instead.

⚙️ Job Cluster

Ephemeral clusters spun up for a specific job run and destroyed after completion. Cheaper DBU rate (~50% of All-Purpose). No warm-up tax shared across jobs. Best practice: every production pipeline gets its own Job cluster. Cold start adds ~2 min to job duration.

🏊 Instance Pool

A pool of pre-allocated idle VMs that clusters draw from instead of provisioning new VMs. Eliminates the ~2 min cold start. Great for jobs triggered frequently (e.g. every 15 minutes). You pay for idle pool VMs, so size the pool to your actual concurrency.

🔌 SQL Warehouse

A dedicated compute type for Databricks SQL only — not for notebook/Spark workloads. Runs Photon (vectorised C++ engine) for fast analytical SQL. Supports Serverless mode (no cluster management, instant start) and Classic mode. Used by BI tools like Power BI and Tableau.

Common Azure VM Sizes for Databricks

Standard_DS3_v2
4 vCPU, 14 GB RAM
Dev / small teams, light ETL
Standard_DS4_v2
8 vCPU, 28 GB RAM
Standard pipelines, medium data
Standard_L8s_v3
8 vCPU, 64 GB + NVMe SSD
Large shuffles, cache-heavy joins
Standard_NC6s_v3
6 vCPU, 112 GB, 1× V100 GPU
Deep learning training, GPU inference

Cluster sizing rule of thumb: Start with 4–8 worker nodes of DS4_v2. If shuffle spills to disk (check Spark UI → Storage tab), switch to memory-optimised (Standard_E series). If jobs are CPU-bound with large joins, try storage-optimised (Standard_L series with local NVMe).

🔍 Deep Dive: Driver vs Worker Nodes

+

Driver node: Runs the SparkContext, DAG scheduler, and your application code (Python main thread). All collect(), toPandas(), and print output lands here. If the driver runs out of memory, the entire application crashes. Size the driver at least as large as one worker — for notebook workloads, often larger.

Worker nodes: Execute distributed tasks — reading partitions, applying transformations, writing output. Each worker runs one or more Executors. Spark assigns each task to one CPU core on one executor. Adding workers adds parallelism.

Single-node cluster: For lightweight ML training or pandas workloads that don't distribute, a single-node cluster (driver only, no workers) avoids Spark overhead and is cheaper. Databricks notebooks can run pandas, scikit-learn, and even PyTorch on single-node clusters.

# Check executor memory and cores from a notebook spark.sparkContext.getConf().get("spark.executor.memory") spark.sparkContext.getConf().get("spark.executor.cores")

🔍 Deep Dive: Cluster Policies

+

Cluster Policies (Premium tier only) are JSON templates that constrain what users can configure when creating clusters. Use them to enforce governance: maximum cluster size, allowed VM types, mandatory auto-termination, required tags for cost allocation.

// Example cluster policy — enforce auto-termination, cap workers at 10 { "autotermination_minutes": { "type": "fixed", "value": 120 }, "num_workers": { "type": "range", "maxValue": 10 }, "node_type_id": { "type": "allowlist", "values": ["Standard_DS3_v2", "Standard_DS4_v2"] } }

Assign a policy to a team (e.g. "Data Analysts") so they can create clusters without spinning up 100-node monsters. Admins are exempt from policies. You can also set custom tags as fixed values in policies to enforce cost attribution by team or project.

✅ Check Your Understanding

Q1: A production pipeline runs hourly. Should it use an All-Purpose cluster or a Job cluster?

Q2: A short job runs every 5 minutes. Cold start takes 2 minutes. What's the best fix?


04

Delta Lake — ACID, Schema, Time Travel & Optimisation

Delta Lake is the storage layer that transforms plain Parquet files into a reliable, queryable table format. Every Databricks table you create is a Delta table by default. Understanding how it works determines how well your pipelines perform at scale.

⚡ ACID Transactions

Every write (INSERT, UPDATE, DELETE, MERGE) is recorded in _delta_log/ as a JSON commit entry. Readers always see the last complete transaction. Concurrent writes use optimistic concurrency — conflict detection prevents data corruption without locking tables.

📋 Schema Enforcement

Delta rejects writes that don't match the table schema by default. Add mergeSchema or run ALTER TABLE to evolve the schema intentionally. This prevents a bad upstream source from silently corrupting your Silver/Gold tables.

⏱️ Time Travel

Query any previous version of a Delta table by version number or timestamp. Data retention controlled by delta.logRetentionDuration (default: 30 days). Essential for auditing, rollbacks, and reproducing ML training datasets.

🔀 MERGE (Upsert)

The most common CDC pattern. MERGE matches rows on a key, UPDATEs matches, INSERTs new rows, and optionally DELETEs flagged rows — all in one atomic operation. Equivalent to SCD Type 1 in a single SQL statement.

🔍 Deep Dive: The Transaction Log (_delta_log/)

+

The _delta_log/ directory sits at the root of every Delta table. Each write creates a JSON file numbered sequentially: 00000000000000000000.json, 00000000000000000001.json, etc. Every 10 commits, Delta checkpoints the state to a Parquet file for fast catch-up.

What the log records: Which files were added, which were removed, schema at the time of commit, operation metadata (MERGE, INSERT, etc.), transaction ID, and statistics (min/max values per column — used for data skipping).

-- View transaction history for any Delta table DESCRIBE HISTORY silver.customers; -- Query data as it was 7 days ago SELECT * FROM silver.customers TIMESTAMP AS OF dateadd('day', -7, current_date()); -- Roll back a table to a specific version RESTORE TABLE silver.customers TO VERSION AS OF 42;

Data skipping: When a query has a filter (WHERE clause), Delta reads the min/max stats in the log to skip entire files that can't contain matching rows. A well-partitioned and Z-Ordered table can skip 90%+ of files, making point-queries fast without an index.

🔍 Deep Dive: OPTIMIZE, Z-Order & Liquid Clustering

+

OPTIMIZE: Small files accumulate from frequent streaming writes. OPTIMIZE compacts them into larger files (target: 1 GB each) to reduce Spark task overhead and improve scan throughput. Run OPTIMIZE on Silver/Gold tables after major write operations.

OPTIMIZE silver.events; -- compact small files OPTIMIZE silver.events ZORDER BY (customer_id); -- + co-locate by key

Z-Order: Multi-dimensional clustering that physically arranges data so rows with similar values on the Z-Order columns land in the same files. Dramatically reduces data read for point lookups and range queries on the clustered columns. Limited to 4 columns. Must re-run OPTIMIZE when new data arrives — not automatic.

Liquid Clustering (newer): Replaces Z-Order with an incremental background clustering algorithm. You define the clustering keys once (CLUSTER BY) and Delta automatically reorganises data as you write. No manual OPTIMIZE ZORDER scheduling needed. Currently the recommended approach for new tables on Databricks Runtime 13.3+.

-- Create table with Liquid Clustering (DBR 13.3+) CREATE TABLE silver.events CLUSTER BY (customer_id, event_date) LOCATION 'abfss://silver@adls.dfs.core.windows.net/events';

VACUUM — removing old files: Delta retains deleted/old files for Time Travel. Run VACUUM tablename RETAIN 7 HOURS to remove files older than 7 hours. Default retention is 7 days. Never vacuum below your Time Travel window, or time-travel queries will fail.


05

Ingestion Patterns — Auto Loader, DLT & Streaming

Getting data into the Lakehouse reliably is the foundation of every pipeline. Databricks offers three primary ingestion mechanisms, each optimised for different arrival patterns and reliability requirements.

PatternBest forLatencyExactly-once?Managed by
Auto Loader New files arriving in ADLS Gen2 (CSV, JSON, Parquet, Avro) Minutes (batch trigger) ✅ Yes Databricks
Structured Streaming Event Hub / Kafka real-time streams Seconds ✅ Yes (w/ checkpoints) You (or DLT)
Delta Live Tables (DLT) Declarative pipeline with auto-retries, quality constraints Minutes (triggered) or Seconds (continuous) ✅ Yes Databricks
COPY INTO One-time or idempotent bulk load from ADLS Gen2 Batch ✅ Yes (tracks loaded files) You

🔍 Deep Dive: Auto Loader

+

Auto Loader uses Azure Event Grid notifications on the ADLS Gen2 container to detect new file arrivals without scanning the entire directory. This is far more efficient than listing all files on each run — critical when you have millions of files in a landing zone.

# Auto Loader — read new files as they arrive df = ( spark.readStream .format("cloudFiles") .option("cloudFiles.format", "json") .option("cloudFiles.schemaLocation", "/checkpoints/schema/events") .load("abfss://landing@adls.dfs.core.windows.net/events/") ) ( df.writeStream .format("delta") .outputMode("append") .option("checkpointLocation", "/checkpoints/bronze/events") .trigger(availableNow=True) # process all pending then stop .toTable("bronze.events") )

Schema inference: Auto Loader can infer schema from the first N files and store it in schemaLocation. When new columns appear, you can configure cloudFiles.schemaEvolutionMode to auto-add columns, raise an error, or rescue into a _rescued_data column (safest choice for production).

🔍 Deep Dive: Delta Live Tables (DLT)

+

DLT is Databricks' declarative pipeline framework. You define tables using Python decorators or SQL; DLT handles ordering, dependencies, retries, monitoring, and lineage automatically.

import dlt # Bronze: ingest raw from Auto Loader @dlt.table(comment="Raw events from landing zone") def bronze_events(): return ( spark.readStream.format("cloudFiles") .option("cloudFiles.format", "json") .load("abfss://landing@adls.dfs.core.windows.net/events/") ) # Silver: cleanse with quality constraints @dlt.expect_all_or_drop({ "valid_event_type": "event_type IS NOT NULL", "non_negative_amount": "amount >= 0" }) @dlt.table() def silver_events(): return dlt.read_stream("bronze_events").select(...)

Expectations (data quality rules): @dlt.expect warns on violations; @dlt.expect_or_drop silently drops invalid rows; @dlt.expect_or_fail halts the pipeline. DLT tracks violation counts over time, visible in the pipeline UI — you get a data quality dashboard for free.


06

Medallion Architecture — Bronze, Silver, Gold

The Medallion pattern is Databricks' standard approach to layering data quality across the Lakehouse. Each layer has a clear contract — what goes in, what comes out, and who may write to it.

🥉 Bronze
Raw ingestion — append only
  • Exact copy of source data
  • No transformations applied
  • Schema as received
  • Append-only — never update
  • Long retention (1+ years)
  • Written by: ingestion pipelines
🥈 Silver
Cleansed & conformed
  • Data type casting & validation
  • Deduplication applied
  • PII masked or hashed
  • Null handling enforced
  • Business keys available
  • Written by: transformation jobs
🥇 Gold
Aggregated & BI-ready
  • Business metric calculations
  • Denormalised for query speed
  • Pre-aggregated dimensions
  • Aligned to business domains
  • Read by: BI tools, APIs, ML
  • Written by: analytics engineers

Key rule: Data flows only forward — Bronze → Silver → Gold. A Gold table never writes back to Silver. If upstream data changes (e.g. a historical correction), reprocess Bronze and cascade forward. This unidirectional flow makes debugging and re-processing predictable.

🔍 Deep Dive: Implementing Medallion with Unity Catalog

+

Map each Medallion layer to a separate Unity Catalog schema (database) within a catalog. This makes access control simple — you grant different teams read access to different schemas.

-- Unity Catalog namespace: catalog.schema.table CREATE CATALOG prod_lakehouse; CREATE SCHEMA prod_lakehouse.bronze MANAGED LOCATION 'abfss://bronze@...'; CREATE SCHEMA prod_lakehouse.silver MANAGED LOCATION 'abfss://silver@...'; CREATE SCHEMA prod_lakehouse.gold MANAGED LOCATION 'abfss://gold@...'; -- Grant analysts read access to silver and gold only GRANT USE SCHEMA ON SCHEMA prod_lakehouse.silver TO `analysts@company.com`; GRANT SELECT ON SCHEMA prod_lakehouse.silver TO `analysts@company.com`; GRANT USE SCHEMA ON SCHEMA prod_lakehouse.gold TO `analysts@company.com`; GRANT SELECT ON SCHEMA prod_lakehouse.gold TO `analysts@company.com`;

Write permissions: Ingestion service principals get WRITE on Bronze only. ETL service principals get READ on Bronze and WRITE on Silver. Analytics engineers get READ on Silver and WRITE on Gold. This strict layering prevents accidental overwrites upstream.


07

Unity Catalog — Governance, Lineage & Access Control

Unity Catalog is Databricks' unified data governance layer. It provides a three-level namespace, centralised permission management, column-level security, and automatic data lineage — all across all workspaces in an account.

Unity Catalog Namespace Hierarchy
Account (Databricks)
→
Metastore (1 per region)
→
Catalog
→
Schema
→
Table / View / Function

🗂️ Three-Level Namespace

Every object is referenced as catalog.schema.table. Multiple workspaces in the same region share one metastore — a table defined in workspace A is immediately discoverable in workspace B, with the same permissions. Eliminates per-workspace Hive metastores.

🔗 Data Lineage

Unity Catalog automatically captures column-level lineage for every SQL query and DLT pipeline. You can trace which upstream tables and columns contributed to any downstream table. No instrumentation required — lineage is derived from query plans.

🔐 Column-Level Security

Apply column masks (dynamic data masking) and row filters using SQL functions. A GDPR-sensitive column (e.g. email) can return a hash for most users and the real value only for the DPO group — enforced at the metastore level, not in each query.

🏷️ Tags & Search

Tag catalogs, schemas, tables, and columns with business metadata (PII, domain, owner). The Unity Catalog UI provides a searchable data catalogue. Integrate with Azure Purview for enterprise-wide discovery across Azure services and Databricks.

🔍 Deep Dive: Storage Credentials & External Locations

+

Unity Catalog separates storage credentials (authentication to ADLS Gen2) from external locations (which paths are accessible). This is how you grant Databricks access to your ADLS Gen2 without embedding storage account keys in notebooks.

-- 1. Create a storage credential (wraps a Managed Identity or Service Principal) CREATE CREDENTIAL adls_prod_credential WITH AZURE MANAGED IDENTITY ('<managed-identity-resource-id>'); -- 2. Create an external location that uses the credential CREATE EXTERNAL LOCATION silver_location URL 'abfss://silver@myadls.dfs.core.windows.net/' WITH (STORAGE CREDENTIAL adls_prod_credential); -- 3. Create an external table at that location CREATE TABLE prod.silver.customers LOCATION 'abfss://silver@myadls.dfs.core.windows.net/customers/';

Users reading prod.silver.customers never touch ADLS credentials directly — Unity Catalog's credential vending hands the cluster a short-lived token for the specific path. Revoke access centrally in Unity Catalog without rotating storage keys.

✅ Check Your Understanding

Q1: Your company has 4 Databricks workspaces across dev, staging, and prod. With Unity Catalog, how many metastores do you need?


08

Azure Security — Identity, Secrets & Network

Databricks on Azure integrates deeply with Azure security primitives. Getting the identity and secret management right from day one prevents the most common audit findings.

🔑 Service Principal vs Managed Identity

Service Principal (SP): An Azure AD application identity with a client ID and secret (or certificate). The secret must be rotated. Store in Key Vault, reference via Databricks Secret Scope. Managed Identity (MI): No credentials to manage — Azure assigns and rotates tokens automatically. Preferred for ADLS Gen2 access where supported by Unity Catalog.

🔒 Databricks Secret Scopes

Secret Scopes are name-indexed credential stores accessible from notebooks/jobs via dbutils.secrets.get(). Two backends: Databricks-managed (encrypted in Databricks' control plane) or Azure Key Vault-backed (secrets live in your Key Vault, Databricks fetches them read-only). Key Vault-backed is the enterprise recommendation — you control the lifecycle.

🌐 IP Access Lists

Restrict Databricks workspace UI and REST API access to approved IP ranges (corporate office IPs, VPN egress). Premium tier only. Prevents notebooks from being accessed from outside your network even with valid credentials.

👥 SCIM Provisioning

Sync users and groups from Azure AD to Databricks automatically using SCIM (System for Cross-domain Identity Management). When a user leaves the company and their Azure AD account is disabled, their Databricks access is revoked within minutes — no manual offboarding.

🔍 Deep Dive: Accessing Key Vault from Notebooks

+

The recommended pattern for secrets in production: store credentials in Azure Key Vault, create a Key Vault-backed Secret Scope in Databricks, reference via dbutils.secrets. The secret value is never printed in notebook output — Databricks redacts it in logs.

# Step 1: Create Key Vault-backed scope (done once via Databricks UI or API) # Scope name: kv-prod, backed by: https://mykeyvault.vault.azure.net/ # Step 2: Reference secret in a notebook jdbc_password = dbutils.secrets.get(scope="kv-prod", key="sql-db-password") # Step 3: Use the secret — it's a string, never print/log it connection_string = f"jdbc:sqlserver://server.database.windows.net;password={jdbc_password}" # dbutils.secrets.list(scope="kv-prod") -- shows key names, never values

What NOT to do: Never hardcode a connection string or password in a notebook cell, even in dev. Notebooks are versioned in the Databricks workspace and can be shared unintentionally. Always use Secret Scopes.

Audit logs: Enable Databricks diagnostic logging to Azure Monitor / Log Analytics. Captures notebook commands, cluster starts/stops, secret reads, permission changes, and login events. Essential for compliance (GDPR audit trails, SOC 2 evidence).


09

Workflows & Orchestration — Jobs, Tasks & Scheduling

Databricks Workflows is the native orchestration layer. It replaces the need for an external orchestrator (Airflow, Azure Data Factory) for Databricks-native workloads — though ADF or Airflow are still common for cross-service orchestration.

📋 Multi-Task Jobs

A Workflow Job can contain multiple Tasks connected by dependencies (DAG). Each task can run a notebook, Python script, JAR, DLT pipeline, dbt project, SQL query, or call a REST API. Tasks run sequentially or in parallel based on depends_on wiring.

⏰ Triggers

Schedule with cron or interval. Trigger on file arrival (file-arrival trigger on ADLS Gen2 path). Trigger via REST API (POST /api/2.1/jobs/run-now) from ADF, GitHub Actions, or any HTTP client. Trigger from another job's completion via webhooks.

🔁 Retries & Repair

Configure per-task retry count and retry interval. After a job run failure, use "Repair Run" to re-run only failed tasks (and their downstream dependents) without re-running successful tasks. Critical for expensive pipelines where early tasks shouldn't re-run.

📊 Job Compute

Each task specifies its compute: a new Job cluster (ephemeral, cheapest), an existing All-Purpose cluster (warm but expensive), an Instance Pool cluster (fast start), or a SQL Warehouse (SQL tasks). Mix and match — notebook tasks can use one cluster size while a heavy transform uses another.

🔍 Deep Dive: Databricks vs Azure Data Factory

+

Use Databricks Workflows when: All tasks are Databricks-native (notebooks, DLT, SQL). You want fine-grained task-level retry, repair runs, and cost attribution. Your team lives in Databricks and wants a single UI for pipeline monitoring.

Use Azure Data Factory (ADF) when: Your pipeline orchestrates multiple Azure services — Synapse, Blob Storage copy, SQL Database, external REST APIs. You need data flow (visual ETL without Spark). Your organisation already has ADF expertise and invested pipelines. ADF can call Databricks notebooks and jobs via the Databricks linked service.

Hybrid pattern (common in enterprise): ADF handles the "outer" orchestration — copy from source systems into ADLS Gen2, trigger Databricks job, wait for completion, trigger downstream Synapse SQL or Data Export. Databricks Workflows handles the "inner" orchestration — the Medallion transformation pipeline.

# ADF triggers a Databricks job via REST and waits POST /api/2.1/jobs/run-now { "job_id": 123, "job_parameters": { "run_date": "2026-06-16" } }

10

Databricks SQL — SQL Warehouses, BI & Photon

Databricks SQL is the BI-facing surface of the Lakehouse. It provides a SQL Warehouse (serverless or classic compute), a query editor, dashboards, and alerts — all reading from Delta tables in Unity Catalog.

⚡ Photon Engine

Photon is a C++ vectorised query engine that replaces Spark's JVM-based execution for SQL workloads. 2–8× faster on analytical queries (aggregations, joins, sorts). Enabled automatically on SQL Warehouses. Available on All-Purpose clusters with DBR 9.1+. No code changes required.

🖥️ Serverless SQL Warehouse

Serverless SQL Warehouses start in under 5 seconds (vs ~2–3 min for Classic). Databricks manages the compute pool — no cluster VM provisioning in your subscription. Scales to zero when idle. Charged per query second of compute. Best for: BI tools, ad-hoc SQL exploration, variable-load dashboards.

📊 BI Connector (Power BI)

Power BI connects to Databricks SQL Warehouses via the native Databricks connector (Partner Connect or direct server hostname). Uses ODBC under the hood. Supports DirectQuery mode — queries run live against Delta tables with Photon. No data import/extract needed.

🔔 Alerts

Create scheduled queries that check a metric (e.g. SELECT COUNT(*) FROM silver.failed_events WHERE dt = current_date()) and fire a notification (email, Slack, PagerDuty) when the result exceeds a threshold. Lightweight data quality alerting without external tooling.

Cost model: SQL Warehouse DBU rate is separate from All-Purpose and Jobs rates. Serverless SQL Warehouses charge per-second of query execution (not idle time). Classic Warehouses charge for VM uptime even when idle. For spiky BI load, Serverless is almost always cheaper. For sustained 8-hour BI workloads, Classic can be comparable.

🔍 Deep Dive: Query Result Caching

+

Databricks SQL caches query results for up to 24 hours. If an identical query (same SQL, same table version) is re-run within the window, the cached result is returned instantly — zero compute cost. The cache is invalidated when the underlying Delta table version changes.

For Power BI DirectQuery dashboards, this means: the first user to open a dashboard page runs the query and populates the cache. Subsequent users in the next 24 hours get instant results from cache as long as the data hasn't been refreshed. Design your Gold tables to write in batch (e.g. once per hour) to maximise cache hit rate rather than streaming every row individually.

Disk cache (Photon cache): Separate from query result cache. Photon caches frequently-accessed Delta table files in SSD on the warehouse nodes. Warm cache = 3–10× faster repeated scans on the same data. Cold-start performance of a new warehouse is lower for the first few queries while the Photon cache fills.


11

MLflow & AI — Experiment Tracking & Model Serving

Databricks ships a fully managed MLflow instance in every workspace — no setup required. It tracks experiments, stores model artefacts, manages model versions, and serves models as REST endpoints.

Feature Engineering
Silver Delta tables
→
Model Training
MLflow autolog
→
MLflow Registry
Staging → Production
→
Model Serving
REST endpoint
→
Predictions → Gold table

🔍 Deep Dive: MLflow Tracking & Autolog

+

MLflow Tracking records experiment runs — each run captures parameters, metrics (loss curves, accuracy), artefacts (model files, plots), and environment info. Autolog does this automatically for scikit-learn, XGBoost, PyTorch, and many other frameworks.

import mlflow import mlflow.sklearn from sklearn.ensemble import RandomForestClassifier # Autolog captures params, metrics, and model automatically mlflow.sklearn.autolog() with mlflow.start_run(run_name="churn_model_v3"): model = RandomForestClassifier(n_estimators=100, max_depth=6) model.fit(X_train, y_train) # Logs: n_estimators, max_depth, accuracy, F1, confusion matrix, model pkl # Log additional custom metrics mlflow.log_metric("business_lift", 0.42) # Register to Model Registry mlflow.register_model("runs:/<run_id>/model", "churn-classifier")

🔍 Deep Dive: Model Serving (Mosaic AI Model Serving)

+

Databricks Model Serving (formerly "Mosaic AI Model Serving") deploys MLflow-registered models as autoscaling REST endpoints. Supports custom models, Hugging Face transformers, and foundation models (Llama, Mistral, DBRX) in the same serving framework.

# Query a served model via REST import requests response = requests.post( "https://<workspace>.azuredatabricks.net/serving-endpoints/churn-classifier/invocations", headers={"Authorization": f"Bearer {DATABRICKS_TOKEN}"}, json={"dataframe_records": [{"age": 34, "tenure_months": 12, ...}]} )

Feature Store integration: Databricks Feature Store lets you define features once and reuse them across training and serving. At serving time, the endpoint looks up feature values from the online Feature Store table automatically — no need to replicate feature logic in the serving layer.

📦 Unity Catalog Models

Models in the new Unity Catalog Model Registry follow the same three-level namespace as tables: catalog.schema.model_name. Same RBAC as tables. Lineage traces which training data version produced which model version. The recommended registry for new workloads (replaces the workspace-level Model Registry).

🤖 Databricks AI / DBRX

Databricks ships foundation models (DBRX, Llama 3, Mistral) accessible via the AI Playground and the Model Serving API. For enterprise RAG, use Mosaic AI Vector Search — a managed vector database integrated with Unity Catalog Delta tables. Query with similarity search or hybrid (BM25 + dense).


12

Common Challenges & Solutions

Real production problems you will encounter — with root causes and the right fix for each.

ChallengeRoot CauseFix
Cost Overrun
Cluster left running overnight
All-Purpose clusters with no auto-termination policy Enforce auto-termination (120 min max) via Cluster Policy. Enable cost attribution tags. Use Workflow Jobs for all scheduled work.
Slow Streaming
Structured Streaming lag grows
Small file accumulation — each micro-batch writes many tiny Parquet files, slowing future reads Enable optimizeWrite on the Delta sink. Run scheduled OPTIMIZE on Bronze/Silver tables. Tune trigger(processingTime='2 minutes') to batch more records per micro-batch.
Shuffle Spill
OOM / spill to disk on joins
Insufficient executor memory for shuffle data. Large skewed keys. Switch to memory-optimised VMs (Standard_E series). Increase shuffle partitions: spark.sql.shuffle.partitions=800. Broadcast small tables. Salting for skewed keys.
Schema Mismatch
Pipeline fails on new upstream column
Source schema changed without coordination. DLT/Auto Loader rejected the new column. Configure cloudFiles.schemaEvolutionMode = "rescue" on Auto Loader. Add mergeSchema = true in Delta writes. Define DLT expectations to quarantine bad rows instead of failing.
Long Cold Starts
Short jobs blocked by 2-min cluster start
Job clusters provision VMs from scratch each run Create an Instance Pool and assign job clusters to it. Pool keeps warm VMs available — cluster start drops to ~10 seconds.
Permission Denied
Notebook can't read ADLS Gen2
Cluster not configured with Managed Identity or Service Principal. Or Unity Catalog external location not granted. Assign MI to workspace. Grant READ FILES on the external location in Unity Catalog. Verify with dbutils.fs.ls('abfss://...').
MERGE Performance
SCD MERGE slows as table grows
Delta reads all files to find matching rows — no index Partition table by a date column + Z-Order/Liquid Cluster on merge key. Reduce target table scan with partition filter in MERGE condition: AND target.dt >= '2026-01-01'.
Stale BI Data
Power BI shows yesterday's Gold table data
Scheduled refresh job failed silently or Power BI import cache not invalidated Add job alerting (email on failure). Switch Power BI to DirectQuery mode against Databricks SQL Serverless — no import cache. Add a SQL alert on MAX(updated_at) threshold.

🔍 Deep Dive: Debugging Spark Performance with Spark UI

+

Access Spark UI from the cluster page → "Spark UI" tab. The key views for performance debugging:

Jobs tab: Each Spark action (count, write, collect) creates a job. Find slow jobs by duration. Click into a job to see stages.

Stages tab: A stage is a set of parallel tasks. Look for: (1) high "GC Time" → executor memory pressure, increase heap or use larger VMs; (2) high "Shuffle Read/Write" → reduce shuffle partitions or broadcast small tables; (3) skew — one task takes 10× longer than others → repartition on a different key or salt.

Storage tab: Shows what's cached in memory. If your iterative ML loop runs slowly, cache the feature DataFrame: df.cache(); df.count().

SQL tab: For Photon queries, shows physical plan, scanned files, and pruned files. If "files pruned" is low, Z-Order or Liquid Clustering on the filter column will help dramatically.


GL

Glossary — Key Terminology

Quick-reference definitions for every term you'll encounter when setting up Azure Databricks.

DBU (Databricks Unit)
The billing unit for Databricks compute. Rate varies by workload type: All-Purpose (most expensive) > Jobs > SQL Warehouse. DBU cost is separate from Azure VM cost.
Lakehouse
Architecture pattern combining the low-cost open-format storage of a data lake with the reliability, governance, and SQL performance of a data warehouse. Databricks pioneered this term.
Delta Lake
An open-source storage framework that adds ACID transactions, schema enforcement, time travel, and upserts (MERGE) to Parquet files on cloud object storage.
Unity Catalog
Databricks' centralised governance layer providing a three-level namespace (catalog.schema.table), column-level security, data lineage, and cross-workspace table sharing.
ADLS Gen2
Azure Data Lake Storage Generation 2. Azure Blob Storage with Hierarchical Namespace (HNS) enabled. The standard underlying storage for Databricks on Azure.
Auto Loader
Databricks feature (format("cloudFiles")) for incrementally ingesting new files from cloud storage. Uses Event Grid notifications for efficient file discovery — no directory listing.
Delta Live Tables (DLT)
A declarative pipeline framework where you define Bronze/Silver/Gold tables as Python functions or SQL. DLT manages dependency ordering, retries, quality enforcement, and lineage automatically.
Photon
Databricks' native C++ vectorised query execution engine. Replaces JVM-based Spark execution for SQL workloads. Enabled automatically on SQL Warehouses; available on All-Purpose clusters with DBR 9.1+.
Instance Pool
A set of pre-allocated idle Azure VMs managed by Databricks. Clusters that draw from a pool start in seconds instead of minutes by skipping VM provisioning.
Managed Identity (MI)
An Azure AD identity assigned to a resource (e.g. Databricks workspace) that allows the resource to authenticate to Azure services without credentials. The recommended method for ADLS Gen2 access.
Service Principal (SP)
An Azure AD application identity with a client ID + secret or certificate. Used for automated processes (pipelines, CI/CD) that need to authenticate to Azure and Databricks APIs.
VNet Injection
Deploying Databricks cluster VMs into a customer-managed Azure Virtual Network instead of a Databricks-managed VNet. Required for private endpoint connectivity and network compliance.
Secret Scope
A Databricks-managed or Azure Key Vault-backed credential store. Accessed via dbutils.secrets.get(scope, key). Values are redacted in notebook output and logs.
Medallion Architecture
Data architecture pattern with three layers: Bronze (raw ingestion), Silver (cleansed/conformed), Gold (aggregated/BI-ready). Data flows forward only. Standard pattern in Databricks Lakehouse implementations.
Z-Order
A Delta Lake file layout optimisation that co-locates rows with similar column values in the same files. Reduces files scanned for point/range queries on the Z-Ordered columns.
Liquid Clustering
The successor to Z-Order in Delta Lake. Defines clustering keys on the table (CLUSTER BY); Delta incrementally reorganises data in the background without manual OPTIMIZE runs.
MERGE (Upsert)
A SQL statement that atomically updates matching rows, inserts new rows, and optionally deletes flagged rows in one operation. The primary mechanism for implementing CDC and SCD Type 1 in Delta Lake.
SCIM
System for Cross-domain Identity Management. Protocol used to sync users and groups from Azure AD to Databricks automatically. Ensures Databricks access is revoked when Azure AD accounts are disabled.
MLflow
Open-source ML lifecycle platform, fully managed in Databricks. Tracks experiments (parameters, metrics, artefacts), provides a Model Registry for versioned model promotion (Staging → Production), and integrates with Model Serving.
Cluster Policy
A JSON template (Premium tier) that constrains what users can configure when creating clusters — VM types, max workers, auto-termination, tags. Enforces governance without removing self-service.
Checkpoints (Streaming)
State stored by Structured Streaming on ADLS Gen2 to track which records have been processed. Checkpoints enable exactly-once processing and allow a stream to resume from the correct offset after a restart.
Photon Cache / Disk Cache
Databricks' local SSD cache on cluster nodes for frequently accessed Delta table files. Distinct from query result cache. Warm cache gives 3–10× faster repeated scans on the same data partitions.

Design system by Wonoments

© 2026 AI Demo Tools — companion to the Data Platform Advisor RFP tool