The Lakehouse — One Platform for Everything
Databricks invented the Lakehouse pattern to collapse the traditional two-tier architecture (data lake + data warehouse) into a single layer. All data lives in open-format files on cloud storage; Delta Lake adds warehouse-grade reliability on top.
Lakehouse vs Data Warehouse (Snowflake): Both separate compute from storage, but the Lakehouse keeps data in open Parquet/Delta files. This means your data scientists can read the same files with Python/pandas directly from ADLS Gen2, without copying data to a separate analytics store. There is no "export to data lake" step — it is the lake.
🔍 Deep Dive: Why Not a Traditional Data Lake?
Classic data lake problems: Raw files on ADLS Gen2 give you cheap storage and any-format flexibility, but with no ACID guarantees. A failing ETL job leaves partial writes visible to readers. No schema enforcement means a bad upstream schema change silently corrupts downstream pipelines. No updates — you must rewrite entire partitions to fix one row.
Delta Lake solves all of this: It wraps your Parquet files in a transaction log (_delta_log/). Every INSERT, UPDATE, MERGE, and DELETE is recorded as an atomic commit. Readers always see a consistent snapshot. Schema is enforced at write time. Old versions are retained so you can query yesterday's data or roll back bad writes.
Lakehouse vs Data Warehouse trade-off: The Lakehouse excels when your team writes Python/Spark, runs ML workloads, and wants one storage layer across structured and semi-structured data. A pure Data Warehouse (Synapse Analytics, Snowflake) wins when your team is primarily SQL analysts who need best-in-class concurrency scaling and fine-grained per-query cost control without cluster management.
✅ Check Your Understanding
Q1: A junior engineer asks why Databricks stores data as Parquet files instead of a proprietary format. What's the best answer?
Q2: An ETL job fails halfway through writing 10M rows. Without Delta Lake, what happens to downstream readers?
Workspace & Azure Infrastructure Setup
A Databricks Workspace is the top-level Azure resource — it maps to one Azure region and one Azure subscription. Understanding the network and identity setup is essential before writing a single notebook.
🏗️ VNet Injection
Deploy cluster VMs into your own VNet instead of a Databricks-managed VNet. Required for: private endpoint access to ADLS Gen2, on-premises connectivity via ExpressRoute/VPN, NSG-level traffic control. Must be configured at workspace creation — cannot change later.
🔒 Private Link
Route traffic between notebooks/clusters and the Databricks control plane over Azure Private Link. Prevents all data and control traffic from crossing the public internet. Recommended for enterprise environments with strict network compliance requirements.
👤 Managed Identity
Assign a User-Assigned Managed Identity to the Databricks workspace to access ADLS Gen2 and Key Vault without secrets. The cluster authenticates using the MI token automatically — no credentials stored in notebooks or config files.
💰 SKU Tiers
Standard tier: core Spark + notebooks + basic jobs. Premium tier: Unity Catalog, cluster policies, enhanced audit logs, SCIM provisioning, IP access lists, credential passthrough. Enterprise customers almost always need Premium.
🔍 Deep Dive: Managed Resource Group Explained
When you create a Databricks workspace, Azure automatically creates a second resource group (e.g. databricks-rg-myworkspace-xxxx) that Databricks manages on your behalf. This is where all cluster VMs, managed disks, and network infrastructure (load balancers, public IPs) live while clusters are running.
Important: You cannot manually delete resources in the managed resource group — Databricks manages their lifecycle. When a cluster terminates, the VMs are deleted (you stop paying). When the cluster starts, new VMs are provisioned fresh. This is why Databricks clusters have a ~1–3 minute cold start time.
Cost implication: VM costs accrue to your Azure subscription under the managed resource group. Databricks DBU (Databricks Unit) fees are billed separately through the Azure Marketplace. Your total cost = Azure VM cost + Databricks DBU cost.
🔍 Deep Dive: ADLS Gen2 Setup for Databricks
Storage account requirements: Enable Hierarchical Namespace (HNS) on the storage account — this turns it into ADLS Gen2. Without HNS, you get plain Blob Storage which is slower for directory-heavy operations. HNS cannot be enabled after creation.
Container layout (recommended):
Access method priority: (1) Unity Catalog storage credentials using Managed Identity — no secrets, no rotation headaches. (2) Service Principal with client secret stored in Key Vault and mounted via Databricks Secret Scope. (3) Storage Account Access Key — avoid in production (too broad, not auditable).
Clusters & Compute — Types, Sizing & Policies
Every computation in Databricks runs on a cluster — a set of Azure VMs coordinated by Apache Spark. Choosing the right cluster type and size is the single biggest lever for controlling cost and performance.
🔬 All-Purpose Cluster
Interactive clusters you start manually and share with your team. Used in notebooks for exploration, ad-hoc analysis, and development. Billed at the All-Purpose DBU rate (most expensive). Auto-terminates after N minutes of inactivity. Never use for production jobs — use Job clusters instead.
⚙️ Job Cluster
Ephemeral clusters spun up for a specific job run and destroyed after completion. Cheaper DBU rate (~50% of All-Purpose). No warm-up tax shared across jobs. Best practice: every production pipeline gets its own Job cluster. Cold start adds ~2 min to job duration.
🏊 Instance Pool
A pool of pre-allocated idle VMs that clusters draw from instead of provisioning new VMs. Eliminates the ~2 min cold start. Great for jobs triggered frequently (e.g. every 15 minutes). You pay for idle pool VMs, so size the pool to your actual concurrency.
🔌 SQL Warehouse
A dedicated compute type for Databricks SQL only — not for notebook/Spark workloads. Runs Photon (vectorised C++ engine) for fast analytical SQL. Supports Serverless mode (no cluster management, instant start) and Classic mode. Used by BI tools like Power BI and Tableau.
Common Azure VM Sizes for Databricks
Cluster sizing rule of thumb: Start with 4–8 worker nodes of DS4_v2. If shuffle spills to disk (check Spark UI → Storage tab), switch to memory-optimised (Standard_E series). If jobs are CPU-bound with large joins, try storage-optimised (Standard_L series with local NVMe).
🔍 Deep Dive: Driver vs Worker Nodes
Driver node: Runs the SparkContext, DAG scheduler, and your application code (Python main thread). All collect(), toPandas(), and print output lands here. If the driver runs out of memory, the entire application crashes. Size the driver at least as large as one worker — for notebook workloads, often larger.
Worker nodes: Execute distributed tasks — reading partitions, applying transformations, writing output. Each worker runs one or more Executors. Spark assigns each task to one CPU core on one executor. Adding workers adds parallelism.
Single-node cluster: For lightweight ML training or pandas workloads that don't distribute, a single-node cluster (driver only, no workers) avoids Spark overhead and is cheaper. Databricks notebooks can run pandas, scikit-learn, and even PyTorch on single-node clusters.
🔍 Deep Dive: Cluster Policies
Cluster Policies (Premium tier only) are JSON templates that constrain what users can configure when creating clusters. Use them to enforce governance: maximum cluster size, allowed VM types, mandatory auto-termination, required tags for cost allocation.
Assign a policy to a team (e.g. "Data Analysts") so they can create clusters without spinning up 100-node monsters. Admins are exempt from policies. You can also set custom tags as fixed values in policies to enforce cost attribution by team or project.
✅ Check Your Understanding
Q1: A production pipeline runs hourly. Should it use an All-Purpose cluster or a Job cluster?
Q2: A short job runs every 5 minutes. Cold start takes 2 minutes. What's the best fix?
Delta Lake — ACID, Schema, Time Travel & Optimisation
Delta Lake is the storage layer that transforms plain Parquet files into a reliable, queryable table format. Every Databricks table you create is a Delta table by default. Understanding how it works determines how well your pipelines perform at scale.
⚡ ACID Transactions
Every write (INSERT, UPDATE, DELETE, MERGE) is recorded in _delta_log/ as a JSON commit entry. Readers always see the last complete transaction. Concurrent writes use optimistic concurrency — conflict detection prevents data corruption without locking tables.
📋 Schema Enforcement
Delta rejects writes that don't match the table schema by default. Add mergeSchema or run ALTER TABLE to evolve the schema intentionally. This prevents a bad upstream source from silently corrupting your Silver/Gold tables.
⏱️ Time Travel
Query any previous version of a Delta table by version number or timestamp. Data retention controlled by delta.logRetentionDuration (default: 30 days). Essential for auditing, rollbacks, and reproducing ML training datasets.
🔀 MERGE (Upsert)
The most common CDC pattern. MERGE matches rows on a key, UPDATEs matches, INSERTs new rows, and optionally DELETEs flagged rows — all in one atomic operation. Equivalent to SCD Type 1 in a single SQL statement.
🔍 Deep Dive: The Transaction Log (_delta_log/)
The _delta_log/ directory sits at the root of every Delta table. Each write creates a JSON file numbered sequentially: 00000000000000000000.json, 00000000000000000001.json, etc. Every 10 commits, Delta checkpoints the state to a Parquet file for fast catch-up.
What the log records: Which files were added, which were removed, schema at the time of commit, operation metadata (MERGE, INSERT, etc.), transaction ID, and statistics (min/max values per column — used for data skipping).
Data skipping: When a query has a filter (WHERE clause), Delta reads the min/max stats in the log to skip entire files that can't contain matching rows. A well-partitioned and Z-Ordered table can skip 90%+ of files, making point-queries fast without an index.
🔍 Deep Dive: OPTIMIZE, Z-Order & Liquid Clustering
OPTIMIZE: Small files accumulate from frequent streaming writes. OPTIMIZE compacts them into larger files (target: 1 GB each) to reduce Spark task overhead and improve scan throughput. Run OPTIMIZE on Silver/Gold tables after major write operations.
Z-Order: Multi-dimensional clustering that physically arranges data so rows with similar values on the Z-Order columns land in the same files. Dramatically reduces data read for point lookups and range queries on the clustered columns. Limited to 4 columns. Must re-run OPTIMIZE when new data arrives — not automatic.
Liquid Clustering (newer): Replaces Z-Order with an incremental background clustering algorithm. You define the clustering keys once (CLUSTER BY) and Delta automatically reorganises data as you write. No manual OPTIMIZE ZORDER scheduling needed. Currently the recommended approach for new tables on Databricks Runtime 13.3+.
VACUUM — removing old files: Delta retains deleted/old files for Time Travel. Run VACUUM tablename RETAIN 7 HOURS to remove files older than 7 hours. Default retention is 7 days. Never vacuum below your Time Travel window, or time-travel queries will fail.
Ingestion Patterns — Auto Loader, DLT & Streaming
Getting data into the Lakehouse reliably is the foundation of every pipeline. Databricks offers three primary ingestion mechanisms, each optimised for different arrival patterns and reliability requirements.
| Pattern | Best for | Latency | Exactly-once? | Managed by |
|---|---|---|---|---|
| Auto Loader | New files arriving in ADLS Gen2 (CSV, JSON, Parquet, Avro) | Minutes (batch trigger) | ✅ Yes | Databricks |
| Structured Streaming | Event Hub / Kafka real-time streams | Seconds | ✅ Yes (w/ checkpoints) | You (or DLT) |
| Delta Live Tables (DLT) | Declarative pipeline with auto-retries, quality constraints | Minutes (triggered) or Seconds (continuous) | ✅ Yes | Databricks |
| COPY INTO | One-time or idempotent bulk load from ADLS Gen2 | Batch | ✅ Yes (tracks loaded files) | You |
🔍 Deep Dive: Auto Loader
Auto Loader uses Azure Event Grid notifications on the ADLS Gen2 container to detect new file arrivals without scanning the entire directory. This is far more efficient than listing all files on each run — critical when you have millions of files in a landing zone.
Schema inference: Auto Loader can infer schema from the first N files and store it in schemaLocation. When new columns appear, you can configure cloudFiles.schemaEvolutionMode to auto-add columns, raise an error, or rescue into a _rescued_data column (safest choice for production).
🔍 Deep Dive: Delta Live Tables (DLT)
DLT is Databricks' declarative pipeline framework. You define tables using Python decorators or SQL; DLT handles ordering, dependencies, retries, monitoring, and lineage automatically.
Expectations (data quality rules): @dlt.expect warns on violations; @dlt.expect_or_drop silently drops invalid rows; @dlt.expect_or_fail halts the pipeline. DLT tracks violation counts over time, visible in the pipeline UI — you get a data quality dashboard for free.
Medallion Architecture — Bronze, Silver, Gold
The Medallion pattern is Databricks' standard approach to layering data quality across the Lakehouse. Each layer has a clear contract — what goes in, what comes out, and who may write to it.
- Exact copy of source data
- No transformations applied
- Schema as received
- Append-only — never update
- Long retention (1+ years)
- Written by: ingestion pipelines
- Data type casting & validation
- Deduplication applied
- PII masked or hashed
- Null handling enforced
- Business keys available
- Written by: transformation jobs
- Business metric calculations
- Denormalised for query speed
- Pre-aggregated dimensions
- Aligned to business domains
- Read by: BI tools, APIs, ML
- Written by: analytics engineers
Key rule: Data flows only forward — Bronze → Silver → Gold. A Gold table never writes back to Silver. If upstream data changes (e.g. a historical correction), reprocess Bronze and cascade forward. This unidirectional flow makes debugging and re-processing predictable.
🔍 Deep Dive: Implementing Medallion with Unity Catalog
Map each Medallion layer to a separate Unity Catalog schema (database) within a catalog. This makes access control simple — you grant different teams read access to different schemas.
Write permissions: Ingestion service principals get WRITE on Bronze only. ETL service principals get READ on Bronze and WRITE on Silver. Analytics engineers get READ on Silver and WRITE on Gold. This strict layering prevents accidental overwrites upstream.
Unity Catalog — Governance, Lineage & Access Control
Unity Catalog is Databricks' unified data governance layer. It provides a three-level namespace, centralised permission management, column-level security, and automatic data lineage — all across all workspaces in an account.
🗂️ Three-Level Namespace
Every object is referenced as catalog.schema.table. Multiple workspaces in the same region share one metastore — a table defined in workspace A is immediately discoverable in workspace B, with the same permissions. Eliminates per-workspace Hive metastores.
🔗 Data Lineage
Unity Catalog automatically captures column-level lineage for every SQL query and DLT pipeline. You can trace which upstream tables and columns contributed to any downstream table. No instrumentation required — lineage is derived from query plans.
🔐 Column-Level Security
Apply column masks (dynamic data masking) and row filters using SQL functions. A GDPR-sensitive column (e.g. email) can return a hash for most users and the real value only for the DPO group — enforced at the metastore level, not in each query.
🏷️ Tags & Search
Tag catalogs, schemas, tables, and columns with business metadata (PII, domain, owner). The Unity Catalog UI provides a searchable data catalogue. Integrate with Azure Purview for enterprise-wide discovery across Azure services and Databricks.
🔍 Deep Dive: Storage Credentials & External Locations
Unity Catalog separates storage credentials (authentication to ADLS Gen2) from external locations (which paths are accessible). This is how you grant Databricks access to your ADLS Gen2 without embedding storage account keys in notebooks.
Users reading prod.silver.customers never touch ADLS credentials directly — Unity Catalog's credential vending hands the cluster a short-lived token for the specific path. Revoke access centrally in Unity Catalog without rotating storage keys.
✅ Check Your Understanding
Q1: Your company has 4 Databricks workspaces across dev, staging, and prod. With Unity Catalog, how many metastores do you need?
Azure Security — Identity, Secrets & Network
Databricks on Azure integrates deeply with Azure security primitives. Getting the identity and secret management right from day one prevents the most common audit findings.
🔑 Service Principal vs Managed Identity
Service Principal (SP): An Azure AD application identity with a client ID and secret (or certificate). The secret must be rotated. Store in Key Vault, reference via Databricks Secret Scope. Managed Identity (MI): No credentials to manage — Azure assigns and rotates tokens automatically. Preferred for ADLS Gen2 access where supported by Unity Catalog.
🔒 Databricks Secret Scopes
Secret Scopes are name-indexed credential stores accessible from notebooks/jobs via dbutils.secrets.get(). Two backends: Databricks-managed (encrypted in Databricks' control plane) or Azure Key Vault-backed (secrets live in your Key Vault, Databricks fetches them read-only). Key Vault-backed is the enterprise recommendation — you control the lifecycle.
🌐 IP Access Lists
Restrict Databricks workspace UI and REST API access to approved IP ranges (corporate office IPs, VPN egress). Premium tier only. Prevents notebooks from being accessed from outside your network even with valid credentials.
👥 SCIM Provisioning
Sync users and groups from Azure AD to Databricks automatically using SCIM (System for Cross-domain Identity Management). When a user leaves the company and their Azure AD account is disabled, their Databricks access is revoked within minutes — no manual offboarding.
🔍 Deep Dive: Accessing Key Vault from Notebooks
The recommended pattern for secrets in production: store credentials in Azure Key Vault, create a Key Vault-backed Secret Scope in Databricks, reference via dbutils.secrets. The secret value is never printed in notebook output — Databricks redacts it in logs.
What NOT to do: Never hardcode a connection string or password in a notebook cell, even in dev. Notebooks are versioned in the Databricks workspace and can be shared unintentionally. Always use Secret Scopes.
Audit logs: Enable Databricks diagnostic logging to Azure Monitor / Log Analytics. Captures notebook commands, cluster starts/stops, secret reads, permission changes, and login events. Essential for compliance (GDPR audit trails, SOC 2 evidence).
Workflows & Orchestration — Jobs, Tasks & Scheduling
Databricks Workflows is the native orchestration layer. It replaces the need for an external orchestrator (Airflow, Azure Data Factory) for Databricks-native workloads — though ADF or Airflow are still common for cross-service orchestration.
📋 Multi-Task Jobs
A Workflow Job can contain multiple Tasks connected by dependencies (DAG). Each task can run a notebook, Python script, JAR, DLT pipeline, dbt project, SQL query, or call a REST API. Tasks run sequentially or in parallel based on depends_on wiring.
⏰ Triggers
Schedule with cron or interval. Trigger on file arrival (file-arrival trigger on ADLS Gen2 path). Trigger via REST API (POST /api/2.1/jobs/run-now) from ADF, GitHub Actions, or any HTTP client. Trigger from another job's completion via webhooks.
🔁 Retries & Repair
Configure per-task retry count and retry interval. After a job run failure, use "Repair Run" to re-run only failed tasks (and their downstream dependents) without re-running successful tasks. Critical for expensive pipelines where early tasks shouldn't re-run.
📊 Job Compute
Each task specifies its compute: a new Job cluster (ephemeral, cheapest), an existing All-Purpose cluster (warm but expensive), an Instance Pool cluster (fast start), or a SQL Warehouse (SQL tasks). Mix and match — notebook tasks can use one cluster size while a heavy transform uses another.
🔍 Deep Dive: Databricks vs Azure Data Factory
Use Databricks Workflows when: All tasks are Databricks-native (notebooks, DLT, SQL). You want fine-grained task-level retry, repair runs, and cost attribution. Your team lives in Databricks and wants a single UI for pipeline monitoring.
Use Azure Data Factory (ADF) when: Your pipeline orchestrates multiple Azure services — Synapse, Blob Storage copy, SQL Database, external REST APIs. You need data flow (visual ETL without Spark). Your organisation already has ADF expertise and invested pipelines. ADF can call Databricks notebooks and jobs via the Databricks linked service.
Hybrid pattern (common in enterprise): ADF handles the "outer" orchestration — copy from source systems into ADLS Gen2, trigger Databricks job, wait for completion, trigger downstream Synapse SQL or Data Export. Databricks Workflows handles the "inner" orchestration — the Medallion transformation pipeline.
Databricks SQL — SQL Warehouses, BI & Photon
Databricks SQL is the BI-facing surface of the Lakehouse. It provides a SQL Warehouse (serverless or classic compute), a query editor, dashboards, and alerts — all reading from Delta tables in Unity Catalog.
⚡ Photon Engine
Photon is a C++ vectorised query engine that replaces Spark's JVM-based execution for SQL workloads. 2–8× faster on analytical queries (aggregations, joins, sorts). Enabled automatically on SQL Warehouses. Available on All-Purpose clusters with DBR 9.1+. No code changes required.
🖥️ Serverless SQL Warehouse
Serverless SQL Warehouses start in under 5 seconds (vs ~2–3 min for Classic). Databricks manages the compute pool — no cluster VM provisioning in your subscription. Scales to zero when idle. Charged per query second of compute. Best for: BI tools, ad-hoc SQL exploration, variable-load dashboards.
📊 BI Connector (Power BI)
Power BI connects to Databricks SQL Warehouses via the native Databricks connector (Partner Connect or direct server hostname). Uses ODBC under the hood. Supports DirectQuery mode — queries run live against Delta tables with Photon. No data import/extract needed.
🔔 Alerts
Create scheduled queries that check a metric (e.g. SELECT COUNT(*) FROM silver.failed_events WHERE dt = current_date()) and fire a notification (email, Slack, PagerDuty) when the result exceeds a threshold. Lightweight data quality alerting without external tooling.
Cost model: SQL Warehouse DBU rate is separate from All-Purpose and Jobs rates. Serverless SQL Warehouses charge per-second of query execution (not idle time). Classic Warehouses charge for VM uptime even when idle. For spiky BI load, Serverless is almost always cheaper. For sustained 8-hour BI workloads, Classic can be comparable.
🔍 Deep Dive: Query Result Caching
Databricks SQL caches query results for up to 24 hours. If an identical query (same SQL, same table version) is re-run within the window, the cached result is returned instantly — zero compute cost. The cache is invalidated when the underlying Delta table version changes.
For Power BI DirectQuery dashboards, this means: the first user to open a dashboard page runs the query and populates the cache. Subsequent users in the next 24 hours get instant results from cache as long as the data hasn't been refreshed. Design your Gold tables to write in batch (e.g. once per hour) to maximise cache hit rate rather than streaming every row individually.
Disk cache (Photon cache): Separate from query result cache. Photon caches frequently-accessed Delta table files in SSD on the warehouse nodes. Warm cache = 3–10× faster repeated scans on the same data. Cold-start performance of a new warehouse is lower for the first few queries while the Photon cache fills.
MLflow & AI — Experiment Tracking & Model Serving
Databricks ships a fully managed MLflow instance in every workspace — no setup required. It tracks experiments, stores model artefacts, manages model versions, and serves models as REST endpoints.
Silver Delta tables
MLflow autolog
Staging → Production
REST endpoint
🔍 Deep Dive: MLflow Tracking & Autolog
MLflow Tracking records experiment runs — each run captures parameters, metrics (loss curves, accuracy), artefacts (model files, plots), and environment info. Autolog does this automatically for scikit-learn, XGBoost, PyTorch, and many other frameworks.
🔍 Deep Dive: Model Serving (Mosaic AI Model Serving)
Databricks Model Serving (formerly "Mosaic AI Model Serving") deploys MLflow-registered models as autoscaling REST endpoints. Supports custom models, Hugging Face transformers, and foundation models (Llama, Mistral, DBRX) in the same serving framework.
Feature Store integration: Databricks Feature Store lets you define features once and reuse them across training and serving. At serving time, the endpoint looks up feature values from the online Feature Store table automatically — no need to replicate feature logic in the serving layer.
📦 Unity Catalog Models
Models in the new Unity Catalog Model Registry follow the same three-level namespace as tables: catalog.schema.model_name. Same RBAC as tables. Lineage traces which training data version produced which model version. The recommended registry for new workloads (replaces the workspace-level Model Registry).
🤖 Databricks AI / DBRX
Databricks ships foundation models (DBRX, Llama 3, Mistral) accessible via the AI Playground and the Model Serving API. For enterprise RAG, use Mosaic AI Vector Search — a managed vector database integrated with Unity Catalog Delta tables. Query with similarity search or hybrid (BM25 + dense).
Common Challenges & Solutions
Real production problems you will encounter — with root causes and the right fix for each.
| Challenge | Root Cause | Fix |
|---|---|---|
| Cost Overrun Cluster left running overnight |
All-Purpose clusters with no auto-termination policy | Enforce auto-termination (120 min max) via Cluster Policy. Enable cost attribution tags. Use Workflow Jobs for all scheduled work. |
| Slow Streaming Structured Streaming lag grows |
Small file accumulation — each micro-batch writes many tiny Parquet files, slowing future reads | Enable optimizeWrite on the Delta sink. Run scheduled OPTIMIZE on Bronze/Silver tables. Tune trigger(processingTime='2 minutes') to batch more records per micro-batch. |
| Shuffle Spill OOM / spill to disk on joins |
Insufficient executor memory for shuffle data. Large skewed keys. | Switch to memory-optimised VMs (Standard_E series). Increase shuffle partitions: spark.sql.shuffle.partitions=800. Broadcast small tables. Salting for skewed keys. |
| Schema Mismatch Pipeline fails on new upstream column |
Source schema changed without coordination. DLT/Auto Loader rejected the new column. | Configure cloudFiles.schemaEvolutionMode = "rescue" on Auto Loader. Add mergeSchema = true in Delta writes. Define DLT expectations to quarantine bad rows instead of failing. |
| Long Cold Starts Short jobs blocked by 2-min cluster start |
Job clusters provision VMs from scratch each run | Create an Instance Pool and assign job clusters to it. Pool keeps warm VMs available — cluster start drops to ~10 seconds. |
| Permission Denied Notebook can't read ADLS Gen2 |
Cluster not configured with Managed Identity or Service Principal. Or Unity Catalog external location not granted. | Assign MI to workspace. Grant READ FILES on the external location in Unity Catalog. Verify with dbutils.fs.ls('abfss://...'). |
| MERGE Performance SCD MERGE slows as table grows |
Delta reads all files to find matching rows — no index | Partition table by a date column + Z-Order/Liquid Cluster on merge key. Reduce target table scan with partition filter in MERGE condition: AND target.dt >= '2026-01-01'. |
| Stale BI Data Power BI shows yesterday's Gold table data |
Scheduled refresh job failed silently or Power BI import cache not invalidated | Add job alerting (email on failure). Switch Power BI to DirectQuery mode against Databricks SQL Serverless — no import cache. Add a SQL alert on MAX(updated_at) threshold. |
🔍 Deep Dive: Debugging Spark Performance with Spark UI
Access Spark UI from the cluster page → "Spark UI" tab. The key views for performance debugging:
Jobs tab: Each Spark action (count, write, collect) creates a job. Find slow jobs by duration. Click into a job to see stages.
Stages tab: A stage is a set of parallel tasks. Look for: (1) high "GC Time" → executor memory pressure, increase heap or use larger VMs; (2) high "Shuffle Read/Write" → reduce shuffle partitions or broadcast small tables; (3) skew — one task takes 10× longer than others → repartition on a different key or salt.
Storage tab: Shows what's cached in memory. If your iterative ML loop runs slowly, cache the feature DataFrame: df.cache(); df.count().
SQL tab: For Photon queries, shows physical plan, scanned files, and pruned files. If "files pruned" is low, Z-Order or Liquid Clustering on the filter column will help dramatically.
Glossary — Key Terminology
Quick-reference definitions for every term you'll encounter when setting up Azure Databricks.
format("cloudFiles")) for incrementally ingesting new files from cloud storage. Uses Event Grid notifications for efficient file discovery — no directory listing.dbutils.secrets.get(scope, key). Values are redacted in notebook output and logs.CLUSTER BY); Delta incrementally reorganises data in the background without manual OPTIMIZE runs.Design system by Wonoments
© 2026 AI Demo Tools — companion to the Data Platform Advisor RFP tool