Debugging & recovery — diagnostics, job repair, parameter overrides debugged jobs and queries. Lakeflow Declarative Pipelines (Lakeflow Spark Declarative Pipelines) have their own debugging tools, because a pipeline is declared — you don't step through it, you validate the graph and read its self-record.

The spine

Beat 1 — Validate: check the code without processing data

You've added new table definitions and transformation logic to a pipeline notebook and want to catch mistakes before a full run that reads real data.

Predict: how do you check for syntax/definition errors without paying for a full data-processing run?

…

The "Validate" option. It checks the pipeline code for syntax errors and resolves the dependency graph without actually processing any data — a fast, cheap correctness gate. Run Validate after editing definitions; only do a full Update once it's clean.

Lock it. Validate = syntax + graph check, no data processed. Use it before a full pipeline update.

Beat 2 — debug a run: the event log + Spark UI

When a pipeline does run and something's wrong, you read its structured self-record — the event log from The pipeline event log — where a Lakeflow pipeline records itself: event_log('<id>'), filter event_type = 'flow_progress' for per-flow status, errors, and data-quality results (passed_records/failed_records). For the Spark-level detail inside a flow (a slow or failing transform), drop to the Spark UI as in Debugging & recovery — diagnostics, job repair, parameter overrides. So: event log for pipeline/flow-level ("which flow failed, how many rows, which expectation"), Spark UI for stage/task-level ("why that transform was slow").

Lock it. Debug LDP: Validate (pre-run) → event log (flow-level) → Spark UI (stage-level).

The dials (skim now; return when a question needs one)

◆ Development vs Production mode

A pipeline runs in Development or Production mode. Development mode keeps the cluster alive between updates (fast iteration) and doesn't retry on failure (so you see the error immediately) — the mode you debug in. Production mode terminates compute after each run and retries on failure. Debug in development, deploy in production.

◆ Capture & version pipeline settings

To version a pipeline's config as code (or clone it), use the CLI get to capture an existing pipeline's settings into JSON, then remove the pipeline_id (that's unique to the original) and rename it, and feed that into a create command for the new pipeline. (This is the pipeline analogue of bundle generate in Declarative Automation Bundles — deploying Databricks as code — capture existing config → version it → recreate.)

Takeaways (rebuild it from these)

1. Validate checks pipeline syntax + dependency graph without processing data — run it before a full update.
2. Debug a run with the event log (flow_progress: status, errors, DQ metrics — The pipeline event log — where a Lakeflow pipeline records itself) for flow-level, Spark UI for stage/task-level.
3. Development mode (cluster stays warm, no auto-retry) is where you debug; Production mode (ephemeral, retries) is where you deploy.
4. Capture/version pipeline config: CLI get → strip pipeline_id + rename → create.

Before you move on — say these without scrolling up

1. Check a pipeline's code for errors without a costly data run — which option, and what does it (not) do?
2. A flow failed — which surface for flow-level vs stage-level detail?
3. Development vs Production pipeline mode — which do you debug in, and why?

Next: packaging and deploying pipelines/jobs as code → Declarative Automation Bundles — deploying Databricks as code.