Reference card (not a deep concept). One idea: a notebook is configured from outside, and each source of outside values has exactly one correct API. Three sources, three APIs — plus one security behaviour that surprises people.
Reading a parameter the Job passed in → widgets
dbutils.widgets.text("date", "null") # declare the widget (with a default)
date = dbutils.widgets.get("date") # read the value the Job passed
Why the others are wrong (this exact set is tested):
| Attempt | Why it fails |
|---|---|
spark.conf.get("date") | reads Spark runtime config, not job/notebook parameters |
input() | blocks for interactive stdin — never works in a scheduled job |
sys.argv[1] | for Python script tasks (spark_python_task), not notebook tasks |
dbutils.notebooks.getParam("date") | not a real API — the namespace is dbutils.notebook (singular), with run/exit, no getParam |
Passing a value between tasks → task values (not widgets)
Recall Jobs & orchestration — multi-task, dependencies, control flow: dbutils.jobs.taskValues.set(key, value) / .get(taskKey, key). Distinct from widgets (which read job-level params). Task values come back as strings — cast to int/float.
Reading a secret → dbutils.secrets (and the REDACTED surprise)
pw = dbutils.secrets.get(scope="db_creds", key="jdbc_password")
print(pw) # prints REDACTED — never the real value
The behaviour to lock in (a favourite question): the retrieved secret is a real, usable string — a DB connection built with it succeeds. But Databricks auto-redacts secret values in any notebook output/logs, so printing one shows the literal REDACTED. No interactive box appears; nothing is written to DBFS. Never hardcode credentials; access is controlled by secret ACLs on the scope.
Sharing values across languages (Python ↔ SQL)
A notebook can mix %python and %sql cells, but they don't share a variable namespace — a Python variable (a list, a string, a DataFrame) is not visible to a %sql cell, and vice-versa. The bridges:
- Python DataFrame → SQL: register it as a view —
df.createOrReplaceTempView("t")— then%sql SELECT * FROM t. A raw Python list is not a table; you'd first build a DataFrame (spark.createDataFrame(...)) and register it. - Python scalar → SQL: inject it with an f-string into
spark.sql(f"… WHERE d = '{date}'"), or set a Spark conf and read it with${…}. You cannot reference the bare Python name inside SQL. - SQL result → Python:
spark.sql("…")returns a DataFrame you can keep working with in Python.
The tell: "a %sql cell can't see my Python list/variable" → it never could — cross-language sharing goes through a temp view or an injected/config value, not shared scope.
The one-line separators
- Job param into a notebook →
dbutils.widgets.get. - Value between tasks →
dbutils.jobs.taskValues(strings — cast). - Secret →
dbutils.secrets.get(works fine; printsREDACTED).
Recall (say without scrolling up)
- Three outside sources of values, three APIs — match them.
- You
print()a secret — what shows, and does a connection built from it still work? - Why is
spark.conf.get/sys.argvwrong for a notebook job param?