What Is Supported
We implement a Databricks-style dbutils object for Yeedu Spark notebooks and jobs. Six namespaces, roughly seventy per cent of the calls a real Databricks codebase actually uses, and no pretending about the rest.
This page is the inventory. Call by call, what we implement, what we don't, and what you do instead.
Getting a dbutils object
| Language | How it is obtained | Notes |
|---|---|---|
| Python | from dbutils import dbutils | Explicit import |
| Python | Already in scope as a builtin | Available once sitecustomize.py from the dbutils package is installed into the runtime's site-packages |
| Scala / Java | DBUtils.get() | Root entry point is com.databricks.sdk.scala.dbutils.DBUtils |
With sitecustomize.py installed, the runtime also exposes YeeduDeltaTable and its alias YeeduDeltaTables as builtins, so Delta table handles work without an import too.
The object itself is a lazy proxy. Importing it starts no Spark session and touches no JVM. Each namespace is constructed the first time you reach for it, which is why a Python REPL that imports dbutils doesn't spend ten seconds booting Spark before it shows you a prompt.
| Namespace accessed | What gets loaded |
|---|---|
dbutils.fs | dbutils.fs_utils.fs.FS, which creates or attaches a SparkSession |
dbutils.secrets | dbutils.secrets.secrets.Secrets |
dbutils.widgets | dbutils.widgets.widgets.Widgets |
dbutils.notebook | dbutils.notebook.notebook_utils.NotebookUtils |
dbutils.library | dbutils.library.library.Library |
dbutils.jobs | dbutils.jobs.task_values.TaskValues, exposed as dbutils.jobs.taskValues |
Six names resolve. Everything else raises AttributeError.
dbutils.jobs.taskValues is the path that works. Databricks code written that way runs unchanged. A bare top-level dbutils.taskValues does not resolve, because there is no such attribute on the proxy.
How to read the support matrix
The Migration check column says whether our job-readiness analyser recognises the call. That analyser compares each dbutils call it finds against a fixed set, and anything outside the set lands in job_readiness_report.csv as a blocker.
The set is deliberately conservative and slightly smaller than what we actually implement, which is why a handful of calls below are marked "Supported" and "Not on the list" at the same time: dbutils.widgets.set, dbutils.widgets.getArgument, dbutils.secrets.list, dbutils.secrets.init, every namespace's help, and YeeduDeltaTable.forName all run perfectly well and none of them appear in the analyser's allowlist, so a job using them gets flagged as not ready even though it will execute. Treat those flags as false positives.
The reverse never happens. Nothing on the allowlist is missing from the runtime.
dbutils.fs
| Call | Python signature | Status | Migration check | Notes |
|---|---|---|---|---|
ls | ls(path, recurse=False) | Supported | Recognised | Returns file-info objects with path, name, size, modificationTime, isDir, isFile. recurse=True walks subdirectories |
head | head(file, maxBytes=65536) | Supported | Recognised | Raises if file is a directory. Reads UTF-8, stopping on the byte budget |
put | put(file, contents, overwrite=False) | Supported | Recognised | Raises if the target exists and overwrite=False |
cp | cp(src, dst, recurse=False, overwrite=False) | Supported | Recognised | Raises if src is a directory and recurse=False |
mv | mv(src, dst, recurse=False) | Supported | Recognised | Implemented as copy-then-delete |
rm | rm(path, recurse=False) | Supported | Recognised | Raises IOError on a non-empty directory when recurse=False |
mkdirs | mkdirs(path) | Supported | Recognised | Creates parents as needed |
help | help(function_name=None) | Supported | Not on the list | With no argument, lists the available fs functions |
mount | Not implemented | Not on the list | No mount API exists in Yeedu dbutils | |
mounts | Not implemented | Not on the list | ||
unmount | Not implemented | Not on the list | ||
refreshMounts | Not implemented | Not on the list | ||
updateMount | Not implemented | Not on the list | ||
cacheFiles / cacheTable | Not implemented | Not on the list | ||
uncacheFiles / uncacheTable | Not implemented | Not on the list |
Path handling
| Path form | How it resolves |
|---|---|
/files/... | Yeedu workspace file, served over the Yeedu REST API, for put, cp, mv, rm, and mkdirs only |
s3://... | Rewritten to s3a:// before the Hadoop FileSystem call |
Anything else (abfss://, file:/, hdfs://, bare paths) | Resolved through the Hadoop FileSystem API using the session's Hadoop configuration |
cp and mv bridge those two worlds, and all four direction combinations work: workspace to workspace, external to workspace, workspace to external, external to external. When the destination is a directory, or ends with /, the source filename is appended.
ls and head have no /files/ branch, in either the Python or the Scala implementation. They always resolve through the Hadoop FileSystem API. Listing or previewing a Yeedu workspace file by its /files/ path is not covered by those two calls.
dbutils.widgets
| Call | Python signature | Status | Migration check | Notes |
|---|---|---|---|---|
text | text(name, default_value="", label=None) | Supported | Recognised | label defaults to name |
dropdown | dropdown(name, default_value, choices, label=None) | Supported | Recognised | |
combobox | combobox(name, default_value, choices, label=None) | Supported | Recognised | |
multiselect | multiselect(name, default_value, choices, label=None) | Supported | Recognised | |
get | get(name) | Supported | Recognised | Raises KeyError if the widget does not exist, matching Databricks |
getAll | getAll() | Supported | Recognised | Returns {name: value} |
remove | remove(name) | Supported | Recognised | Idempotent; a missing widget is not an error |
removeAll | removeAll() | Supported | Recognised | |
set | set(name, value) | Supported | Not on the list | A Yeedu addition, absent from the Databricks dbutils.widgets API |
getArgument | getArgument(name, default=None) | Supported | Not on the list | Does not create a widget. Raises KeyError if the widget is absent and no default is given |
help | help(function_name=None) | Supported | Not on the list |
dbutils.widgets.get resolves a value from the first source that has one.
| Priority | Source | Set by |
|---|---|---|
| 1 | Runtime bindings | The arguments passed to dbutils.notebook.run in the parent notebook |
| 2 | Airflow XCom | The surrounding Airflow task context |
| 3 | Widget backend state | Widgets created or edited in the notebook itself |
Values from sources 2 and 3 containing {{ ... }} are template-resolved before they come back. Runtime bindings are returned literally. Requests against the widget backend retry up to five times before a missing widget is reported.
Arguments passed to dbutils.notebook.run are stringified. None becomes an empty string, and every key and value is coerced with str().
dbutils.secrets
| Call | Python signature | Status | Migration check | Notes |
|---|---|---|---|---|
get | get(scope, key) | Supported | Recognised | Returns a SecretValue, a str subclass whose repr is [REDACTED] |
getBytes | getBytes(scope, key) | Supported | Recognised | Returns a SecretBytes, a bytes subclass whose str and repr are [REDACTED] |
listScopes | listScopes() | Supported | Recognised | |
list | list(scope) | Supported | Not on the list | Lists secret names within a scope |
init | init(host=None, token=None) | Supported | Not on the list | Not part of the Databricks API. Optional, since the namespace auto-initialises from configuration on first use |
help | help(function_name=None) | Supported | Not on the list |
Secrets are fetched from the Databricks Secrets API at /api/2.0/secrets using the configured host and token. Your scopes and keys stay exactly where they are.
We also mask them.
| Behaviour | Detail |
|---|---|
print() masking | The package replaces builtins.print, so a retrieved secret value comes out as [REDACTED] |
| Notebook cell output masking | With IPython present, the display formatter's text/plain output is wrapped and masked the same way |
| Explicit reveal | SecretValue.reveal() returns the real str; SecretBytes.reveal() returns the real bytes |
| String concatenation | SecretValue + other returns a plain str, so concatenations are not themselves registered as secrets |
SecretBytes concatenation | Concatenating with a str raises TypeError. Use bytes |
Masking is substring-based over values the process has already retrieved. It hides secrets from print and from notebook cell output. It is not a guarantee against every exfiltration path.
dbutils.notebook
| Call | Python signature | Status | Migration check | Notes |
|---|---|---|---|---|
run | run(path, timeout_seconds, arguments=None) | Supported | Recognised | See the constraints below |
exit | exit(msg="") | Supported | Recognised | Inside a run, raises an internal CleanExit so the parent receives msg. Outside a run, returns msg |
entry_point.getCurrentBindings | getCurrentBindings() | Supported | Not on the list | Returns the current runtime bindings dictionary |
getContext | Not implemented | Not on the list |
dbutils.notebook.run is implemented, and it's the call most people expect to lose. The child notebook is driven over a WebSocket connection to the Yeedu notebook kernel. In Scala it's overloaded for a java.util.Map, a Scala Map, and a JSON String.
Four constraints are enforced in code, and one of them differs from Databricks.
| Rule | Behaviour |
|---|---|
path is required | Omitting it raises ValueError |
| Extension inference | If the final path segment has no ., .ipynb is appended |
timeout_seconds is required | 0, None, or omitting it raises ValueError. Unlike Databricks, there is no "no timeout" mode |
arguments type | A dict, a JSON string, or None. Anything else raises TypeError; malformed JSON raises ValueError |
dbutils.jobs.taskValues
| Call | Python signature | Status | Migration check | Notes |
|---|---|---|---|---|
set | set(key, value) | Supported | Recognised | value must be JSON-serialisable, otherwise TypeError |
get | get(taskKey, key, default=None, debugValue=None) | Supported | Recognised | On a missing value, returns debugValue, then default, otherwise raises KeyError |
Task values are stored as Airflow XCom entries through the Yeedu API, so they need Airflow context to be configured.
| Constraint | Value |
|---|---|
| Maximum serialised payload | 49152 bytes, 48 KiB; exceeding it raises ValueError |
| Requires | dag_id, dag_run_id, and task_id to be resolvable |
dbutils.library
| Call | Python signature | Status | Migration check | Notes |
|---|---|---|---|---|
restartPython | restartPython() | Supported | Recognised | Restarts the active IPython kernel via IPython.Application.instance().kernel.do_shutdown(restart=True). Raises if no IPython kernel is present |
YeeduDeltaTable.forName | forName(spark, table) | Supported, a Yeedu addition | Not on the list | Resolves the table's storage location with DESCRIBE DETAIL, then returns DeltaTable.forPath |
install, installPyPI | Not implemented | Not on the list | There is no library install API. Only restartPython exists |
The Scala and Java Library.restartPython() takes a different route. It calls the Yeedu REST endpoint workspace/{workspace_id}/notebook/run/{notebook_id}/kernel/restart rather than driving IPython.
Namespaces we do not implement
| Databricks namespace | Alternative |
|---|---|
dbutils.data | No equivalent. Use Spark SQL or DataFrame operations directly |
dbutils.credentials | No equivalent |
dbutils.api | No equivalent |
dbutils.meta | No equivalent |
dbutils.preview | No equivalent |
dbutils.taskValues (top-level) | Use dbutils.jobs.taskValues |
Working code
from dbutils import dbutils
# Filesystem
for info in dbutils.fs.ls("/data"):
print(info.path, info.size, info.isDir)
dbutils.fs.put("/files/reports/summary.txt", "row_count=42", overwrite=True)
dbutils.fs.cp("/files/reports/summary.txt", "abfss://container@account.dfs.core.windows.net/archive/")
# Widgets
dbutils.widgets.text("input", "default_value", "Input Parameter")
dbutils.widgets.dropdown("env", "dev", ["dev", "qa", "prod"], "Environment")
user_input = dbutils.widgets.get("input")
# Secrets
secret = dbutils.secrets.get("my_scope", "my_key")
print(secret) # [REDACTED]
print(secret.reveal()) # actual value
# Notebook orchestration; timeout_seconds is mandatory
result = dbutils.notebook.run(
"/Workspace/Team/child",
timeout_seconds=600,
arguments={"limit": 10},
)
# Task values
dbutils.jobs.taskValues.set("row_count", 42)
count = dbutils.jobs.taskValues.get("upstream_task", "row_count", default=0)
SQL that does not carry across
These constructs are common against Databricks Delta tables and unsupported in Yeedu. Alternatives where one exists.
| Unsupported feature | Databricks form | Yeedu alternative |
|---|---|---|
DELETE FROM with a subquery | DELETE FROM <table> WHERE id IN (SELECT id FROM <table>) | Build a replacement table using MINUS |
UPDATE with a subquery | UPDATE <table> SET <column> = <value> WHERE <condition> | Use MERGE INTO for conditional updates |
TRUNCATE TABLE | TRUNCATE TABLE <table> | Use DELETE FROM <table> |
USE DATABASE syntax | USE DATABASE <database_name> | Use USE <database_name> |
SHOW EXTERNAL LOCATIONS | SHOW EXTERNAL LOCATIONS | No equivalent. Our utility flags any job using it as not ready |
SHOW GRANTS ON EXTERNAL LOCATIONS | SHOW GRANTS ON EXTERNAL LOCATION <name> | No equivalent. Also flagged as not ready |
Identity-bound SQL functions
Four built-in functions resolve against the Databricks identity running the query. Our utility detects each of them and records an OAuth requirement, because validating them needs Databricks OAuth credentials.
| Function |
|---|
current_user() |
session_user() |
is_member() |
is_account_group_member() |
Behaviour after migration depends on the identity Yeedu executes under, which is often not the identity Databricks executed under. Review every use before trusting the result.
Databricks features with no Yeedu equivalent
Databricks Workflows
Yeedu has no native multi-task workflow engine with dependencies. Orchestration runs on Airflow or Prefect, and we generate the Airflow DAGs from your Databricks job definitions. See Job Migration.
Multi-language notebooks
One language per notebook. Pick Scala or Python. Databricks-style %python and %scala switching inside a single notebook isn't available, so mixed notebooks have to be split before they'll run.
DBFS
No DBFS, and no mount API, since dbutils.fs.mount and its siblings aren't implemented. Job task files under /dbfs/ are skipped by the migration utility rather than downloaded. Move them to fully qualified storage URIs such as abfss:// and s3a://, or to Yeedu workspace paths under /files/.
Delta Lake table handles
DeltaTable.forName(spark, "catalog.schema.table") becomes YeeduDeltaTable.forName(spark, "catalog.schema.table"), which resolves the table's physical location before opening it. Our utility performs this rewrite for you and comments out the from delta.tables import DeltaTable import.
Configuring dbutils
Settings come from Spark configuration entries and environment variables.
Precedence differs between the two implementations. The Python package reads the Spark configuration entry first and falls back to the environment variable. The Scala and Java package reads the environment variable first and falls back to the Spark configuration entry. Where both may be set, set them consistently.
Full configuration reference
Settings with both a Spark conf key and an environment variable:
| Purpose | Spark conf key | Environment variable | Default in code |
|---|---|---|---|
| Yeedu API token | spark.yeedu.token | YEEDU_TOKEN | default-token |
| Workspace ID | spark.yeedu.workspace_id | YEEDU_WORKSPACE_ID | -1 |
| Notebook run ID | spark.yeedu.job_id | YEEDU_JOB_ID | -1 |
| Default catalog name | spark.sql.defaultCatalog | YEEDU_CATALOG_NAME | default_catalog |
| Databricks host for secrets | spark.sql.catalog.<catalog>.uri | YEEDU_DATABRICKS_HOST | default-host |
| Databricks token for secrets | spark.sql.catalog.<catalog>.token | YEEDU_DATABRICKS_TOKEN | default-token |
| Airflow DAG ID | spark.yeedu.dag_id | YEEDU_DAG_ID | empty |
| Airflow DAG run ID | spark.yeedu.dag_run_id | YEEDU_DAG_RUN_ID | empty |
| Airflow task ID | spark.yeedu.task_id | YEEDU_TASK_ID | empty |
| Mapped-task index | spark.yeedu.map_index | YEEDU_MAP_INDEX | -1 |
<catalog> in the secrets keys is the resolved default catalog name.
Environment-variable-only settings:
| Variable | Default | Purpose |
|---|---|---|
YEEDU_RESTAPI_SSL_ENABLED | true | true selects https, anything else selects http |
YEEDU_RESTAPI_HOSTNAME | localhost | Yeedu REST API host |
YEEDU_RESTAPI_PORT | 8080 | Yeedu REST API port |
YEEDU_AIRFLOW_VERIFY_SSL | false | true enables TLS verification for notebook orchestration calls |
YEEDU_SSL_CERT_FILE | unset | Custom CA bundle path, used when YEEDU_AIRFLOW_VERIFY_SSL=true |
YEEDU_TASK_ID_JOB_CONF_KEY | task_id | Job-conf key from which the Airflow task ID is read |
The API base URL is assembled as <protocol>://<hostname>:<port>/api/v1.