Python
Python jobs provide flexibility for data transformations, analytics, automation, machine learning, and ETL pipelines. Python jobs allow users to run Python scripts or Spark-based PySpark workloads. You'll reach it from the Create Job form via Job Type → Custom Code → Language → Python.
Required Fields
| Field | Required | Notes |
|---|---|---|
| Cluster | - | Cluster where the job will execute |
| Job Type | Yes | Custom Code |
| Language | Yes | Python |
| Path | Yes | Python script location, selected with Choose file |
| Arguments | No | Arguments required by the Python script |
Advanced Options
An Append checkbox plus six dependency tabs:
| Tab | Purpose |
|---|---|
| Configs | Spark configuration Key / Value rows (with Bulk Edit) |
| Packages | Install required dependencies |
| Jars | Attach external libraries |
| Repositories | Repositories used to resolve dependencies |
| Archives | Add compressed resources |
| Files | Include supporting files |
Followed by Principal & Key Tab, Driver Config, Concurrent Runs and Timeout. See Create Job.
When to Use Python Jobs
We recommend Python jobs for:
- Machine learning workflows
- Data wrangling scripts
- PySpark ETL