Skip to main content
Version: v2.9.1

Python

Python3 jobs allow users to run Python scripts or Spark-based PySpark workloads.

Python Job

Overview​

Python jobs provide flexibility for data transformations, analytics, automation, machine learning, and ETL pipelines.

Required Fields​

Arguments​

Provide the arguments required by the Python script.

Cluster​

Select the cluster where the job will execute.

Path​

Choose the Python script path from the dependency repository.

Advanced Options​

Supports configurations such as:

  • Python packages
  • Additional files
  • Archives
  • Spark configs
  • PyFiles
  • Driver configs

Advanced Options

When to Use Python Jobs​

Python jobs are ideal for:

  • Machine learning workflows
  • Data wrangling scripts
  • PySpark ETL