Monitoring
The cluster dashboard contains 5 main windows:
- Events
- Configuration
- Dependencies
- JDBC
- Access Management
1. Events Window
Basic Configuration Display
Shows:
- ID
- Cloud Provider
- Cluster Type
- Runtime Version
- Instance Size
- Min & Max Instances
- Auto Shutdown
- Parallel Execution
- Workers per Node
Job Statistics
Includes job counts by status:
- Submitted
- Running
- Done
- Error
- Terminated
- Stopping
- Stopped
- Total Jobs
Event Logs
Tabs:
- Stdout
- Stderr
Actions:
- Refresh
- Copy
- Download (.log)
2. Configuration Window
Allows editing cluster settings.
Important Note
- Cluster must be destroyed to edit.
- Cluster Type cannot be edited after creation.
- All other fields are editable.
3. Dependencies
- Manage dependency repositories.
- Upload new files using + File.
- Delete via 3-dot action menu.
4. JDBC Tab
Provides:
- JDBC driver
- Connection URL
- Username
- Auth token
- Workspaces attached to this cluster
Note: Thrift SQL not supported for current runtime.
5. Access Management
- Manage which workspaces can attach to this cluster.
- Add/remove by searching or selecting checkbox.
- Can Attach To option links workspace to cluster.
Additional Features
Auto Shutdown
- Cluster stops automatically after idle timeout.
- Minimum timeout: 1 minute
- Minimizes compute cost.
Warm Start/Stop
- Reduces cluster startup from 6–7 minutes → ~1 minute
- Improves responsiveness and resource utilization.
Start/Stop a Cluster
Start:
- Go to cluster dashboard → click Start
Stop:
- Click Stop to halt jobs & free resources
Cluster Details (Once Running)
Displays:
- Cloud provider
- Compute type
- Runtime version
- Disk configuration
- Instance size
- Execution slots
Job Management
Shows:
- Submitted jobs
- Running jobs
- Completed & stopped jobs
- Errors
Users can click View Logs to troubleshoot.
Graviton (ARM64) Support
- ARM-based compute
- Higher memory/disk throughput
- Ideal for Spark, Python pipelines, ML workloads
Enable during cluster creation: