Skip to main content
Version: v2.10.0

Architecture

Yeedu architecture diagram

Yeedu is a multi-cloud platform that delivers a cost-optimized infrastructure for Apache Spark workloads on AWS, Azure, and Google Cloud, so you deploy, manage, and scale your whole Spark estate from a single control plane while we handle the cloud-specific plumbing underneath.

The platform has three core components: the Yeedu Control Plane, Yeedu Compute, and the UI and CLI.

Yeedu Control Plane​

The Control Plane is the management backbone. It's where the services that operate the platform run:

  • Web application. A central hub where you create and manage workspaces, clusters, jobs, and notebooks.
  • Monitoring. Continuously checks the health of compute and workloads, and alerts you to anomalies.
  • Dependency management. A built-in repository for the libraries and packages your workloads need at runtime.
  • Cluster management. Holds the configuration for every cluster, across every cloud you've connected.
  • Notebooks and jobs. Bundled together inside workspaces for interactive analysis and scheduled execution.
  • Tenant management. Logical isolation of compute and resources, with role-based access on each tenant.

Yeedu Compute​

We provision the actual machines that run your Spark workloads through Yeedu Compute, using configuration held in the Control Plane.

  • Cluster creation. You launch clusters from the Control Plane and use them to run jobs and notebooks, subject to workspace permissions.
  • Cluster types. We support three cluster modes for Spark infrastructure:
    • Yeedu mode : fully managed, cost-optimized Spark infrastructure that delivers performance and scalability with the smallest operational footprint.
    • Standalone mode : Spark master and worker nodes run as Docker containers inside a single virtual machine, for streamlined deployments.
    • Cluster mode : Spark master and worker nodes run on dedicated virtual machines, for horizontal scalability under heavy workloads.

UI and CLI​

You can drive Yeedu through either a web UI or a command-line interface. It doesn't matter which. Both surface the same capabilities:

  • Access control. Authenticated users perform operations according to the roles they hold in each workspace and tenant.
  • Authentication. We integrate with LDAP, SSO, and Azure Active Directory based on the identity provider configured in the Control Plane.
note

We deploy every Yeedu component inside your own Virtual Private Cloud (VPC), which keeps data and metadata under your security and governance controls.