Skip to main content
Version: v2.10.0

Disaster Recovery Strategy

Disaster Recovery Strategy Introduction​

Yeedu is a cutting-edge data processing and orchestration platform designed to handle modern workloads with a focus on reliability, resilience, and scalability. Here we set out our Disaster Recovery (DR) strategy in detail, which is how we keep services uninterrupted and data intact during unforeseen disasters. The strategy is optimized for cloud-native deployments on Kubernetes across major cloud providers, including AWS, GCP, and Azure.

Overview​

Disaster Recovery (DR) is a critical component of our architecture. We employ an Active-Passive DR model to ensure service continuity during disasters. This model involves deploying services across multiple cloud regions, with one region (primary) actively serving requests and another region (secondary) remaining dormant until a disaster occurs.

The key principles behind our DR strategy:

  • Data Independence: We don't store user data. Instead, we process data from user-configured sources and destinations, ensuring no risk to data integrity.
  • Cloud-Native Deployments: All Yeedu services are orchestrated using Kubernetes, leveraging its scalability and resilience.
  • Cloud-Managed Services: Critical components like PostgreSQL, Object Storage, and NFS are configured for geo-redundancy to minimize downtime.

Active-Passive Strategy​

Primary and Secondary Regions​

  1. Primary Region (Active): The primary region (e.g., us-central) hosts all active services and handles user requests.
  2. Secondary Region (Passive): The secondary region (e.g., us-east) has resources deployed but kept inactive to reduce costs. These resources are activated only during a disaster.

Steps During Disaster​

In the event of a disaster in the primary region, the following steps are executed:

  1. Activate Passive Resources: Start cloud services (e.g., PostgreSQL, Object Storage, NFS) and the Kubernetes cluster in the secondary region.
  2. Restore Data: If data replication isn't automatic, restore data from backups.
  3. Deploy Yeedu Services: Deploy all Yeedu services on the secondary Kubernetes cluster.
  4. Resume Operations: Services resume on the same domain (e.g., dev.yeedu.io), ensuring minimal disruption.

Components for DR​

CategoryComponentsDR Config Required
CloudPostgreSQL, Object Storage, NFS, KubernetesYes
InternalBroker, Cache, UI, API, Vault, Reactors, History Server, DashboardDeployed via Kubernetes

yeedu_clouds

Active-Active Strategy​

In an Active-Active DR model, resources in both regions are always active. This approach reduces recovery time but comes with higher operational costs due to the continuous operation of resources in both regions.

warning

Active-Active is described here for comparison only. Yeedu doesn't currently ship a multi-region Active-Active topology: there's no multi-region deployment configuration, cross-region traffic distribution, or cross-region state synchronization in the Yeedu deployment tooling. Active-Passive is the model Yeedu deployments use today. If you need Active-Active, contact Yeedu before planning around it.

Pros and Cons of Active-Active vs. Active-Passive​

Active-Active​

Pros:

  1. Minimal Downtime: Failover is close to instantaneous because the secondary region is already serving.
  2. Load Balancing: Traffic can be distributed across both regions, improving performance and reducing latency.
  3. High Resilience: Both regions are fully operational, reducing the risk of a single point of failure.
  4. No Manual Activation: Resources are already active, eliminating the need for manual intervention during failover.

Cons:

  1. Higher Costs: Resources in both regions are always active, leading to increased operational expenses.
  2. Complexity: Requires sophisticated configuration and monitoring to ensure consistency between regions.
  3. Data Synchronization: Continuous synchronization of data between regions can be challenging and resource-intensive.

Active-Passive​

Pros:

  1. Cost-Effective: Resources in the secondary region are kept inactive, reducing operational costs.
  2. Simpler Configuration: Easier to manage since only one region is active at a time.
  3. Controlled Failover: Failover is a deliberate process, allowing for better planning and execution.

Cons:

  1. Longer Downtime: Requires manual activation of resources in the secondary region, so recovery takes as long as your cloud provider needs to bring up the database, storage and Kubernetes cluster, plus the time to redeploy Yeedu services.
  2. Risk of Delayed Recovery: If the failover process encounters issues, recovery time may increase.
  3. Underutilized Resources: The secondary region remains idle until a disaster occurs, leading to resource inefficiency.

Typical Recovery Workflow​

A typical disaster recovery scenario for Yeedu involves the following steps:

  1. Failure Detection: A failure occurs in a critical service in the primary region. This could be a failure in a data source, network, or cloud infrastructure impacting the Yeedu deployment.
  2. Investigation: The issue is investigated with the cloud provider to determine the scope and expected resolution time.
  3. Decision to Failover: If the issue can't be resolved promptly, a decision is made to failover to the secondary region.
  4. Verification: Ensure that the same issue doesn't impact the secondary region.
  5. Failover Execution:
    • Stop all activities in the primary region. Users stop workloads, and administrators back up recent changes if possible.
    • Shut down jobs that haven't already failed due to the outage.
    • Start the recovery procedure in the secondary region. This includes updating routing and renaming connections to redirect traffic to the secondary region.
  6. Testing and Declaration: After testing, declare the secondary region operational. Users can resume workloads, and delayed jobs can be retriggered.
  7. Failback to Primary Region:
    • Once the issue in the primary region is resolved, stop all activities in the secondary region.
    • Start the recovery procedure in the primary region, including routing and renaming connections back to the primary region.
    • Replicate data back to the primary region as needed. Minimize replication complexity by focusing on critical data.
    • Test the deployment in the primary region and declare it operational.

What Needs to Be Backed Up and Restored​

What Needs to Be Backed Up​

  1. Database (PostgreSQL):
    • Regular backups of the database are essential to ensure data consistency.
    • Use cloud-managed database services with geo-redundancy or scheduled backups.
  2. Object Storage:
    • Backup critical files and data stored in object storage.
    • Enable versioning and replication across regions.
  3. NFS (Network File System):
    • Backup shared file systems used by Yeedu services.
    • Use cloud-native solutions like AWS EFS, GCP Filestore, or Azure Files with cross-region replication.
  4. Configuration Files:
    • Backup Kubernetes manifests, Helm charts, and other deployment configurations.
    • Include spark-defaults.conf, spark-env.sh, and other service-specific configurations.
  5. Application State:
    • Backup stateful data for services like Yeedu Cache, Broker, and Vault.
    • Ensure logs and metadata are included for debugging and auditing.

What Needs to Be Restored​

  1. Database (PostgreSQL):
    • Restore the latest backup or use the replicated database in the secondary region.
    • Ensure all transactions up to the point of failure are applied.
  2. Object Storage:
    • Restore critical files and data to the secondary region.
    • Validate the integrity of the restored data.
  3. NFS:
    • Mount the replicated or restored file system in the secondary region.
    • Ensure all services dependent on NFS can access the required files.
  4. Configuration Files:
    • Deploy the backed-up Kubernetes manifests and configurations in the secondary region.
    • Update any region-specific settings (e.g., DNS, IP addresses).
  5. Application State:
    • Restore the stateful data for services like Yeedu Cache, Broker, and Vault.
    • Ensure services are fully operational and consistent with the primary region.

Recovery Objectives​

Recovery Point Objective (RPO)​

The Recovery Point Objective (RPO) defines the maximum acceptable amount of data loss measured in time. For Yeedu, the RPO depends on the frequency of data replication between the primary and secondary regions. With proper configuration, the RPO can be as low as a few seconds or minutes.

Recovery Time Objective (RTO)​

The Recovery Time Objective (RTO) defines the maximum acceptable time to restore services after a disaster.

We don't publish an RTO figure. In the Active-Passive model, recovery time is dominated by factors outside our deployment:

  • The time your cloud provider takes to start the secondary region's PostgreSQL instance, object storage and NFS.
  • The time to bring up the secondary Kubernetes cluster.
  • The time to restore data, if replication isn't automatic.
  • The time to deploy the Yeedu services onto the secondary cluster.

Measure these stages in your own environment with a failover drill, and use the measured total as your RTO. Don't plan against an assumed figure.

Conclusion​

Our Disaster Recovery approach relies on cloud-native services, Kubernetes orchestration, and an Active-Passive model in which the secondary region is provisioned in advance and activated during a disaster. Because we don't store user data, recovery is a matter of restoring the metadata database, shared storage and configuration, and then redeploying the services.

Run a failover drill to establish your own recovery time and recovery point figures before you rely on this plan.