Requirement Checklist (AWS)
Software Requirements
Ensure the following command-line tools (CLIs) are installed:
| CLI | Version |
|---|---|
| kubectl | v1.27.3 |
| helmfile | v0.142.0 |
| helm | v3.9.3 |
Cloud Requirements
1. Amazon Web Serivces (AWS) account
Make sure to have an active AWS account.
2. AWS Access & Secret Key Pair
Create a AWS Access & Secret Key Pair with following permissions: Create EC2 machines, Read VPC, Storage Object Admin, Read and Write access to ECR, CloudWatch Log Writer
3. Elastic Kubernetes Services (EKS)
Configure Elastic Kubernetes Service with the following recommendations:
- Namespace:
yeedu(Recommended) - Cluster Version:
v1.26.10 - Number of Nodes: 2
- Min CPU: 16
- Min Memory: 32GB
4. S3 Storage Bucket
Create an S3 Storage Bucket for storing configuration files and logs. Ensure the service principal from Step 2 has the necessary read and write permissions.
5. AWS RDS (PostgreSQL)
Deploy an AWS RDS (PostgreSQL) instance with the following configurations:
- CPU: 8 CPUs
- Memory: 32GB RAM
- Database Name: yeedu
- Persistent Disk: 500GB
- Backup Policy: Everyday (For Production)
- High Availability and DR: Recommended for Production
6. Elastic Container Registry (ECR)
Create an Elastic Container Registry (ECR) for storing Docker images. Refer to setup-docker-registry section for a list of container registries to be created and instructions on uploading Yeedu images.
7. AWS CloudWatch
Set up a AWS CloudWatch to view machine logs, create log groups in CloudWatch. Ensure the service principal from Step 2 has the necessary write access.
8. Firewalls
Ensure the following ports are open:
| Service Name | Port | Access From Workstation | Access From Apache Spark Cluster | Access From Yeedu Control Plane Server |
|---|---|---|---|---|
| REST API | 8080 | Yes | Yes | Yes |
| Jupyter Notebook | 8888-9088 | No | No | Yes |
| RabbitMQ UI | 15672 | Yes | No | No |
| Redis | 6379 | No | No | Yes |
| RabbitMQ | 5672 | No | Yes | Yes |
| PostgreSQL | 5432 | Yes | Yes | Yes |
| History-server | 10000 | Yes | No | Yes |
| Grafana | 3000 | Yes | Yes | Yes |
| Influx DB | 8086 | No | Yes | Yes |
| LDAP | 389 | No | No | Yes |
Make sure to fulfill these prerequisites for a successful Yeedu installation.
EFS prerequisites
- Create an EFS file system and corresponding mount targets.
- EC2 instance and EFS mount targets should be in the same VPC or connected VPCs to facilitate network communication.
- Allow inbound traffic on port 2049 from EC2 instances to the EFS mount targets.
- Availability Zone : regional
- Permissions : elasticfilesystem:ClientRootAccess, elasticfilesystem:ClientWrite, elasticfilesystem:ClientMount