Requirement Checklist (GCP)
Software Requirements
Ensure the following command-line tools (CLIs) are installed:
| CLI | Version |
|---|---|
| kubectl | v1.27.3 |
| helmfile | v0.142.0 |
| helm | v3.9.3 |
Cloud Requirements
1. Google Cloud Platform (GCP) project
Make sure to have an active GCP project.
2. Google Cloud Platform (GCP) Serivce Account
Create a service account with following permissions: Compute Engine Admin, VPC Reader, GCR Reader, Storage Object Admin, Kubernetes admin
3. Google Kubernetes Engine (GKE)
Configure Azure Kubernetes Service with the following recommendations:
- Namespace:
yeedu(Recommended) - Cluster Version:
v1.26.10 - Number of Nodes: 2
- Min CPU: 16
- Min Memory: 32GB
4. Google Cloud Storage (GCS)
Create a bucket in the same project with Storage Object Admin Access. Ensure the service principal from Step 2 has the necessary read and write permissions.
5. Google Cloud SQL (PostgreSQL)
Deploy an Google Cloud SQL (PostgreSQL) instance with the following configurations:
- CPU: 8 CPUs
- Memory: 32GB RAM
- Database Name: yeedu
- Persistent Disk: 500GB (pd-ssd)
- Backup Policy: Everyday (For Production)
- High Availability and DR: Recommended for Production
6. Google Container Registry (GCR)
Create an Google Container Registry (GCR) for storing Docker images. Refer to setup-docker-registry section for a list of container registries to be created and instructions on uploading Yeedu images.
7. Google Stackdriver
Set up a Google Stackdriver to store logs. Ensure the service principal from Step 2 has the necessary write access.
8. Firewalls
Ensure the following ports are open:
| Service Name | Port | Access From Workstation | Access From Apache Spark Cluster | Access From Yeedu Control Plane Server |
|---|---|---|---|---|
| REST API | 8080 | Yes | Yes | Yes |
| Jupyter Notebook | 8888-9088 | No | No | Yes |
| RabbitMQ UI | 15672 | Yes | No | No |
| Redis | 6379 | No | No | Yes |
| RabbitMQ | 5672 | No | Yes | Yes |
| PostgreSQL | 5432 | Yes | Yes | Yes |
| History-server | 10000 | Yes | No | Yes |
| Grafana | 3000 | Yes | Yes | Yes |
| Influx DB | 8086 | No | Yes | Yes |
| LDAP | 389 | No | No | Yes |
Make sure to fulfill these prerequisites for a successful Yeedu installation.
Google Cloud Filestore prerequisites
- Provision a Filestore instance within a Google Cloud project.
- Ensure the Filestore instance resides in the same VPC network and subnet as the connected resources.
- Access Control: Grant admin access to the listed ip ranges.