Requirement Checklist (Azure)
Software Requirements
Ensure the following command-line tools (CLIs) are installed:
| CLI | Version |
|---|---|
| kubectl | v1.27.3 |
| helmfile | v0.142.0 |
| helm | v3.9.3 |
Cloud Requirements
1. Azure Subscription and Resource Group
Make sure to have an active Azure subscription and a designated resource group.
2. Azure Service Principal
Create an Azure service principal with admin access to ADLS2, AKS, ACR, and LAW.
3. Azure Kubernetes Service (AKS)
Configure Azure Kubernetes Service with the following recommendations:
- Namespace:
yeedu(Recommended) - Cluster Version:
v1.26.10 - Number of Nodes: 2
- Min CPU: 16
- Min Memory: 32GB
- Machine Type: Standard_D4as_v5 (Recommended)
4. Azure Data Lake Gen 2 Blob Storage (ADLS2)
Set up an Azure Data Lake Gen 2 Storage account. Ensure the service principal from Step 2 has the necessary read and write permissions.
5. Azure PostgreSQL
Deploy an Azure PostgreSQL instance with the following configurations:
- CPU: 8 CPUs
- Memory: 32GB RAM
- Machine Type: Standard_D8_v4
- Persistent Disk: 500GB PremiumV2_LRS (SSD)
- Backup Policy: Everyday (For Production)
- High Availability and DR: Recommended for Production
6. Azure Container Registry (ACR)
Create an Azure Container Registry for storing Docker images. Refer to setup-docker-registry section for a list of container registries to be created and instructions on uploading Yeedu images.
7. Log Analytics Workspace
Set up a Log Analytics Workspace to store logs. Obtain the LAW_ID and LAW_SECRET values. Ensure the service principal from Step 2 has the necessary write access.
8. Firewalls
Ensure the following ports are open:
| Service Name | Port | Access From Workstation | Access From Apache Spark Cluster | Access From Yeedu Control Plane Server |
|---|---|---|---|---|
| REST API | 8080 | Yes | Yes | Yes |
| Jupyter Notebook | 8888-9088 | No | No | Yes |
| RabbitMQ UI | 15672 | Yes | No | No |
| Redis | 6379 | No | No | Yes |
| RabbitMQ | 5672 | No | Yes | Yes |
| PostgreSQL | 5432 | Yes | Yes | Yes |
| History-server | 10000 | Yes | No | Yes |
| Grafana | 3000 | Yes | Yes | Yes |
| Influx DB | 8086 | No | Yes | Yes |
| LDAP | 389 | No | No | Yes |
Make sure to fulfill these prerequisites for a successful Yeedu installation.
Azure File Share prerequisites (NFS only)
- Create an Azure Storage Account with Premium tier and NFS protocol enabled.
- Configure a Private Endpoint to ensure secure access.
- Access Control: Ensure the private endpoint is accessible.