Setup Yeedu Control Plane

• 1. Credentials Setup
• 2. Download Yeedu Control Plane Files
• 3. Deploy Yeedu Services on Kubernetes
  • 3.1. Provide execute permissions
  • 3.2. Authenticate to Kubernetes Cluster (Google Kubernetes Engine)
  • 3.3. Update SSL Certificates
  • 3.4. Create DNS Entries
• 4. Configure Yeedu Control Plane Properties
  • 4.1. Configuration Files
  • 4.2. Variables Reference
  • 4.3. Setup Ingress in Provided Namespace
  • 4.4. Deploy Yeedu Control Plane on Kubernetes

1. Credentials Setup

Create two files for to store below mentioned credentials:

1. Cloud Credentials:

   • Create cloud_credentials.json.
   • Paste JSON credentials in the specified format.

2. Registry Credentials:

   • Create docker_registry_credentials.json.
   • Paste JSON credentials for your container registry.

Credential Format for Google Cloud Platfom

{
    "type": "service_account",
    "project_id": "<project_id>",
    "private_key_id": "<private_key_id>",
    "private_key": "<private_key>",
    "client_email": "<client_email>",
    "client_id": "<client_id>",
    "auth_uri": "<auth_uri>",
    "token_uri": "<token_uri>",
    "auth_provider_x509_cert_url": "<auth_provider_x509_cert_url>",
    "client_x509_cert_url": "<client_x509_cert_url>"
}

note

Replace all placeholder values (e.g., <project_id>, <private_key>) with your actual credentials. Keep these files safe and secure, since they hold secrets you don't want in a Git repository.

2. Download Yeedu Control Plane Files

To initiate your Yeedu Control Plane Setup, download the essential scripts from the yeedu-runtime.

1. Open a terminal or command prompt.

2. Extract the Yeedu Runtime Setup Files:

   tar -xzvf /local/path/to/local/storage/yeedu-runtime.tar

note

Reach out to the Yeedu team to get access to the Yeedu release artifacts (yeedu-runtime.tar) and to the container registry that holds the Yeedu images.

3. Change the working directory to yeedu-runtime:

   cd yeedu-runtime

3. Deploy Yeedu Services on Kubernetes

3.1. Provide execute permissions

After updating properties, go to the Yeedu Control Plane folder. Find yeedu-core-services.sh and setup-ingress.sh script. Grant execute permissions:

chmod +x yeedu-core-services.sh
chmod +x setup-ingress.sh

3.2. Authenticate to Kubernetes Cluster (Google Kubernetes Engine)

Authenticate to the Kubernetes Cluster using the Google Cloud CLI and kubectl commands provided below.

gcloud auth activate-service-account --key-file=PATH_TO_YOUR_SERVICE_ACCOUNT_JSON
gcloud config set project YOUR_PROJECT_ID
gcloud container clusters get-credentials CLUSTER_NAME --region=CLUSTER_REGION

Validate successful cluster authentication by running the command below:

kubectl config get-contexts

3.3. Update SSL Certificates

In the yeedu-runtime directory, find the ssl folder. Update yeedu.crt and yeedu.key with the your specific SSL certificate and SSL RSA key. These files are used by the Yeedu Ingress Controller in Kubernetes.

3.4. Create DNS Entries

Create DNS entries for the Ingress record in the GCP cloud environment. Ensure entries are created for all configured hostnames in the Ingress. Execute the following command:

gcloud dns record-sets transaction start --zone=YOUR_DNS_ZONE_NAME
gcloud dns record-sets transaction add <ip> --name=<dns> --ttl=300 --type=A --zone=YOUR_DNS_ZONE_NAME
gcloud dns record-sets transaction execute --zone=c

4. Configure Yeedu Control Plane Properties

After downloading the essential files, customize your Yeedu environment by adjusting key configuration settings. Update the following property files to reflect your specific infrastructure and cloud provider details. The split between the two is worth understanding before you start editing. Everything that describes a running service, a hostname, a port, a username, a password, an SSL toggle, belongs in yeedu-connection.properties, which is why that file carries Postgres, RabbitMQ, Redis, LDAP, the REST API, Vault, the UI, the History Server, Grafana and InfluxDB between them. Everything that describes the deployment itself, the YEEDU_VERSION to install, the deployment mode, the object storage bucket, the container registry and the Kubernetes namespace, belongs in yeedu-system-config.properties.

We ship both files fully commented out.

4.1. Configuration Files

1. yeedu-connection.properties:

   • This file connects Yeedu to essential services like Postgres, RabbitMQ, and Redis. Update connection parameters based on your service configurations, including database hostnames, usernames, and passwords.

2. yeedu-system-config.properties:

   • This file manages cloud-based services such as Object Storage and Container Registry. Align settings with your chosen cloud provider and configurations for seamless integration.

note

Uncomment and provide input values for the properties specified in the files. Detailed explanations for each parameter are available in the respective property files.

4.2. Variables Reference

4.2.1. yeedu-connection.properties

Metadata DB

Variable	Description	Default Value	Required
YEEDU_DEPLOY_LOCAL_POSTGRES	Enable local Postgres deployment	true	Yes
YEEDU_PG_DB	Name of the Postgres database	yeedu	Yes
YEEDU_PG_USER	Username for Postgres connection	postgres	Yes
YEEDU_PG_PASSWORD	Password for Postgres user	postgres	Yes
YEEDU_PG_HOSTNAME	Hostname or IP address of the Postgres server	Endpoint Url of the service	Yes
YEEDU_PG_PORT	Port number for the Postgres server	5432	No
YEEDU_PG_SSL_ENABLED	Enable SSL encryption for Postgres connection	false	No
YEEDU_PG_MIN_CONNECTIONS	Minimum size of the Postgres connection pool	15	No
YEEDU_PG_MAX_CONNECTIONS	Maximum size of the Postgres connection pool	200	No

Messaging Service

Variable	Description	Default Value	Required
YEEDU_DEPLOY_LOCAL_RABBITMQ	Enable local RabbitMQ deployment	true	Yes
YEEDU_MQ_USER	Username for RabbitMQ connection	guest	Yes
YEEDU_MQ_PASSWORD	Password for RabbitMQ user	guest	Yes
YEEDU_MQ_PORT	Port number for RabbitMQ	5672	No
YEEDU_MQ_UI_PORT	Port number for RabbitMQ management UI	15672	No
YEEDU_MQ_UI_HOSTNAME	Hostname for RabbitMQ management UI	Endpoint Url of the service	Yes
YEEDU_MQ_HOSTNAME	Hostname or IP address of the RabbitMQ server	Endpoint Url of the service	Yes
YEEDU_MQ_VH	Virtual host for RabbitMQ	"/"	No
YEEDU_MQ_SSL_ENABLED	Enable SSL encryption for RabbitMQ connection	false	No

Redis Cache

Variable	Description	Default Value	Required
YEEDU_DEPLOY_LOCAL_REDIS	Enable local Redis deployment	true	Yes
YEEDU_REDIS_PASSWORD	Password for Redis	admin	No
YEEDU_REDIS_PORT	Port number for Redis	6379	No
YEEDU_REDIS_HOSTNAME	Hostname or IP address of the Redis server	Endpoint Url of the service	Yes
YEEDU_REDIS_SSL_ENABLED	Enable SSL encryption for Redis connection	false	No

LDAP Service

Variable	Description	Default Value	Required
YEEDU_DEPLOY_LOCAL_LDAP	Enable local LDAP deployment	true	Yes
YEEDU_AUTH_LDAP_ORGANISATION	LDAP organization name	yeedu	Yes
YEEDU_AUTH_LDAP_DOMAIN	LDAP domain	yeedu.com	Yes
YEEDU_AUTH_LDAP_ADMIN_PASSWORD	Password for LDAP administrator	Ndbk5Lgb	Yes
YEEDU_AUTH_LDAP_HOSTNAME	Hostname or IP address of the LDAP server	Endpoint Url of the service	Yes
YEEDU_AUTH_LDAP_PORT	Port number for LDAP server	389	No
YEEDU_AUTH_LDAP_BASE_DN	Base DN for LDAP	dc=yeedu,dc=com	Yes
YEEDU_AUTH_LDAP_BIND_DN	Bind DN for LDAP	cn=admin,dc=yeedu,dc=com	Yes
YEEDU_AUTH_LDAP_BIND_PASSWORD	Password for LDAP bind user	Ndbk5Lgb	Yes
YEEDU_AUTH_LDAP_SSL_ENABLED	Enable SSL encryption for LDAP connection	false	No
YEEDU_AUTH_LDAP_GROUP_SEARCH_FILTER	LDAP query filter used to search for groups	<YOUR_LDAP_GROUP_SEARCH_FILTER>	No
YEEDU_AUTH_LDAP_USER_SEARCH_FILTER	LDAP query filter used to search for users	<YOUR_LDAP_USER_SEARCH_FILTER>	No
YEEDU_AUTH_LDAP_USER_GROUPS_SEARCH_FILTER	Filter to retrieve all groups of which a user is a member	<YOUR_LDAP_USER_GROUPS_SEARCH_FILTER>	No
YEEDU_AUTH_PLATFORM_ADMIN_USERNAMES	Comma-separated usernames granted platform-admin rights	YSU0000	No

REST-API

Variable	Description	Default Value	Required
YEEDU_RESTAPI_PORT	Port number for REST API server	8080	No
YEEDU_RESTAPI_HOSTNAME	Hostname or IP address of REST API server	Endpoint Url of the service	Yes
YEEDU_SECRET_KEY	Secret key for authentication and encryption	yeedu	No
YEEDU_RESTAPI_LOG_LEVEL	Log level for REST API server (e.g., info, debug)	info	No
YEEDU_RESTAPI_SSL_ENABLED	Enable SSL encryption for REST API connection	false	No
YEEDU_RESTAPI_REPLICA_COUNT	Number of replicas/instances for REST API server	1	No

Vault

Variable	Description	Default Value	Required
YEEDU_RESTAPI_VAULT_HOSTNAME	Hostname or IP address of the Vault server	Host's detected IP address	Yes
YEEDU_RESTAPI_VAULT_PORT	Port number for Vault	8200	No
YEEDU_RESTAPI_VAULT_UI_PORT	Port number for the Vault UI	9200	No
YEEDU_RESTAPI_VAULT_AUTH_TYPE	Authentication type used against Vault	BASIC_AUTH	No
YEEDU_RESTAPI_VAULT_MOUNT_PREFIX	Vault mount prefix for Yeedu credentials	yeedu/credentials	No
YEEDU_RESTAPI_VAULT_SSL_ENABLED	Enable SSL encryption for the Vault connection	false	No

Yeedu UI

Variable	Description	Default Value	Required
YEEDU_UI_HOSTNAME	Hostname or IP address of the Yeedu UI	Host's detected IP address	Yes
YEEDU_UI_PORT	Port number for the Yeedu UI	5173	No
YEEDU_UI_SSL_ENABLED	Enable SSL encryption for the Yeedu UI	false	No

History Server

Variable	Description	Default Value	Required
YEEDU_HISTORY_SERVER_WEB_PORT	Port number for History Server web interface	10000	No
YEEDU_HISTORY_SERVER_WEB_HOSTNAME	Hostname for History Server web interface	Host's detected IP address	No
YEEDU_HISTORY_SERVER_WEB_SSL_ENABLED	To Enable/Disable History server on HTTPS	false	No

Monitor

Variable	Description	Default Value	Required
YEEDU_GRAFANA_HOSTNAME	Hostname or IP address of Grafana server	Endpoint Url of the service	Yes
YEEDU_GRAFANA_PORT	Port number for Grafana	3000	No
YEEDU_GRAFANA_USERNAME	Username for Grafana login	YSU0000	No
YEEDU_GRAFANA_PASSWORD	Password for Grafana login	YSU0000	No
YEEDU_GRAFANA_LDAP_USER_GROUP_CN	LDAP user group Common Name (CN) for Grafana	cn=yeedu-user,ou=groups,dc=yeedu,dc=com	No
YEEDU_GRAFANA_LDAP_USER_BASE_DN	LDAP user base DN for Grafana	ou=users,dc=yeedu,dc=com	No
YEEDU_GRAFANA_LDAP_GROUP_SEARCH_BASE_DNS	LDAP group search base for Grafana	ou=groups,dc=yeedu,dc=com	No
YEEDU_GRAFANA_SSL_ENABLED	To Enable/Disable Grafana server on HTTPS	false	No

InfluxDB

Variable	Description	Default Value	Required
YEEDU_INFLUXDB_HOSTNAME	Hostname or IP address of InfluxDB server	Endpoint Url of the service	Yes
YEEDU_INFLUXDB_PORT	Port number for InfluxDB	8086	No
YEEDU_INFLUXDB_DB	Name of the InfluxDB database	influx	No
YEEDU_INFLUXDB_USERNAME	Username for InfluxDB login	admin	No
YEEDU_INFLUXDB_PASSWORD	Password for InfluxDB login	yeeduadmin	No
YEEDU_INFLUXDB_TOKEN	The authentication token for InfluxDB	yeedu-admin-secret-token	No
YEEDU_INFLUXDB_ORGANISATION	Organization of InfluxDB	yeedu	No

Cache Proxy

Variable	Description	Default
YEEDU_CACHE_PROXY_ENABLED	Enable the Cache Proxy	false
YEEDU_CACHE_PROXY_HOSTNAME	Hostname for Cache Proxy	Host's detected IP address
YEEDU_CACHE_PROXY_PORT	Port for Cache Proxy	8765
YEEDU_CACHE_PROXY_SSL_ENABLED	Enable SSL for Cache Proxy	false

Scheduler

Variable	Description	Default
YEEDU_SCHEDULER_HOSTNAME	Scheduler host	Host's detected IP address
YEEDU_SCHEDULER_PORT	Scheduler port	8081
YEEDU_SCHEDULER_USERNAME	Scheduler username	airflow
YEEDU_SCHEDULER_PASSWORD	Scheduler password	airflow
YEEDU_SCHEDULER_SSL_ENABLED	Enable SSL for Scheduler	false
YEEDU_SCHEDULER_REPLICA_COUNT	Number of scheduler replicas	1

Assistant

Variable	Description	Default
YEEDU_ASSISTANT_ENABLED	Enable the Yeedu Assistant	false
YEEDU_ASSISTANT_HOSTNAME	Hostname of Yeedu Assistant	Host's detected IP address
YEEDU_ASSISTANT_PORT	Assistant API port	8000
YEEDU_ASSISTANT_SSL_ENABLED	SSL toggle for Assistant	false
YEEDU_ASSISTANT_FASTAPI_PORT	FastAPI internal service port	8001
YEEDU_ASSISTANT_REDIS_NAMESPACE	Redis cache namespace for Assistant	yeedu_ai_assistant
YEEDU_ANTHROPIC_API_KEY	API key for Anthropic LLM integration	<ANTHROPIC_API_KEY>
YEEDU_OPENAI_API_KEY	API key for OpenAI LLM integration	<OPENAI_API_KEY>

note

Set either YEEDU_ANTHROPIC_API_KEY or YEEDU_OPENAI_API_KEY, depending on the LLM provider you configure for the Assistant. Both are read only when YEEDU_ASSISTANT_ENABLED is true.

Yeedu Functions

Variable	Description	Default
YEEDU_FUNCTIONS_PROXY_HOSTNAME	Hostname for Functions Proxy	Host's detected IP address
YEEDU_FUNCTIONS_SSL_ENABLED	Enable SSL for Functions Proxy	false
YEEDU_FUNCTIONS_SCHEDULER_PORT	Scheduler port for Functions service	5171
YEEDU_FUNCTIONS_PROXY_PORT	Proxy port for Functions service	5172

4.2.2. yeedu-system-config.properties

Yeedu Version

Variable	Description	Value	Required
YEEDU_VERSION	Specifies the version of the Yeedu platform to be deployed.	v2.10.1	Yes

Deployment Parameters

Variable	Description	Default Value	Required
YEEDU_DEPLOYMENT_MODE	Deployment mode. Allowed values: docker, eks, aks, gke, minikube	docker	Yes

caution

The default is docker, and it won't work here. For this Google Kubernetes Engine (GKE) installation you must set YEEDU_DEPLOYMENT_MODE=gke.

System Properties

Variable	Description	Default Value	Required
YEEDU_RUN_INIT_SERVICE	Runs the initialization service during deployment.	true	Yes
YEEDU_ENVIRONMENT	Specifies the deployment environment (dev, prod, staging, etc.).	dev	Yes
YEEDU_ENABLE_GRAFANA_MONTORING	Enables Grafana monitoring for system metrics.	true	No
YEEDU_DEPLOYMENT_LOG_LEVEL	Sets the log level for deployment (debug, info, warn, error).	info	No
YEEDU_ASSISTANT_ENABLED	Enables AssistantX features within the platform.	false	No
YEEDU_ENABLE_SCHEDULER	Enables the Yeedu Scheduler component.	false	No
YEEDU_CACHE_PROXY_ENABLED	Enables the Cache Proxy service.	false	No

System Object Storage Configuration

Variable	Description	Default Value	Required
YEEDU_SYSTEM_CLOUD_PROVIDER	Cloud provider for object storage	-	Yes
YEEDU_SYSTEM_PROJECT_ID	Project ID for object storage	-	Yes
YEEDU_SYSTEM_OBJECT_STORAGE	Object storage bucket or container name	-	Yes
YEEDU_SYSTEM_CLOUD_CREDENTIALS_FILE_PATH	File path to cloud provider credentials (JSON)	-	Yes
YEEDU_SYSTEM_OBJECT_STORAGE_ENCRYPTION_ENABLED	Enable object storage encryption	-	Yes
YEEDU_SYSTEM_SPARK_EVENTS_RETENTION_DAYS	Retention period for spark events	90	Yes
YEEDU_SYSTEM_NFS_HOSTNAME	Hostname of the NFS server	<your-hostname>	Yes
YEEDU_SYSTEM_NFS_MOUNT_TARGET	The mount path on the NFS server	<your-mount-path>	Yes

System Container Registry Configuration

Variable	Description	Default Value	Required
YEEDU_SYSTEM_DOCKER_REGISTRY_PROVIDER	Docker registry provider	-	Yes
YEEDU_SYSTEM_DOCKER_REGISTRY_URL	Docker registry URL	-	Yes
YEEDU_SYSTEM_DOCKER_REGISTRY_CREDENTIALS_FILE_PATH	File path to Docker registry credentials (JSON)	-	Yes

Kubernetes Parameters

Variable	Description	Default Value	Required
YEEDU_SYSTEM_KUBERNETES_NAMESPACE	Namespace for Kubernetes	yeedu	No
YEEDU_SYSTEM_KUBERNETES_STATIC_IP	Static IP address for NGINX in Kubernetes	<your-static-ip>	No

note

"Required" indicates whether the variable is required for Yeedu setup

4.3. Setup Ingress in Provided Namespace

Run the following command to setup Nginx Ingress Controller:

Execute the setup ingress script ./setup-ingress.sh and it'll deploy the ingress.

After the ingress is created, obtain the IP address:

kubectl get ingress -n $YEEDU_SYSTEM_KUBERNETES_NAMESPACE

4.4. Deploy Yeedu Control Plane on Kubernetes

Execute the Yeedu Control Plane setup script yeedu-core-services.sh. Run ./yeedu-core-services.sh --help for more information. This script installs kubectl, Helm, Helmfile CLI, psql, rabbitmqadmin, and Redis CLI's on the machine, adapting to the machine's architecture and OS and setup up Yeedu Control Plane on Kubernetes.

Run the following command to start the Yeedu Control Plane on Kubernetes:

• Start the services

./yeedu-core-services.sh start

• Stop the services

./yeedu-core-services.sh stop

• Restart the services

./yeedu-core-services.sh restart

• Clean the services and delete data

./yeedu-core-services.sh clean

After executing these commands, the script generates necessary secrets, configmaps, deployments, pods, and services to run Yeedu Services on Kubernetes. Access the Yeedu RestAPI at the configured YEEDU_RESTAPI_HOSTNAME URL with HTTPS TLS termination provided by the ingress.

The script writes a timestamped deployment log under $HOME/.yeedu/deployment/logs/ and prints its path when it starts. Check that file if a step fails.