Skip to main content
Version: v2.5.0

Change Log

New Features​

UI Support​

  • Seamlessly navigate and interact with the platform through an enhanced user interface, boosting user satisfaction and productivity.

Workspace​

  • Empower users to curate personalized workspaces with jobs and notebooks, enhancing collaboration and efficiency in project management.

Spark SQL​

  • Execute queries and leverage Spark compute seamlessly with our latest update, featuring Spark SQL support within jobs.

Notebook​

  • Dynamically execute Scala and Python notebooks to cater to varying workload demands, enhancing data analysis flexibility.

Cluster-Level Global Configuration​

  • Optimize Spark clusters by configuring settings at the cluster level, improving performance and resource utilization.

Access Management​

  • Fine-tune workspace accessibility and permissions with distinct roles: "Can View," "Can Run," "Can Edit," and "Can Manage," ensuring streamlined collaboration.

Kubernetes Support​

  • Integrate Kubernetes architecture seamlessly with Yeedu's core services, ensuring scalability and fault tolerance for enterprise-grade deployments.

Multicloud/Multi-zone Capability​

  • Simplify cluster management with a unified Cloud Environment, consolidating cloud-related configurations and reducing administrative overhead.

Increased Reliability​

  • Notebook Idle Timeout: Optimize resource usage by automatically terminating idle notebooks after 15 minutes, ensuring efficient resource allocation.

Vault Support​

  • Securely store credentials with vault support, enhancing protection of sensitive information.

Tab Complete Support for CLI​

  • Enhance navigation and productivity within the CLI interface with effortless command auto-completion triggered by the Tab key press.

Minor Changes​

  1. Nginx Configuration: Updated to utilize upstream servers.
  2. Kubernetes Ingress: Implemented the setup-ingress.sh script for configuration.
  3. Randomized Kubernetes Passwords: Replaced default passwords with randomly generated ones.
  4. Telegraf Configuration: Included network interfaces in the configuration file.
  5. Checkpoint Volume Mounts: Introduced volume mounts for the checkpoint file in the yeed-reactors-logs-collector service.
  6. Kubernetes Helm: Segregated third-party and core services.
  7. Cluster Resource Checks: Added checks for Kubernetes cluster resources.
  8. GPU Drivers: Updated GPU drivers setup.
  9. Enhanced Kubernetes Functionality:
    • Conditional storageClassName in PVC creation.
    • Added labels.
    • Implemented HA for MQ, Redis, and Vault.
    • Applied node affinity and anti-affinity.
    • Set limits for Pods.
    • Added support for helm upgrades.
    • Added nginx Ingress annotations for improved proxy settings and timeouts.
    • Configured Grafana-admin username and password from environment variables.
  10. History Server Improvements: Various enhancements made.
  11. Telegraf Metrics: Modified Telegraf to send metrics to the staging table.
  12. Password Handling: Added support for special characters in passwords.
  13. SSL Support: Added SSL support for history server nginx.
  14. History Server Configurations: Added configurations in the nginx conf file.
  15. Spark Scratch Configurations: Incorporated additional configurations.
  16. Azure Custom Script Extension: Removed the custom_script extension.
  17. Setup Custom Spark Configs: Introduced setup_custom_spark_configs with redaction of secrets.
  18. Azure CFE Zone and Region: Adjusted according to data model changes.
  19. OSM Container Loggers: Implemented loggers.
  20. Terraform Upgrade: Upgraded Terraform to version 1.7.4.
  21. Scratch Disk Setup: Modified to use RAID.
  22. Password Security: Handled special characters in passwords.
  23. CUDA Support Column: Added support for the new "has_cuda_support" column.
  24. Linux Distro Validation: Implemented validation.
  25. GCP Disk Change: Changed local SSD disk from NVME to SCSI.
  26. Docker Image Update: Updated the spark docker image name accordingly.
  27. Logging Methods: Enhanced logging methods.
  28. CLI Installation Verification: Verified installations.
  29. Local Storage Mounting: Updated cfe.py for mounting local storage to /scratch.
  30. GCP Terraform GPU Changes: Made necessary adjustments.
  31. Credential Security: Ensured credentials are hidden in logs.
  32. Telegraf Network Configuration: Added --network=host to Telegraf.
  33. Spark Environment: Added SPARK_LOCAL_DIRS in spark-env.sh for the worker container in standalone mode.

Bug Fixes​

  1. User Input Cleanup: Addressed clean-up issue.
  2. Grafana and MQ Ports: Fixed port issues in Docker.
  3. SSL Connection: Resolved SSL connection issue in Validate_third_party_services.
  4. Grafana CPU Usage: Fixed CPU usage issue.
  5. Permission Denied Issues: Resolved for cfe.py and updated configurations.
  6. Hive Metastore: Fixed metastore issue.
  7. Spark Master Container: Addressed start issues.
  8. RHEL Images Space: Fixed space issues.
  9. Bootstrap Shell Script: Corrected query.
  10. Reactor Container: Fixed startup command issues.
  11. Setup Reactors: Addressed spin issue for Spark version 2.4.8.
  12. Unsupported Notebook Versions: Fixed setup reactors.
  13. Ubuntu CUDA Version: Addressed version issues.
  14. Azure Mounted Disks: Fixed /mnt mounted disks issue.
  15. AWS Worker Numbers: Fixed standalone worker numbers.