Merge qss-poc branch for HCC back to main

applications/hcc/README_update_zone.md

@@ -0,0 +1,44 @@
+# GCP Zone and Region Availability Updater
+
+This Python script updates zones and regions in a metadata YAML file and a JSON file based on the availability of specific Google Compute Engine (GCE) machine types in a given Google Cloud Project. It provides two main functionalities:
+
+1. **Updating Zones in YAML:** It updates the `allowlisted_zones` field in a YAML file (`metadata.display.yaml`) for specified machine types (e.g., "a3-megagpu-8g", "a3-ultragpu-8g"). 
+
+2. **Generating Zone-to-Region Mapping:** It generates a JSON file (e.g., `zone_to_region.json`) that provides a mapping between zones and their corresponding regions in your GCP project. The main reason why we need `zone_to_region.json` is for region lookup in [region.tf](https://github.com/GoogleCloudPlatform/ai-on-gke/blob/qss-poc/applications/hcc/region.tf#L2).
+
+The script uses the Google Cloud Compute Engine API to retrieve information about regions, zones, and machine type availability. It leverages the `google-cloud-compute` and `PyYAML` libraries for interacting with the API and processing YAML files.
+
+## Prerequisites
+
+* **Python 3.6 or higher**
+* **Google Cloud Project:** A project with the necessary APIs enabled (Compute Engine API).
+* **Authentication:** Ensure your environment is authenticated to your Google Cloud Project. The easiest way is to use the `gcloud` command-line tool:
+   ```bash
+   gcloud auth application-default login
+   ```
+* **Required Python packages:**
+  ```Bash
+  pip install google-cloud-compute PyYAML
+  ```
+## Usage
+* Run the script with the `--project_id` argument:
+  ```
+  python update-region.py --project_id your-project-id
+  ```
+  Replace `your-project-id` with your actual Google Cloud Project ID.
+
+
+### Example
+To update the `metadata.display.yaml` file with the available zones for the `a3-megagpu-8g` and `a3-ultragpu-8g` machine types in the `my-gcp-project` project, and generate the `zone_to_region.json` file:
+```Bash
+python update-region.py --project_id my-gcp-project 
+```
+This will:
+
+* Create or update the `zone_to_region.json` file with the zone-to-region mapping for your project.
+* Update the `allowlisted_zones` in `metadata.display.yaml` for both `a3-megagpu-8g` (under `a3_mega_zone`) and `a3-ultragpu-8g` (under `a3_ultra_zone`) with the zones where these machine types are currently available.
+
+
+## Notes
+* The script uses the application default credentials for authentication. Make sure you've authenticated your environment using `gcloud auth application-default login`.
+* The script currently supports updating allowlisted_zones for `a3-megagpu-8g` and `a3-ultragpu-8g` machine types. You can extend it to support other machine types by modifying the `update_blueprint_metadata` function.

applications/hcc/a3mega_workloads.tfvars

@@ -0,0 +1,45 @@
+/**
+  * Copyright 2023 Google LLC
+  *
+  * Licensed under the Apache License, Version 2.0 (the "License");
+  * you may not use this file except in compliance with the License.
+  * You may obtain a copy of the License at
+  *
+  *      http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+
+authorized_cidr = "0.0.0.0/0"
+
+goog_cm_deployment_name = "a3mega-qss-test"
+
+labels = {
+  ghpc_blueprint  = "gke-a3-mega"
+  ghpc_deployment = "a3mega-qss-test"
+}
+
+project_id = "gke-ai-eco-dev"
+
+a3_mega_zone = "australia-southeast1-c"
+a3_ultra_zone = ""
+
+gpu_type = "A3 Mega"
+
+reservation = "hcc-a3mega"
+reservation_block = ""
+placement_policy_name = "kevinmcw-test"
+
+a3mega_recipe = "gke"
+a3ultra_recipe = ""
+node_count_gke_nccl = -1
+node_count_gke = 0
+node_count_nemo = -1
+node_count_maxtext = -1
+node_count_llama_3_7b = -1
+a3_mega_consumption_model = "Reservation"
+a3_ultra_consumption_model = ""

applications/hcc/a3ultra_workloads.tfvars

@@ -0,0 +1,54 @@
+/**
+  * Copyright 2023 Google LLC
+  *
+  * Licensed under the Apache License, Version 2.0 (the "License");
+  * you may not use this file except in compliance with the License.
+  * You may obtain a copy of the License at
+  *
+  *      http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+
+authorized_cidr = "0.0.0.0/0"
+
+goog_cm_deployment_name = "ultra-cluster-test"
+
+labels = {
+  ghpc_blueprint  = "gke-a3-ultra"
+  ghpc_deployment = "a3ultra-hcc-test"
+}
+
+project_id = "gke-ai-eco-dev"
+
+a3_mega_zone = ""
+a3_ultra_zone = "europe-west1-b"
+
+node_count_gke_nccl = -1
+node_count_gke = 0
+node_count_nemo = -1
+node_count_maxtext = -1
+node_count_llama_3_7b = -1
+
+# A3 Ultra recipe options:
+# - "gke"
+# - "gke-nccl"
+# - "llama3.1_7b_nemo_pretraining"
+# - "llama3.1_70b_nemo_pretraining"
+# - "llama3.1_70b_maxtext_pretraining"
+# - "mixtral8_7b_nemo_pretraining"
+# - "mixtral8_7b_maxtext_pretraining"
+a3ultra_recipe = "gke"
+a3mega_recipe=""
+
+reservation = ""
+reservation_block = ""
+placement_policy_name = ""
+
+gpu_type = "A3 Ultra"
+a3_ultra_consumption_model = "Reservation"
+a3_mega_consumption_model = ""