Skip to content

Commit 59a0d19

Browse files
hankfreundhankfreund
authored
feat: Add terraform samples for GKE Attached clusters. (#667)
* feat: Add Terraform samples for GKE Attached clusters. Add samples demonstrating use of GKE attached clusters with both AKS and EKS. The samples: 1. create a cluster 2. prepare the cluster for attach 3. attach the cluster as a single Terraform apply step which has been problematic for users. * Fix lint issues. * Add the multicloud team as owners for attached clusters samples.
1 parent 56ac603 commit 59a0d19

File tree

18 files changed

+2371
-0
lines changed

18 files changed

+2371
-0
lines changed

‎.github/workflows/ci_any_pr.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@ jobs:
5454
# list of directories in the repo that hosts Terraform samples
5555
# update this list as new terraform samples are added to the repo
5656
tf-sample: [
57+
'anthos-attached-clusters',
5758
'anthos-bm-gcp-terraform',
5859
'anthos-bm-openstack-terraform',
5960
'anthos-multi-cloud/AWS',

‎.github/workflows/ci_main_branch.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@ jobs:
5454
# list of directories in the repo that hosts Terraform samples
5555
# update this list as new terraform samples are added to the repo
5656
tf-sample: [
57+
'anthos-attached-clusters',
5758
'anthos-bm-gcp-terraform',
5859
'anthos-bm-openstack-terraform',
5960
'anthos-multi-cloud/AWS',

‎CODEOWNERS

+1
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
/aws-logging-monitoring/ @GoogleCloudPlatform/onyx-gke-observability @yoshi-approver
1111
/attached-logging-monitoring/ @GoogleCloudPlatform/onyx-gke-observability @yoshi-approver
1212
/gmp-grafana-dashboards/ @GoogleCloudPlatform/onyx-gke-observability @yoshi-approver
13+
/anthos-attached-clusters/ @GoogleCloudPlatform/anthos-multicloud @yoshi-approver
1314
/anthos-multi-cloud/ @GoogleCloudPlatform/anthos-multicloud @yoshi-approver
1415
/anthos-bm-utils/ @GoogleCloudPlatform/anthos-baremetal-eng @yoshi-approver
1516
/anthos-bm-apigee/ @GoogleCloudPlatform/app-mod-customer-engineers @yoshi-approver

‎anthos-attached-clusters/AAC-architecture.svg

+1,459
Loading

‎anthos-attached-clusters/README.md

+84
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Using Terraform with GKE Attached Clusters
2+
3+
The scripts provided here demonstrate how you can use Terraform to automate the process of
4+
creating, bootstrapping, and attaching a Kubernetes cluster. For a complete reference of the
5+
GKE attached clusters Terraform resource, see the
6+
[google_container_attached_cluster](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_attached_cluster)
7+
and
8+
[google_container_attached_install_manifest](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/container_attached_install_manifest)
9+
documentation.
10+
11+
With GKE attached clusters you can manage any standard, [CNCF-compliant](https://www.cncf.io/certification/software-conformance/)
12+
Kubernetes installation, including clusters already in production. You can then add
13+
[Google Kubernetes Engine (GKE) Enterprise edition](https://cloud.google.com/kubernetes-engine)
14+
features to standardize and secure your clusters across multiple cloud environments and Kubernetes
15+
vendors. To learn more, see the
16+
[GKE attached clusters documentation](https://cloud.google.com/anthos/clusters/docs/multi-cloud/attached).
17+
18+
![GKE Attached Clusters](./AAC-architecture.svg)
19+
20+
## High Level Process
21+
22+
Attaching a cluster in GKE involves taking the steps below. The Terraform scripts provided in this
23+
github repository automatically perform steps 1-4, and are meant to provide a quick start to
24+
working with GKE attached clusters.
25+
1. Create a cluster.
26+
1. Invoke the [GenerateAttachedClustersInstallManifest](https://cloud.google.com/anthos/clusters/docs/multi-cloud/reference/rest/v1/projects.locations/generateAttachedClusterInstallManifest)
27+
API to retrieve a manifest of the bootstrapping deployment.
28+
1. Apply the manifest from step 1 to the cluster.
29+
1. Invoke the [Create](https://cloud.google.com/anthos/clusters/docs/multi-cloud/reference/rest/v1/projects.locations.attachedClusters/create)
30+
API to attach the cluster.
31+
1. (Optional) Delete the resources applied in step 2.
32+
33+
## What Is Provided
34+
35+
* The **AKS** folder contains a sample script for creating an AKS cluster on Azure and attaching it.
36+
* The **EKS** folder contains a sample script for creating an EKS cluster on AWS and attaching it.
37+
* The **modules** folder contains the `attached-install-manifest` module which demonstrates how to
38+
retrieve the manifest and apply it to the cluster using Helm. Both the AKS and EKS examples use it.
39+
40+
## Prerequisites
41+
42+
All samples assume the availability of ambient credentials, which are the default credentials
43+
automatically provided in the environment where you run the Terraform scripts. These credentials
44+
are typically obtained by authenticating your account using the Google Cloud SDK (gcloud) with the
45+
command `gcloud auth application-default login`. See the
46+
[documentation](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login)
47+
for more information.
48+
49+
The prerequisites for running the Terraform scripts are the following:
50+
1. Ensure the latest version of gcloud is [installed](https://cloud.google.com/sdk/docs/install).
51+
1. If you haven't already done so, [create](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project)
52+
your Google Cloud project. This generates a Google Cloud project ID and a project number.
53+
1. Set your active Google Cloud project and authenticate your account with the following commands:
54+
```sh
55+
export PROJECT_ID=<your project id>
56+
gcloud auth login
57+
gcloud config set project $PROJECT_ID
58+
gcloud auth application-default login
59+
```
60+
1. Enable the GKE attached clusters API and its required services with the following commands:
61+
```sh
62+
gcloud services enable gkemulticloud.googleapis.com
63+
gcloud services enable gkeconnect.googleapis.com
64+
gcloud services enable connectgateway.googleapis.com
65+
gcloud services enable cloudresourcemanager.googleapis.com
66+
gcloud services enable anthos.googleapis.com
67+
gcloud services enable logging.googleapis.com
68+
gcloud services enable monitoring.googleapis.com
69+
gcloud services enable opsconfigmonitoring.googleapis.com
70+
```
71+
1. Clusters will be created at a specific Kubernetes version. Attached clusters have an additional
72+
platform version that must be specified. The platform version’s _major.minor_ should match the
73+
cluster’s Kubernetes version. Both versions must be specified when attaching.
74+
You can list all supported platform versions using:
75+
```sh
76+
gcloud container attached get-server-config --location=GOOGLE_CLOUD_REGION
77+
```
78+
There is also a Terraform data source that provides the same information:
79+
```
80+
data "google_container_attached_versions" "versions" {
81+
location = GCP_LOCATION
82+
project = GCP_PROJECT_ID
83+
}
84+
```

‎anthos-attached-clusters/aks/README.md

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Attach an AKS cluster using Terraform
2+
3+
## Prerequisites
4+
The sample assumes the availability of ambient credentials, which are the default credentials
5+
automatically provided in the environment where you run the Terraform scripts. For instructions
6+
on authenticating to Azure, see the
7+
[Terraform documentation](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#authenticating-to-azure)
8+
on the topic.
9+
1. Ensure the latest version of the Azure CLI is [installed](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli)
10+
and logged in
11+
12+
## Usage
13+
14+
1. Edit the values in the terraform.tfvars file to suit your needs. Descriptions for each variable
15+
can be found in `variables.tf`. Additional optional features are also available and commented out
16+
in the `google_container_attached_cluster` resource in `main.tf`.
17+
18+
If you modify the cluster creation, ensure it meets
19+
[Cluster Prerequisites](https://cloud.google.com/anthos/clusters/docs/multi-cloud/attached/aks/reference/cluster-prerequisites).
20+
1. Initialize Terraform:
21+
```bash
22+
terraform init
23+
```
24+
1. Create and apply the plan:
25+
```bash
26+
terraform apply
27+
```
28+
1. The process should take about 10 minutes to complete.
29+
1. Login to the cluster:
30+
```bash
31+
gcloud container attached clusters get-credentials CLUSTER_NAME
32+
```
33+
This will allow you to access the cluster using kubectl, if appropriate RBAC permissions have
34+
been applied. For more information, see [Connect to your AKS Cluster](https://cloud.google.com/anthos/clusters/docs/multi-cloud/attached/aks/how-to/connect-to-cluster).
35+

‎anthos-attached-clusters/aks/main.tf

+125
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
/**
2+
* Copyright 2024 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
locals {
18+
tags = {
19+
"owner" = var.owner
20+
}
21+
}
22+
23+
data "azurerm_subscription" "current" {
24+
}
25+
26+
resource "azurerm_resource_group" "aks" {
27+
name = "${var.name_prefix}-rg"
28+
location = var.azure_region
29+
tags = merge(local.tags, var.tags)
30+
}
31+
32+
resource "azurerm_kubernetes_cluster" "aks" {
33+
name = "${var.name_prefix}-cluster"
34+
location = azurerm_resource_group.aks.location
35+
resource_group_name = azurerm_resource_group.aks.name
36+
dns_prefix = "${var.name_prefix}-dns"
37+
kubernetes_version = var.k8s_version
38+
39+
# If not enabling the OIDC issuer, extra steps need to be taken to manually retrieve JWKs from the cluster.
40+
oidc_issuer_enabled = true
41+
42+
default_node_pool {
43+
name = "default"
44+
node_count = var.node_count
45+
vm_size = "Standard_D2_v2"
46+
tags = merge(local.tags, var.tags)
47+
}
48+
49+
identity {
50+
type = "SystemAssigned"
51+
}
52+
53+
tags = merge(local.tags, var.tags)
54+
}
55+
56+
data "google_project" "project" {
57+
}
58+
59+
provider "helm" {
60+
alias = "bootstrap_installer"
61+
kubernetes {
62+
host = azurerm_kubernetes_cluster.aks.kube_config.0.host
63+
username = azurerm_kubernetes_cluster.aks.kube_config.0.username
64+
password = azurerm_kubernetes_cluster.aks.kube_config.0.password
65+
client_certificate = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate)
66+
client_key = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_key)
67+
cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)
68+
}
69+
}
70+
71+
module "attached_install_manifest" {
72+
source = "../modules/attached-install-manifest"
73+
attached_cluster_name = "${var.name_prefix}-cluster"
74+
attached_cluster_fleet_project = data.google_project.project.project_id
75+
gcp_location = var.gcp_location
76+
platform_version = var.platform_version
77+
providers = {
78+
helm = helm.bootstrap_installer
79+
}
80+
depends_on = [
81+
azurerm_kubernetes_cluster.aks
82+
]
83+
}
84+
85+
resource "google_container_attached_cluster" "primary" {
86+
name = "${var.name_prefix}-cluster"
87+
project = data.google_project.project.project_id
88+
location = var.gcp_location
89+
description = "AKS attached cluster example"
90+
distribution = "aks"
91+
platform_version = var.platform_version
92+
oidc_config {
93+
issuer_url = azurerm_kubernetes_cluster.aks.oidc_issuer_url
94+
# NOTE: If `oidc_issuer_enabled` is not set to true above, `jwks` needs to be set here.
95+
# JWKs can be retrieved from the cluster using: `kubectl get --raw /openid/v1/jwks` and
96+
# must be base64 encoded.
97+
}
98+
fleet {
99+
project = "projects/${data.google_project.project.number}"
100+
}
101+
102+
# Optional:
103+
# logging_config {
104+
# component_config {
105+
# enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
106+
# }
107+
# }
108+
109+
# Optional:
110+
# monitoring_config {
111+
# managed_prometheus_config {
112+
# enabled = true
113+
# }
114+
# }
115+
116+
# Optional:
117+
# authorization {
118+
# admin_users = ["[email protected]", "[email protected]"]
119+
# admin_groups = ["[email protected]", "[email protected]"]
120+
# }
121+
122+
depends_on = [
123+
module.attached_install_manifest
124+
]
125+
}

‎anthos-attached-clusters/aks/provider.tf

+40
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
/**
2+
* Copyright 2024 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
terraform {
18+
required_providers {
19+
azurerm = {
20+
source = "hashicorp/azurerm"
21+
version = ">=3.17.0"
22+
}
23+
google = {
24+
source = "hashicorp/google"
25+
version = ">=5.0.0"
26+
}
27+
helm = {
28+
source = "hashicorp/helm"
29+
}
30+
}
31+
required_version = ">= 0.13"
32+
}
33+
34+
provider "azurerm" {
35+
features {}
36+
}
37+
38+
provider "google" {
39+
project = var.gcp_project_id
40+
}

‎anthos-attached-clusters/aks/variables.tf

+66
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
/**
2+
* Copyright 2024 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
variable "owner" {
18+
description = "Owner of the resources"
19+
type = string
20+
}
21+
22+
variable "name_prefix" {
23+
description = "Common prefix to use for generating names"
24+
type = string
25+
}
26+
27+
variable "azure_region" {
28+
description = "Azure region to deploy to"
29+
type = string
30+
default = "East US"
31+
}
32+
33+
variable "tags" {
34+
description = "List of tags to apply to resources"
35+
type = map(string)
36+
default = {}
37+
}
38+
39+
variable "k8s_version" {
40+
description = "Kubernetes version of the AKS cluster"
41+
type = string
42+
default = "1.28"
43+
}
44+
45+
variable "node_count" {
46+
description = "The number of nodes in the default node pool"
47+
type = number
48+
default = 1
49+
}
50+
51+
variable "gcp_project_id" {
52+
description = "The GCP project id where the cluster will be registered"
53+
type = string
54+
}
55+
56+
variable "gcp_location" {
57+
description = "GCP location to create the attached resource in"
58+
type = string
59+
default = "us-west1"
60+
}
61+
62+
variable "platform_version" {
63+
description = "Platform version of the attached cluster resource"
64+
type = string
65+
default = "1.28.0-gke.3"
66+
}

0 commit comments

Comments
 (0)