Add module to wait for data fusion instance to become healthy

modules/wait_healthy/README.md

@@ -0,0 +1,108 @@
+# Data Fusion Wait Healthy
+
+This module provides a way to wait for a Data Fusion Instance to become healthy.
+
+Data Fusion Instances currently load their services in an asynchronous
+manner. Thus, if one tries to create a pipeline or resource immediately after
+the Data Fusion Instance has been created, they will get an error.
+
+Rather than waiting for an arbitrary amount of time, this module will ping the
+Data Fusion Instance until it has loaded the google-cloud plugin, which usually
+indicates a healthy instance.
+
+NOTE: In the future this module may be removed if Data Fusion allows users to
+wait for specific services during instance creation.
+
+## Usage
+
+Basic usage of this module is as follows:
+
+```hcl
+data "google_client_config" "current" {}
+
+resource "google_data_fusion_instance" "instance" {
+  provider = google-beta
+  name     = "example-instance"
+  project  = "example-project"
+  region   = "us-central1"
+  type     = "BASIC"
+}
+
+module "wait_healthy" {
+  source  = "terraform-google-modules/data-fusion/google//modules/wait_healthy"
+  version = "~> 0.1"
+
+  service_endpoint = google_data_fusion_instance.instance.service_endpoint
+  access_token     = data.google_client_config.current.access_token
+}
+
+provider "cdap" {
+  host  = "${module.wait_healthy.service_endpoint}/api/"
+  token = data.google_client_config.current.access_token
+}
+
+resource "cdap_application" "app" {
+   ...
+}
+```
+
+Functional examples are included in the
+[examples](./examples/) directory.
+
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|:----:|:-----:|:-----:|
+| access\_token | Token for authenticating requests to the CDF instance. | string | n/a | yes |
+| service\_endpoint | Endpoint of the the CDF instance to check for health. | string | n/a | yes |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| service\_endpoint | The input service_endpoint. Useful for creating a dependency on this module. |
+
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+
+## Requirements
+
+These sections describe requirements for using this module.
+
+### Software
+
+The following dependencies must be available:
+
+- [Terraform][terraform] v0.12
+- [Terraform Provider for GCP][terraform-provider-gcp] plugin v2.0
+
+### Service Account
+
+A service account with the following roles must be used to provision
+the resources of this module:
+
+- Data Fusion Admin: `roles/datafusion.admin`
+
+The [Project Factory module][project-factory-module] and the
+[IAM module][iam-module] may be used in combination to provision a
+service account with the necessary roles applied.
+
+### APIs
+
+A project with the following APIs enabled must be used to host the
+resources of this module:
+
+- Google Cloud Data Fusion API: `datafusion.googleapis.com`
+
+The [Project Factory module][project-factory-module] can be used to
+provision a project with the necessary APIs enabled.
+
+## Contributing
+
+Refer to the [contribution guidelines](./CONTRIBUTING.md) for
+information on contributing to this module.
+
+[iam-module]: https://registry.terraform.io/modules/terraform-google-modules/iam/google
+[project-factory-module]: https://registry.terraform.io/modules/terraform-google-modules/project-factory/google
+[terraform-provider-gcp]: https://www.terraform.io/docs/providers/google/index.html
+[terraform]: https://www.terraform.io/downloads.html

modules/wait_healthy/main.tf

@@ -0,0 +1,21 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+resource "null_resource" "wait_healthy" {
+  provisioner "local-exec" {
+    command = "${path.module}/wait_healthy.sh -e ${var.service_endpoint} -t ${var.access_token}"
+  }
+}

modules/wait_healthy/outputs.tf

@@ -0,0 +1,23 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+output "service_endpoint" {
+    description = "The input service_endpoint. Useful for creating a dependency on this module."
+    value = var.service_endpoint
+    depends_on = [
+        null_resource.wait_healthy,
+    ]
+}

modules/wait_healthy/variables.tf

@@ -0,0 +1,25 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+variable "service_endpoint" {
+    description = "Endpoint of the the CDF instance to check for health."
+    type = string
+}
+
+variable "access_token" {
+    description = "Token for authenticating requests to the CDF instance."
+    type = string
+}

modules/wait_healthy/wait_healthy.sh

@@ -0,0 +1,71 @@
+#!/bin/bash
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+
+ENDPOINT=""
+ACCESS_TOKEN=""
+
+while getopts 'e:t:' c
+do
+  case $c in
+    e) ENDPOINT=${OPTARG} ;;
+    t) ACCESS_TOKEN=${OPTARG} ;;
+    *)
+      echo "Invalid flag ${OPTARG}"
+      exit 1
+      ;;
+  esac
+done
+
+if [[ -z ${ENDPOINT} ]]; then
+    echo "-e must be set"
+    exit 1
+fi
+
+if [[ -z ${ACCESS_TOKEN} ]]; then
+    echo "-t must be set"
+    exit 1
+fi
+
+
+for ((i=1;i<=50;i++)); do
+    sleep 10
+    resp=$(curl -s -w "\n%{http_code}" -H "Authorization: Bearer ${ACCESS_TOKEN}" "${ENDPOINT}/api/v3/namespaces/default/artifacts")
+
+    code=$(tail -n 1 <<< "${resp}")
+    body=$(head -n 1 <<< "${resp}")
+
+   if [[ "${code}" == 200 ]]; then
+        name=$(jq -r '.[] | select(.name=="google-cloud") | .name' <<< "${body}")
+        if [[ "${name}" == "google-cloud" ]]; then
+            echo "Artifact google-cloud loaded, instance healthy"
+            exit 0
+        fi
+
+        echo "Artifact google-cloud not loaded (${name}), instance unhealthy, checking again in 10 seconds"
+
+    elif [[ "${code}" =~ ^(400|502|504)$ ]]; then
+        echo "Instance unhealthy: ${code}, checking again in 10 seconds"
+
+    else
+        echo "Unexpected code: ${code}"
+        echo "${body}"
+        exit 1
+    fi
+done
+
+echo "Failed to wait for instance to become healthy in allocated time"
+exit 1