Skip to content

Commit c48c2b7

Browse files
umairidrisumairidris
authored
feat: Add module to wait for data fusion instance to become healthy (#31)
* add wait_healthy module * rm make file * fix name in readme and add headers
1 parent 34b419a commit c48c2b7

File tree

5 files changed

+248
-0
lines changed

5 files changed

+248
-0
lines changed

‎modules/wait_healthy/README.md

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# Data Fusion Wait Healthy
2+
3+
This module provides a way to wait for a Data Fusion Instance to become healthy.
4+
5+
Data Fusion Instances currently load their services in an asynchronous
6+
manner. Thus, if one tries to create a pipeline or resource immediately after
7+
the Data Fusion Instance has been created, they will get an error.
8+
9+
Rather than waiting for an arbitrary amount of time, this module will ping the
10+
Data Fusion Instance until it has loaded the google-cloud plugin, which usually
11+
indicates a healthy instance.
12+
13+
NOTE: In the future this module may be removed if Data Fusion allows users to
14+
wait for specific services during instance creation.
15+
16+
## Usage
17+
18+
Basic usage of this module is as follows:
19+
20+
```hcl
21+
data "google_client_config" "current" {}
22+
23+
resource "google_data_fusion_instance" "instance" {
24+
provider = google-beta
25+
name = "example-instance"
26+
project = "example-project"
27+
region = "us-central1"
28+
type = "BASIC"
29+
}
30+
31+
module "wait_healthy" {
32+
source = "terraform-google-modules/data-fusion/google//modules/wait_healthy"
33+
version = "~> 0.1"
34+
35+
service_endpoint = google_data_fusion_instance.instance.service_endpoint
36+
access_token = data.google_client_config.current.access_token
37+
}
38+
39+
provider "cdap" {
40+
host = "${module.wait_healthy.service_endpoint}/api/"
41+
token = data.google_client_config.current.access_token
42+
}
43+
44+
resource "cdap_application" "app" {
45+
...
46+
}
47+
```
48+
49+
Functional examples are included in the
50+
[examples](./examples/) directory.
51+
52+
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
53+
## Inputs
54+
55+
| Name | Description | Type | Default | Required |
56+
|------|-------------|:----:|:-----:|:-----:|
57+
| access\_token | Token for authenticating requests to the CDF instance. | string | n/a | yes |
58+
| service\_endpoint | Endpoint of the the CDF instance to check for health. | string | n/a | yes |
59+
60+
## Outputs
61+
62+
| Name | Description |
63+
|------|-------------|
64+
| service\_endpoint | The input service_endpoint. Useful for creating a dependency on this module. |
65+
66+
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
67+
68+
## Requirements
69+
70+
These sections describe requirements for using this module.
71+
72+
### Software
73+
74+
The following dependencies must be available:
75+
76+
- [Terraform][terraform] v0.12
77+
- [Terraform Provider for GCP][terraform-provider-gcp] plugin v2.0
78+
79+
### Service Account
80+
81+
A service account with the following roles must be used to provision
82+
the resources of this module:
83+
84+
- Data Fusion Admin: `roles/datafusion.admin`
85+
86+
The [Project Factory module][project-factory-module] and the
87+
[IAM module][iam-module] may be used in combination to provision a
88+
service account with the necessary roles applied.
89+
90+
### APIs
91+
92+
A project with the following APIs enabled must be used to host the
93+
resources of this module:
94+
95+
- Google Cloud Data Fusion API: `datafusion.googleapis.com`
96+
97+
The [Project Factory module][project-factory-module] can be used to
98+
provision a project with the necessary APIs enabled.
99+
100+
## Contributing
101+
102+
Refer to the [contribution guidelines](./CONTRIBUTING.md) for
103+
information on contributing to this module.
104+
105+
[iam-module]: https://registry.terraform.io/modules/terraform-google-modules/iam/google
106+
[project-factory-module]: https://registry.terraform.io/modules/terraform-google-modules/project-factory/google
107+
[terraform-provider-gcp]: https://www.terraform.io/docs/providers/google/index.html
108+
[terraform]: https://www.terraform.io/downloads.html

‎modules/wait_healthy/main.tf

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
/**
2+
* Copyright 2020 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
resource "null_resource" "wait_healthy" {
18+
provisioner "local-exec" {
19+
command = "${path.module}/wait_healthy.sh -e ${var.service_endpoint} -t ${var.access_token}"
20+
}
21+
}

‎modules/wait_healthy/outputs.tf

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
/**
2+
* Copyright 2020 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
output "service_endpoint" {
18+
description = "The input service_endpoint. Useful for creating a dependency on this module."
19+
value = var.service_endpoint
20+
depends_on = [
21+
null_resource.wait_healthy,
22+
]
23+
}

‎modules/wait_healthy/variables.tf

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
/**
2+
* Copyright 2020 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
variable "service_endpoint" {
18+
description = "Endpoint of the the CDF instance to check for health."
19+
type = string
20+
}
21+
22+
variable "access_token" {
23+
description = "Token for authenticating requests to the CDF instance."
24+
type = string
25+
}

‎modules/wait_healthy/wait_healthy.sh

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
#!/bin/bash
2+
# Copyright 2018 Google LLC
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
set -e
17+
18+
ENDPOINT=""
19+
ACCESS_TOKEN=""
20+
21+
while getopts 'e:t:' c
22+
do
23+
case $c in
24+
e) ENDPOINT=${OPTARG} ;;
25+
t) ACCESS_TOKEN=${OPTARG} ;;
26+
*)
27+
echo "Invalid flag ${OPTARG}"
28+
exit 1
29+
;;
30+
esac
31+
done
32+
33+
if [[ -z ${ENDPOINT} ]]; then
34+
echo "-e must be set"
35+
exit 1
36+
fi
37+
38+
if [[ -z ${ACCESS_TOKEN} ]]; then
39+
echo "-t must be set"
40+
exit 1
41+
fi
42+
43+
44+
for ((i=1;i<=50;i++)); do
45+
sleep 10
46+
resp=$(curl -s -w "\n%{http_code}" -H "Authorization: Bearer ${ACCESS_TOKEN}" "${ENDPOINT}/api/v3/namespaces/default/artifacts")
47+
48+
code=$(tail -n 1 <<< "${resp}")
49+
body=$(head -n 1 <<< "${resp}")
50+
51+
if [[ "${code}" == 200 ]]; then
52+
name=$(jq -r '.[] | select(.name=="google-cloud") | .name' <<< "${body}")
53+
if [[ "${name}" == "google-cloud" ]]; then
54+
echo "Artifact google-cloud loaded, instance healthy"
55+
exit 0
56+
fi
57+
58+
echo "Artifact google-cloud not loaded (${name}), instance unhealthy, checking again in 10 seconds"
59+
60+
elif [[ "${code}" =~ ^(400|502|504)$ ]]; then
61+
echo "Instance unhealthy: ${code}, checking again in 10 seconds"
62+
63+
else
64+
echo "Unexpected code: ${code}"
65+
echo "${body}"
66+
exit 1
67+
fi
68+
done
69+
70+
echo "Failed to wait for instance to become healthy in allocated time"
71+
exit 1

0 commit comments

Comments
 (0)