Description
Heya,
I was wondering something a bit drastic - what would it look like if this provider was used as a data-only provider, and you deferred creation of kubernetes resources to the hashi/kubernetes provider?
That would mean (I think, maybe I'm wrong) that this wouldn't need to deal with cluster access at all - it would just run kustomize locally, that's it. You'd immediately be able to punt on all the "please make this work with X kubeconfig" issues, as it wouldn't be be your problem at all. Let alone any logic to do with actually applying changes to the cluster.
I had a bit of a play, and this actually seems to work OK already. kustomization_overlay is used with absolutely no changes:
data "kustomization_overlay" "simpy" {
resources = [
"${local.here}/manifests"
]
patches {
target {
kind = "Namespace"
name = "simpy-REPLACE_ME"
}
patch = jsonencode({
"kind" : "Namespace"
"metadata" : {
"name" : "simpy-${var.stage}"
"labels" : {
"accurate.cybozu.com/parent" : "${var.stage}-apps"
}
}
})
}
namespace = "simpy-${var.stage}"
images {
name = "SIMPY_IMAGE"
new_name = local.image_uri_repo
digest = local.image_uri_digest
}
}
module "simpy" {
source = "./modules/kustomization"
kustomization = data.kustomization_overlay.simpy
}./modules/kustomization is nearly identical to your recommended apply system, just with kustomization_resource swapped out for kubernetes_manifest and a couple of very minor tweaks:
kubernetes_manifestswantsjsondecode(manifest: string)instead ofmanifest: stringwaitis a bit annoying and will probably need a resource type switch on it.
terraform {
required_providers {
# kustomization = {
# source = "kbst/kustomization"
# }
kubernetes = {
source = "hashicorp/kubernetes"
}
}
}
variable "kustomization" {
type = object({
ids = set(string)
ids_prio = list(set(string))
manifests = map(string)
})
}
# first loop through resources in ids_prio[0]
resource "kubernetes_manifest" "p0" {
for_each = var.kustomization.ids_prio[0]
manifest = (
contains(["_/Secret"], regex("(?P<group_kind>.*/.*)/.*/.*", each.value)["group_kind"])
? sensitive(jsondecode(var.kustomization.manifests[each.value]))
: jsondecode(var.kustomization.manifests[each.value])
)
}
# then loop through resources in ids_prio[1]
# and set an explicit depends_on on kubernetes_manifest.p0
# wait 2 minutes for any deployment or daemonset to become ready
resource "kubernetes_manifest" "p1" {
for_each = var.kustomization.ids_prio[1]
manifest = (
contains(["_/Secret"], regex("(?P<group_kind>.*/.*)/.*/.*", each.value)["group_kind"])
? sensitive(jsondecode(var.kustomization.manifests[each.value]))
: jsondecode(var.kustomization.manifests[each.value])
)
# TODO - this doesn't quite work yet, the hashi provider crashes if this is run
# on something that doesn't have a rollout
# wait {
# rollout = true
# }
timeouts {
create = "2m"
update = "2m"
}
depends_on = [kubernetes_manifest.p0]
}
# finally, loop through resources in ids_prio[2]
# and set an explicit depends_on on kubernetes_manifest.p1
resource "kubernetes_manifest" "p2" {
for_each = var.kustomization.ids_prio[2]
manifest = (
contains(["_/Secret"], regex("(?P<group_kind>.*/.*)/.*/.*", each.value)["group_kind"])
? sensitive(jsondecode(var.kustomization.manifests[each.value]))
: var.kustomization.manifests[each.value]
)
depends_on = [kubernetes_manifest.p1]
}The only hiccup I've hit so far is the ID format - kubernetes wants IDs to look like apiVersion=<string>,kind=<string>,[namespace=<string>,]name=<string> link. This doesn't seem block usage as above, it just makes importing a bit annoying as you can't copy/paste the IDs quite as easily.
What are your thoughts? It can kinda be used like this already, and this would make an absolute boatload of concerns here just disappear as not being your problem to deal with anymore - you'd no longer be responsible for doing anything with the cluster at all. :)
EDIT: missed these related discussions because I can't search properly: