Update kustomization_user_deploy docs and examples

Author: Markus Heinemann

README.md

@@ -178,7 +178,7 @@ As Cilium has a lot of interesting and powerful config possibilities, we give yo
 
 Cilium supports full kube-proxy replacement. Cilium runs by default in hybrid kube-proxy replacement mode. To achieve a completely kube-proxy-free cluster, set `disable_kube_proxy = true`.
 
-It is also possible to enable Hubble using `cilium_hubble_enabled = true`. In order to access the Hubble UI, you need to port-forward the Hubble UI service to your local machine. By default, you can do this by running `kubectl port-forward -n kube-system service/hubble-ui 12000:80` and then opening `http://localhost:12000` in your browser. 
+It is also possible to enable Hubble using `cilium_hubble_enabled = true`. In order to access the Hubble UI, you need to port-forward the Hubble UI service to your local machine. By default, you can do this by running `kubectl port-forward -n kube-system service/hubble-ui 12000:80` and then opening `http://localhost:12000` in your browser.
 However, it is recommended to use the [Cilium CLI](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-the-cilium-cli) and [Hubble Client](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-the-cilium-cli) and running the `cilium hubble ui` command.
 
 ## Scaling Nodes
@@ -293,7 +293,7 @@ If you need to install additional Helm charts or Kubernetes manifests that are n
 
 These files need to be valid `Kustomization` manifests, additionally supporting terraform templating! (The templating parameters can be passed via the `extra_kustomize_parameters` variable (via a map) to the module).
 
-All files in the `extra-manifests` directory including the rendered versions of the `*.yaml.tpl` will be applied to k3s with `kubectl apply -k` (which will be executed after and independently of the basic cluster configuration).
+All files in the `extra-manifests` directory and its subdirectories including the rendered versions of the `*.yaml.tpl` will be applied to k3s with `kubectl apply -k` (which will be executed after and independently of the basic cluster configuration).
 
 See a working example in [examples/kustomization_user_deploy](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/tree/master/examples/kustomization_user_deploy).
 

examples/kustomization_user_deploy/README.md

@@ -1,34 +1,145 @@
 # How install deploy a additional / extra stuff while terraforming the cluster
 
-## With a `HelmChart` and `HelmChartConfig`
+Kube-Hetzner allows to provide user-defined resources after the initial setup of the Kubernetes cluster. Additional Kustomize scripts in the `extra-manifests` directory with the extension `.yaml.tpl` are recursively copied onto the control plane and are deployed with `kubectl apply -k`. The main entry point is the file called `kustomization.yaml.tpl`. Therefor the names of other manifests must be listed in the `kustomization.yaml.tpl` `recources` without the `.tpl` extension.
 
-This is how it worked for me, note I'm a total beginner with kustomize.<br>
-Pretty sure I butchered a lot ;)
+The manifests are deployed automaticly when executing `terrafrom apply`.
 
-### Assuming you followed the `DO not Skip` part of the installation
+## Examples
 
-In the project folder, where the `kube.tf` is located.<br>
-Create a folder named `extra-manifests`<br>
-In it create a file named `kustomization.yaml.tpl`<br>
-and **your** manifest file(s), there names must be listed in the `kustomization.yaml.tpl` `recources` without the `.tpl`<br>
-e.g. `some-random-name.yaml.tpl`
+Some examples for the most common usecases are provided.
 
-## Apply the kustomized configuration
+> When tring out the demos, make sure that the files from the demo folders are located in the `extra-manifests` directory.
 
-Assuming no errors have been made, apply this by run `terraform apply`<br>
+### Deploying simple resources
 
-## ReRun the kustomization (debugging)
+The easiest use case is to deploy some simple resources to the cluster. Since the kustomize resources are [terrafrom template files](https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/file) the can make use of parameters provided in the `extra_kustomize_parameters` map of the `kube.tf`.
 
-In the highly unlikely case that an actual error has occurred...<br>
-Anyway, you can rerun just the kustomization part like this:
+#### `kube.tf`
 
-    terraform apply -replace='module.kube-hetzner.null_resource.kustomization_user["kustomization.yaml.tpl"]' --auto-approve
+```tf
+...
+extra_kustomize_parameters = {
+  my_config_key = "somestring"
+}
+...
+```
+
+The variable can be used in any `.yaml.tpl` manifest
+
+#### `configmap.tf`
+
+```
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: demo-config
+  data:
+    someConfigKey: ${sealed_secrets_crt}
+```
+
+For a full example see [simple-resources](simple-resources/)
+
+### Deploying a Helm Chart
+
+You may want to deploy a helm chart to your cluster. K3s includes a [Helm Chart controller](https://docs.k3s.io/helm) which provides the CRDs `HelmChart` and `HelmChartConfig`.
+
+For a full demo see the [helm-chart](helm-chart/) example.
+
+### Multiple Namespaces
+
+In some more complex uses cases you may want to deploy to multi namespaces with a common base. This behaviour is [supported by Kustomize](https://github.com/kubernetes-sigs/kustomize/blob/master/examples/multibases/multi-namespace.md) and can be done with Kube-Hetzner since it considers all subdirectors of `extra-manifests` as well.
+
+For a full demo see the [multiple-namespaces](multiple-namespaces/) example.
+
+## Debugging
 
 Check what kustomization exists:
 
-    (⎈|dev3:default)➜ dev3-cluster (main) ✗ terraform state list | grep kustom
-    ...
-    module.kube-hetzner.null_resource.kustomization
-    module.kube-hetzner.null_resource.kustomization_user["some-random-name.yaml.tpl"]
-    module.kube-hetzner.null_resource.kustomization_user["kustomization.yaml.tpl"]
-    ...
+```
+$ terraform state list | grep kustom
+  ...
+  module.kube-hetzner.null_resource.kustomization
+  module.kube-hetzner.null_resource.kustomization_user["demo-config-map.yaml.tpl"]
+  module.kube-hetzner.null_resource.kustomization_user["demo-pod.yaml.tpl"]
+  module.kube-hetzner.null_resource.kustomization_user["kustomization.yaml.tpl"]
+  ...
+```
+
+You can rerun just the kustomization part like this:
+
+```
+terraform apply -replace='module.kube-hetzner.null_resource.kustomization_user["kustomization.yaml.tpl"]' --auto-approve
+```
+
+# How to Install and Deploy Additional Resources with Terraform and Kube-Hetzner
+
+Kube-Hetzner allows you to provide user-defined resources after the initial setup of the Kubernetes cluster. You can deploy additional resources using Kustomize scripts in the `extra-manifests` directory with the extension `.yaml.tpl`. These scripts are recursively copied onto the control plane and deployed with `kubectl apply -k`. The main entry point for these additional resources is the `kustomization.yaml.tpl` file. In this file, you need to list the names of other manifests without the `.tpl` extension in the resources section.
+
+When you execute terraform apply, the manifests in the extra-manifests directory, including the rendered versions of the `*.yaml.tpl` files, will be automatically deployed to the cluster.
+
+## Examples
+
+Here are some examples of common use cases for deploying additional resources:
+
+> **Note**: When trying out the demos, make sure that the files from the demo folders are located in the `extra-manifests` directory.
+
+### Deploying Simple Resources
+
+The easiest use case is to deploy simple resources to the cluster. Since the Kustomize resources are [Terraform template](https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/file) files, they can make use of parameters provided in the `extra_kustomize_parameters` map of the `kube.tf` file.
+
+#### **`kube.tf`**
+
+```
+...
+extra_kustomize_parameters = {
+  my_config_key = "somestring"
+}
+...
+```
+
+The variable defined in `kube.tf` can be used in any `.yaml.tpl` manifest.
+
+#### **`configmap.tf`**
+
+```
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: demo-config
+  data:
+    someConfigKey: ${sealed_secrets_crt}
+```
+
+For a full example, you can refer to the [simple-resources](simple-resources/) demo.
+
+### Deploying a Helm Chart
+
+If you want to deploy a Helm chart to your cluster, you can use the [Helm Chart controller](https://docs.k3s.io/helm) included in K3s. The Helm Chart controller provides the CRDs `HelmChart` and `HelmChartConfig`.
+
+For a full demo see the [helm-chart](helm-chart/) example.
+
+### Multiple Namespaces
+
+In more complex use cases, you may want to deploy to multiple namespaces with a common base. Kustomize supports this behavior, and it can be since Kube-Hetzner is considering all subdirectories of `extra-manifests`.
+
+For a full demo see the [multiple-namespaces](multiple-namespaces/) example.
+
+## Debugging
+
+To check the existing kustomization, you can run the following command:
+
+```
+$ terraform state list | grep kustom
+  ...
+  module.kube-hetzner.null_resource.kustomization
+  module.kube-hetzner.null_resource.kustomization_user["demo-config-map.yaml.tpl"]
+  module.kube-hetzner.null_resource.kustomization_user["demo-pod.yaml.tpl"]
+  module.kube-hetzner.null_resource.kustomization_user["kustomization.yaml.tpl"]
+  ...
+```
+
+If you want to rerun just the kustomization part, you can use the following command:
+
+```
+terraform apply -replace='module.kube-hetzner.null_resource.kustomization_user["kustomization.yaml.tpl"]' --auto-approve
+```

examples/kustomization_user_deploy/extra-manifests/cert-manager-webhook-inwx.yaml.tpl

@@ -0,0 +1,12 @@
+apiVersion: helm.cattle.io/v1
+kind: HelmChart
+metadata:
+  name: argocd
+  namespace: argocd
+spec:
+  repo: https://argoproj.github.io/argo-helm
+  chart: argo-cd
+  targetNamespace: argocd
+  valuesContent: |-
+    global:
+      domain: argocd.example.com

examples/kustomization_user_deploy/extra-manifests/some-random-name.yaml.tpl

@@ -0,0 +1,6 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - namespace.yaml
+  - helm-chart.yaml

examples/kustomization_user_deploy/helm-chart/helm-chart.yaml.tpl

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: argocd

examples/kustomization_user_deploy/helm-chart/kustomize.yaml.tpl

@@ -0,0 +1,10 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: myapp-pod
+  labels:
+    app: myapp
+spec:
+  containers:
+    - name: nginx
+      image: nginx:1.7.9

examples/kustomization_user_deploy/helm-chart/namespace.yaml.tpl

@@ -2,5 +2,5 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 
 resources:
-  - some-random-name.yaml
-  - cert-manager-webhook-inwx.yaml
+  - namespace-a
+  - namespace-b

examples/kustomization_user_deploy/mutliple-namespaces/base/kustomization.yaml.tpl

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - namespace.yaml
+  - ../base
+namespace: namespace-a

examples/kustomization_user_deploy/mutliple-namespaces/base/pod.yaml.tpl

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: namespace-a

examples/kustomization_user_deploy/extra-manifests/kustomization.yaml.tpl renamed to examples/kustomization_user_deploy/mutliple-namespaces/kustomization.yaml.tpl

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - namespace.yaml
+  - ../base
+namespace: namespace-b

examples/kustomization_user_deploy/mutliple-namespaces/namespace-a/kustomization.yaml.tpl

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: namespace-b

examples/kustomization_user_deploy/mutliple-namespaces/namespace-a/namespace-a.yaml.tpl

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: demo-config
+data:
+  someConfigKey: ${sealed_secrets_crt}