introduce the var to delete the kube-proxy so we can test the cilium … (#26)

* introduce the var to delete the kube-proxy so we can test the cilium kpr properly

Author: liyihuang

README.md

@@ -3,30 +3,30 @@
 An opinionated Terraform module that can be used to install and manage Cilium on top of a Kubernetes cluster.
 
 <!-- BEGIN_TF_DOCS -->
-## Requirements
+### Requirements
 
 | Name | Version |
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.2.0 |
 | <a name="requirement_null"></a> [null](#requirement\_null) | >= 3.1.1 |
 
-## Providers
+### Providers
 
 | Name | Version |
 |------|---------|
 | <a name="provider_null"></a> [null](#provider\_null) | >= 3.1.1 |
 
-## Modules
+### Modules
 
 No modules.
 
-## Resources
+### Resources
 
 | Name | Type |
 |------|------|
 | [null_resource.main](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource |
 
-## Inputs
+### Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
@@ -39,17 +39,19 @@ No modules.
 | <a name="input_cilium_namespace"></a> [cilium\_namespace](#input\_cilium\_namespace) | The namespace in which to install Cilium. | `string` | `"kube-system"` | no |
 | <a name="input_control_plane_nodes_label_selector"></a> [control\_plane\_nodes\_label\_selector](#input\_control\_plane\_nodes\_label\_selector) | The label selector used to filter control-plane nodes. | `string` | `"node-role.kubernetes.io/control-plane"` | no |
 | <a name="input_deploy_etcd_cluster"></a> [deploy\_etcd\_cluster](#input\_deploy\_etcd\_cluster) | Whether to deploy an 'etcd' cluster suitable for usage as the Cilium key-value store (HIGHLY EXPERIMENTAL). | `bool` | `false` | no |
+| <a name="input_disable_kube_proxy"></a> [disable\_kube\_proxy](#input\_disable\_kube\_proxy) | Whether to disable the kube proxy so the cluster uses kube-proxy replacement | `bool` | `false` | no |
 | <a name="input_extra_provisioner_environment_variables"></a> [extra\_provisioner\_environment\_variables](#input\_extra\_provisioner\_environment\_variables) | A map of extra environment variables to include when executing the provisioning script. | `map(string)` | `{}` | no |
 | <a name="input_install_kube_prometheus_servicemonitor_crd"></a> [install\_kube\_prometheus\_servicemonitor\_crd](#input\_install\_kube\_prometheus\_servicemonitor\_crd) | Whether to install the 'kube-prometheus' ServiceMonitor CRD. | `bool` | `true` | no |
 | <a name="input_ipsec_key"></a> [ipsec\_key](#input\_ipsec\_key) | The IPsec key to use for transparent encryption. Leave empty for none to be created (in which case encryption should be disabled in Helm as well). | `string` | `""` | no |
 | <a name="input_kube_prometheus_crds_version"></a> [kube\_prometheus\_crds\_version](#input\_kube\_prometheus\_crds\_version) | Version of the 'kube-prometheus' ServiceMonitor CRD to install. | `string` | `"v0.13.0"` | no |
+| <a name="input_kube_proxy_namespace"></a> [kube\_proxy\_namespace](#input\_kube\_proxy\_namespace) | Whether to disable the kube proxy so the cluster uses kube-proxy replacement | `string` | `"kube-system"` | no |
 | <a name="input_path_to_kubeconfig_file"></a> [path\_to\_kubeconfig\_file](#input\_path\_to\_kubeconfig\_file) | The path to the kubeconfig file to use. | `string` | n/a | yes |
 | <a name="input_post_cilium_install_script"></a> [post\_cilium\_install\_script](#input\_post\_cilium\_install\_script) | A script to be run right after installing Cilium. | `string` | `""` | no |
 | <a name="input_pre_cilium_install_script"></a> [pre\_cilium\_install\_script](#input\_pre\_cilium\_install\_script) | A script to be run right before installing Cilium. | `string` | `""` | no |
 | <a name="input_total_control_plane_nodes"></a> [total\_control\_plane\_nodes](#input\_total\_control\_plane\_nodes) | The number of control-plane nodes expected in the cluster. | `number` | `3` | no |
 | <a name="input_wait_for_total_control_plane_nodes"></a> [wait\_for\_total\_control\_plane\_nodes](#input\_wait\_for\_total\_control\_plane\_nodes) | Whether to wait for the expected number of control-plane nodes to be registered before applying any changes. | `bool` | `false` | no |
 
-## Outputs
+### Outputs
 
 No outputs.
 <!-- END_TF_DOCS -->

locals.tf

@@ -32,6 +32,8 @@ locals {
     POST_CILIUM_INSTALL_SCRIPT         = var.post_cilium_install_script != "" ? base64encode(var.post_cilium_install_script) : "" // The script to execute after installing Cilium.
     TOTAL_CONTROL_PLANE_NODES          = var.total_control_plane_nodes                                                            // The number of control-plane nodes expected in the cluster.
     WAIT_FOR_TOTAL_CONTROL_PLANE_NODES = var.wait_for_total_control_plane_nodes                                                   // Whether to wait for the expected number of control-plane nodes to be registered before applying any changes.
+    DISABLE_KUBE_PROXY                 = var.disable_kube_proxy                                                                   // Wether to disable the kube proxy after the cilium
+    KUBE_PROXY_NAMESPACE               = var.kube_proxy_namespace                                                                 // the namespace contains the kube-proxy, it should be kube-system most of the case but leave this as the var in case we found some k8s distribution use something else
   }
   provisioner_path = "${abspath(path.module)}/scripts/provisioner.sh"
 }

scripts/provisioner.sh

@@ -133,3 +133,18 @@ if [[ "${POST_CILIUM_INSTALL_SCRIPT}" != "" ]];
 then
   base64 --decode <<< "${POST_CILIUM_INSTALL_SCRIPT}" | bash
 fi
+
+# try to delete the kube-proxy and clear the iptabls using the cilum pods after we install the cilium
+if [[ "${DISABLE_KUBE_PROXY}" == "true" ]]; then
+  kubectl -n "${KUBE_PROXY_NAMESPACE}" delete daemonset kube-proxy || true
+  kubectl -n "${KUBE_PROXY_NAMESPACE}" delete cm kube-proxy || true
+  kubectl wait --for=condition=Ready pod -l k8s-app=cilium -n "${CILIUM_NAMESPACE}" 
+  pods=$(kubectl get pods -l k8s-app=cilium -o name -n "${CILIUM_NAMESPACE}")
+  if [ -n "$pods" ]; then
+      while IFS= read -r pod; do
+          kubectl -n "${CILIUM_NAMESPACE}" exec $pod -- sh -c 'iptables-save | grep -v KUBE | iptables-restore' 
+      done <<< "$pods"
+  else
+      echo "No pods found with label k8s-app=cilium in cilium namespace"
+  fi
+fi

variables.tf

@@ -110,6 +110,19 @@ variable "install_kube_prometheus_servicemonitor_crd" {
   type        = bool
 }
 
+variable "disable_kube_proxy" {
+  default     = false
+  description = "Whether to disable the kube proxy so the cluster uses kube-proxy replacement"
+  type        = bool
+}
+
+variable "kube_proxy_namespace" {
+  default     = "kube-system"
+  description = "Whether to disable the kube proxy so the cluster uses kube-proxy replacement"
+  type        = string
+}
+
+
 variable "kube_prometheus_crds_version" {
   default     = "v0.13.0"
   description = "Version of the 'kube-prometheus' ServiceMonitor CRD to install."