Azure · aramase · Feb 24, 2021 · Feb 19, 2021
@@ -12,15 +12,14 @@ From the Kubernetes documentation on [Encrypting Secret Data at Rest]:
 
 ⚠️ **NOTE**: Currently, KMS plugin for Key Vault does not support key rotation. If you create a new key version in KMS, decryption will fail since it won't match the key used for encryption when the cluster was created.
 
-💡 **NOTE**: To integrate your application secrets from a key management system outside of Kubernetes, use [Key Vault FlexVolume].
+💡 **NOTE**: To integrate your application secrets from a key management system outside of Kubernetes, use [Azure Key Vault Provider for Secrets Store CSI Driver].
 
 ## Features
 
 * Use a key in Key Vault for etcd encryption
-* Generate keys protected by a Hardware Security Module (HSM)
+* Use a key in Key Vault protected by a Hardware Security Module (HSM)
 * Bring your own keys
 * Store secrets, keys, and certs in etcd, but manage them as part of Kubernetes
-* Restrict access using Kubernetes core concepts: RBAC, Service Accounts, and namespaces
 
 ## Getting Started
 
@@ -42,9 +41,13 @@ For more information about K8s secrets in AKS follow this [doc](https://docs.mic
 
 AKS does encrypt secrets at rest, but keys are managed by the service and users cannot bring their own.
 
+### Setting up KMS Plugin manually
+
+Refer to [doc](docs/manual-install.md) for steps to setup the KMS Key Vault plugin on an existing cluster.
+
 ## Verifying that Data is Encrypted
 
-Now that your cluster has `--experimental-encryption-provider-config` turned on, it will encrypt the data in etcd. Let's verify that is working:
+Now that Azure KMS provider is running in your cluster and the encryption configuration is setup, it will encrypt the data in etcd. Let's verify that is working:
 
 1. Create a new secret:
 
@@ -76,10 +79,13 @@ The KMS Plugin for Key Vault project welcomes contributions and suggestions. Ple
 
 This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information, see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq) or contact [[email protected]](mailto:[email protected]) with any additional questions or comments.
 
+## Support
+
+KMS Plugin for Key Vault is an open source project that is [**not** covered by the Microsoft Azure support policy](https://support.microsoft.com/en-us/help/2941892/support-for-linux-and-open-source-technology-in-azure). [Please search open issues here](https://github.com/Azure/kubernetes-kms/issues), and if your issue isn't already represented please [open a new one](https://github.com/Azure/kubernetes-kms/issues/new/choose). The project maintainers will respond to the best of their abilities.
 
 [AKS]: https://azure.microsoft.com/services/kubernetes-service/
 [AKS Engine]: https://github.com/Azure/aks-engine
 [Azure Key Vault Data Encryption]: https://github.com/Azure/aks-engine/blob/master/docs/topics/features.md#azure-key-vault-data-encryption
 [Encrypting Secret Data at Rest]: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#providers
 [example cluster configuration]: https://github.com/Azure/aks-engine/blob/master/examples/kubernetes-config/kubernetes-keyvault-encryption.json
-[Key Vault FlexVolume]: https://github.com/Azure/kubernetes-keyvault-flexvol
+[Azure Key Vault Provider for Secrets Store CSI Driver]: https://github.com/Azure/secrets-store-csi-driver-provider-azure
@@ -0,0 +1,175 @@
+# 🛠 Manual Configurations #
+
+This guide demonstrates steps required to enable the KMS Plugin for Key Vault in an existing cluster.
+
+### 1. Create a Keyvault
+
+  If you're bringing your own keys, skip this step.
+
+  ```bash
+  KEYVAULT_NAME=k8skv
+  RG=mykubernetesrg
+  LOC=eastus
+
+  # create resource group that'll contain the keyvault instance
+  az group create -n $RG -l $LOC
+  # create keyvault
+  az keyvault create -n $KV_NAME -g $RG
+  # create key that will be used for encryption
+  az keyvault key create -n k8s --vault-name $KV_NAME --kty RSA --size 2048
+  ```
+
+### 2. Give the cluster identity permissions to access the keys in keyvault
+
+  The KMS Plugin uses the cluster service principal or managed identity to access the keyvault instance.
+
+  #### More on authentication methods
+
+  [`/etc/kubernetes/azure.json`](https://kubernetes-sigs.github.io/cloud-provider-azure/install/configs/) is a well-known JSON file in each node that provides the details about which method KMS Plugin uses for access to Keyvault:
+
+  | Authentication method            | `/etc/kubernetes/azure.json` fields used                                                    |
+  | -------------------------------- | ------------------------------------------------------------------------------------------- |
+  | System-assigned managed identity | `useManagedIdentityExtension: true` and `userAssignedIdentityID:""`                         |
+  | User-assigned managed identity   | `useManagedIdentityExtension: true` and `userAssignedIdentityID:"<UserAssignedIdentityID>"` |
+  | Service principal (default)      | `aadClientID: "<AADClientID>"` and `aadClientSecret: "<AADClientSecret>"`                   |
+
+  #### Obtaining the ID of the cluster managed identity/service principal
+
+  After your cluster is provisioned, depending on your cluster identity configuration, run one of the following commands to retrieve the **ID** of your managed identity or service principal, which will be used for role assignment to access Keyvault:
+
+  | Cluster configuration                            | Command                                                                                                                                                                       |
+  | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+  | AKS cluster with service principal               | `az aks show -g <AKSResourceGroup> -n <AKSClusterName> --query servicePrincipalProfile.clientId -otsv`                                                                        |
+  | AKS cluster with managed identity                | `az aks show -g <AKSResourceGroup> -n <AKSClusterName> --query identityProfile.kubeletidentity.clientId -otsv`                                                                |
+  | aks-engine cluster with service principal        | Use the client ID of the service principal defined in the API model                                                                                                           |
+  | aks-engine cluster with system-assigned identity | `az <vm\|vmss> identity show -g <NodeResourceGroup> -n <VM\|VMSS Name> --query principalId -otsv`                                                                             |
+  | aks-engine cluster with user-assigned identity   | `az <vm\|vmss> identity show -g <NodeResourceGroup> -n <VM\|VMSS Name> --query userAssignedIdentities -otsv`, then copy the `clientID` of the selected user-assigned identity |
+
+  Assign the following permissions:
+
+  ```bash
+  az keyvault set-policy -n $KEYVAULT_NAME --key-permissions decrypt encrypt --spn <YOUR SPN CLIENT ID>
+  ```
+
+### 3. Deploy the KMS Plugin
+
+  For all Kubernetes master nodes, add the static pod manifest to `/etc/kubernetes/manifests`
+
+  ```yaml
+  apiVersion: v1
+  kind: Pod
+  metadata:
+    name: azure-kms-provider
+    namespace: kube-system
+    labels:
+      tier: control-plane
+      component: azure-kms-provider
+  spec:
+    priorityClassName: system-node-critical
+    hostNetwork: true
+    containers:
+      - name: azure-kms-provider
+        image: mcr.microsoft.com/oss/azure/kms/keyvault:v0.0.11
+        imagePullPolicy: IfNotPresent
+        args:
+          - --listen-addr=unix:///opt/azurekms.socket             # [OPTIONAL] gRPC listen address. Default is unix:///opt/azurekms.socket
+          - --keyvault-name=${KV_NAME}                            # [REQUIRED] Name of the keyvault
+          - --key-name=${KEY_NAME}                                # [REQUIRED] Name of the keyvault key used for encrypt/decrypt
+          - --key-version=${KEY_VERSION}                          # [REQUIRED] Version of the key to use
+          - --log-format-json=false                               # [OPTIONAL] Set log formatter to json. Default is false.
+          - --healthz-port=8787                                   # [OPTIONAL] port for health check. Default is 8787
+          - --healthz-path=8787                                   # [OPTIONAL] path for health check. Default is /healthz
+          - --healthz-timeout=20s                                 # [OPTIONAL] RPC timeout for health check. Default is 20s
+          - -v=1
+        ports:
+          - containerPort: 8787                                   # Must match the value defined in --healthz-port
+            protocol: TCP
+        livenessProbe:
+          httpGet:
+            path: /healthz                                        # Must match the value defined in --healthz-path
+            port: 8787                                            # Must match the value defined in --healthz-port
+          failureThreshold: 2
+          periodSeconds: 10
+        resources:
+          requests:
+            cpu: 100m
+            memory: 128Mi
+          limits:
+            cpu: 4
+            memory: 2Gi
+        volumeMounts:
+          - name: etc-kubernetes
+            mountPath: /etc/kubernetes
+          - name: etc-ssl
+            mountPath: /etc/ssl
+            readOnly: true
+          - name: sock
+            mountPath: /opt
+    volumes:
+      - name: etc-kubernetes
+        hostPath:
+          path: /etc/kubernetes
+      - name: etc-ssl
+        hostPath:
+          path: /etc/ssl
+      - name: sock
+        hostPath:
+          path: /opt
+  ```
+
+  View logs from the kms pod:
+
+  ```bash
+  kubectl logs -l component=azure-kms-provider -n kube-system
+
+  I0219 17:35:33.608840       1 main.go:60] "Starting KeyManagementServiceServer service" version="v0.0.11" buildDate="2021-02-19-17:33"
+  I0219 17:35:33.609090       1 azure_config.go:27] populating AzureConfig from /etc/kubernetes/azure.json
+  I0219 17:35:33.609420       1 auth.go:66] "azure: using client_id+client_secret to retrieve access token" clientID="9a7a##### REDACTED #####bb26" clientSecret="23T.##### REDACTED #####vw-r"
+  I0219 17:35:33.609568       1 keyvault.go:66] "using kms key for encrypt/decrypt" vaultName="k8skmskv" keyName="key1" keyVersion="5cdf48ea6bb9456ebf637e1130b7751a"
+  I0219 17:35:33.609897       1 main.go:86] Listening for connections on address: /opt/azurekms.socket
+  ...
+  ```
+
+### 4. Create encryption configuration
+
+  Create a new encryption configuration file `/etc/kubernetes/manifests/encryptionconfig.yaml` using the appropriate properties for the `kms` provider:
+
+  ```yaml
+  kind: EncryptionConfiguration
+  apiVersion: apiserver.config.k8s.io/v1
+  resources:
+    - resources:                                        # List of kubernetes resources that will be encrypted in etcd using the KMS plugin
+        - secrets
+      providers:
+        - kms:
+            name: azurekmsprovider
+            endpoint: unix:///opt/azurekms.socket       # This endpoint must match the value defined in --listen-addr for the KMS plugin
+            cachesize: 1000
+        - identity: {}
+  ```
+
+  The encryption configuration file needs to be accessible by all the api servers.
+
+### 5. Modify `/etc/kubernetes/kube-apiserver.yaml`
+
+  Add the following flag:
+
+  ```yaml
+  --encryption-provider-config=/etc/kubernetes/encryptionconfig.yaml
+  ```
+
+  Mount `/opt` to access the socket:
+
+  ```yaml
+  ...
+  volumeMounts:
+  - name: "sock"
+    mountPath: "/opt"
+  ...
+  volumes:
+    - name: "sock"
+      hostPath:
+        path: "/opt"
+  ```
+
+### 6. Restart your API server