Preprocess ExtSec template with templating engine before upserting K8S secret

[eshepelyuk]

Context

Started as this idea and continued in Slack

This issue is a companion to a corresponding PR that is going to be provided.

Problem that is addressed

• Application may need more complex Secret keys, like yaml files, for example, that could be constructed from several secrets in secret backend.
• Also, some backend ( like Vault) may contain complex data - as nested JSON, while resulting K8S Secret should contain only several fields from complex data.
• Currently it's only possible to retrieve either a single field from JSON-like secret, or the whole secret is inserted as json string.

Introducing template engine allows users of this operator to preprocess complex remote secret structures and generate arbitrary flexible values for K8S Secret keys.

Design

Currenlty, the operator does following

1. Retrieves sensible data from secret backend ( like Vault ) using ExtSec.data as a source.
2. Generates yaml manifest to be upserted into existing K8S Secret object.
3. Merges resulting yaml with yaml from ExtSec.template
4. Eventually upserts K8S Secret.

The change happens before 3. yaml from ExtSec.template is preprocessed with template engine before merging.
Confidential data retrieved from secret backend is accessible via data map in template context.

lodash.template is used as templating engine:

• allows to minimize dependencies since lodash is already used in project
• extremely flexible, since allows using pure JavaScript for producing output

Additionally, yaml object will be provided to template context to parse \ dump from \ to Yaml. The object is a instance of js-yaml that is already utilized by the operator.

Example

Let's have 2 secrets in Vault with complex JSON structure.

Having following ExternalSecret

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: tmpl-ext-sec
spec:
  backendType: vault
  data:
  - key: kv/data/eshepelyuk/secret1
    name: s1
  - key: kv/data/eshepelyuk/secret2
    name: s2
  kvVersion: 2
  template:
    data:
      file.txt: |
        <%= Buffer.from(JSON.stringify(JSON.parse(data.s1).objKey)).toString("base64") %>
    metadata:
      labels:
        label1: <%= JSON.parse(data.s1).intKey %>
        label2: <%= JSON.parse(data.s1).objKey.strKey.replace(" ", "-") %>
    stringData:
      file.yaml: |
        <%= yaml.dump(JSON.parse(data.s1)) %>
        <% let s2 = JSON.parse(data.s2) %><% s2.arrKey.forEach((e, i) => { %>arr_<%= i %>: <%= e %>
        <% }) %>`
  vaultMountPoint: kubernetes
  vaultRole: demo

this operator will generate following Secret

apiVersion: v1
data:
  file.txt: eyJzdHJLZXkiOiJoZWxsbyB3b3JsZCJ9
  file.yaml: aW50S2V5OiAxMQpvYmpLZXk6CiAgc3RyS2V5OiBoZWxsbyB3b3JsZAoKYXJyXzA6IDEKYXJyXzE6IDIKYXJyXzI6IDMKYAo=
  s1: eyJpbnRLZXkiOjExLCJvYmpLZXkiOnsic3RyS2V5IjoiaGVsbG8gd29ybGQifX0=
  s2: eyJhcnJLZXkiOlsxLDIsM119
kind: Secret
metadata:
  name: tmpl-ext-sec
type: Opaque

That could be inspected to see that result is generated by lodash templating engine

$ kubectl get secret/tmpl-ext-sec -ogo-template='{{ index .data "s1" | base64decode }}'
{"intKey":11,"objKey":{"strKey":"hello world"}}

$ kubectl get secret/tmpl-ext-sec -ogo-template='{{ index .data "s2" | base64decode }}'
{"arrKey":[1,2,3]}

$ kubectl get secret/tmpl-ext-sec -ogo-template='{{ index .data "file.txt" | base64decode }}'
{"strKey":"hello world"}

$ kubectl get secret/tmpl-ext-sec -ogo-template='{{ index .data "file.yaml" | base64decode }}'
intKey: 11
objKey:
  strKey: hello world

arr_0: 1
arr_1: 2
arr_2: 3

$ kubectl get secret/tmpl-ext-sec -ogo-template='{{ .metadata.labels }}'
map[label1:11 label2:hello-world]

I.e. templating can be used for

• K8S Secret keys
  • for upserting plaint text via stringData
  • for upserting base64 encoded via data
• For dynamic labeles, annotations and other fields available in K8S Secret

[eshepelyuk]

@Flydiverny @moolen @knelasevero @mcavoyk please take a look at this issue and corresponding PR #626 provided by @intpp

[moolen]

#626 is merged. This will be in the next release (6.4.0 i guess)

[eshepelyuk]

#626 is merged. This will be in the next release (6.4.0 i guess)

Thnx a lot !
Although the most intesting thing is not a number of release but a date.