env.AWS_REGION='us-west-2' causes failure

[kwalsh413]

when performing the helm install, add --set env.AWS_REGION=‘us-west-2’ to the command. Apply an external secret and query your ES. You see an error like ERROR, Invalid character in header content ["Authorization"] . I did not encounter the same issue when using the us-east-1 region. Of course us-west-2 is the default so this is not a major issue, however it did lead to confusion amongst our team.

[Flydiverny]

Could you provide some insights into what kind of external secret you applied? Preferably the full manifest (feel free to edit out key names or so), but at least which backend type was used to see if this could be related :)

[kwalsh413]

The backend is Secrets Manager. A simple manifest like below should recreate the issue

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: es-keys
spec:
  backendType: secretsManager
  data:
    - key: es-keys/key-1
       name: key1

[d-michail]

I am seeing the same error when using region 'eu-west-1'. The relevant logs are

{"level":50,
"message":"Invalid character in header content [\"Authorization\"]","code":"NetworkingError",
"region":"eu-west-1",
"hostname":"secretsmanager.eu-west-1.amazonaws.com","retryable":true,
"msg":"failure while polling the secret foo/testsecret",
"stack":"TypeError [ERR_INVALID_CHAR]: Invalid character in header content [\"Authorization\"]\n    at ClientRequest.setHeader (_http_outgoing.js:488:3)\n    at new ClientRequest (_http_client.js:221:14)\n    at Object.request (https.js:309:10)\n    at features.constructor.handleRequest (/app/node_modules/aws-sdk/lib/http/node.js:45:23)\n    at executeSend (/app/node_modules/aws-sdk/lib/event_listeners.js:342:29)\n    at Request.SEND (/app/node_modules/aws-sdk/lib/event_listeners.js:356:9)\n    at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:102:18)\n    at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n    at Request.emit (/app/node_modules/aws-sdk/lib/request.js:683:14)\n    at Request.transition (/app/node_modules/aws-sdk/lib/request.js:22:10)","type":"Error","v":1}

The secret looks like

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: testsecret
  namespace: foo
spec:
  backendType: secretsManager
  dataFrom:
    - foo

Authorization is performed using AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with a user that has the necessary permissions (tested using the aws cli).

[d-michail]

I found the problem.

When the SDK is initialized without a specific region (unlike in the tests or using localstack) the environmental variable which has the region should be called AWS_DEFAULT_REGION and not AWS_REGION. See config-vars documentation for more details.

Thus, the helm chart needs to be updated. I used

  env:
  - name: "AWS_REGION"
    value: "eu-west-1"
  - name: "AWS_DEFAULT_REGION"
    value: "eu-west-1"

and it works. Hope this helps someone.

[ecout]

How come it has been 25 days and no one has updated the chart?
Edit, I just joined the project, still deciding whether to use or not based on bugs and based on whether or not key rotation works or not. As of May 3 2019, it didn't according to this presentation.
That is my main goal.
https://www.youtube.com/watch?v=BhSzc7-0RQY

[rotemjac]

I still receive this error.