Unable to set a proxy for alertmanager-receivers

[breadly7]

Describe the bug
When running cortex in a corporate environment, where all traffic to the internet must go through a certain proxy, configuring an alertmanager receiver (e.g. OpsGenie) with proxy settings seems currently not possible. 

Setting the proxy in the http_config parts from the receivers was disabled / is not possible due to #4129.

In addition, the alertmanager is not designed in a way, where configuring the proxy for receivers through the environment is possible. See this comment here: prometheus/alertmanager#853 (comment) 

We have tried to set HTTP_PROXY, HTTPS_PROXY, but the alertmanager ignored this for the receivers.

 Is there a way to set this proxy that I missed or is this simply not possible at the moment?

To Reproduce
Setting the proxy_url through the alertmanager-api is not possible:

1. Start Cortex (v1.11.0)
2. Perform a Post-Request to alertmanager (/api/v1/alerts) with a configured http_config part.

receivers:
  - name: opsgenie_receiver
    opsgenie_configs:
      - api_url: "https://opsgenie-url"
        api_key: api_key
        message: "OpsGenie Alert from Cortex alertmanager"
        description: "Something happened"

route:
  group_wait: 10s
  group_interval: 10s
  receiver: opsgenie_receiver
  routes:
    - receiver: opsgenie_receiver
      match:
        type: cortex-alert
      continue: true
      http_config:
        proxy_url: "http://my-proxy:1234/"

3. Observe an error about not being allowed to set the http_config.

Proxy environment variables are ignored

1. Start Cortex (v1.11.0)
2. Edit the pods from alertmanager and add the following

HTTPS_PROXY: http://my-proxy:1234
HTTP_PROXY: http://my-proxy:1234
https_proxy: http://my-proxy:1234

http_proxy: http://my-proxy:1234

3. Add any alert rule and receiver config to ruler/alertmanager
4. Observe in the alertmanager logs, that the proxy-variables are ignored

Expected behavior
I can configure a proxy for the alertmanager receivers, like the “regular” alertmanager is able to.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm
• Cortex-Version: v1.11.0


Storage Engine

• Blocks
• Chunks

Additional Context
Nothing to add here

[lulukas86]

+1 We're also missing that feature.