configs/openshift/installer: Add jobs for Azure testing #3877

abhinavdahiya · 2019-05-23T18:46:07Z

ci-operator: add cluster-profile-azure4 secrets

azure4 cluster-profile will be used for OpenShift 4 Azure testing.

templates/openshift/installer: Add azure cluster test configuration

Azure uses a separate base-domain because OpenshShift 4 Azure clusters need a DNS Zone in the same subscription as the cluster.
Therefore, the base domain logic is moved into the setup container where base domain is chosen per platform

The CLUSTER_TYPE for Azure tests is azure4 to differentiate from pre-existing profile for other testing. PRs for ci-operator 1 and ci-oerpator-prowgen 2 should solidify the choice.

Also updates the test container to run Azure e2e and conformance tests.

WIP configs/openshift/installer: Add jobs for Azure testing

Requires openshift/ci-operator-prowgen#169 to allow auto generated jobs.

 docker run -it -v $(pwd)/ci-operator:/ci-operator:z registry.svc.ci.openshift.org/ci/ci-operator-prowgen:latest --from-dir /ci-operator/config/ --to-dir /ci-operator/jobs
....
ERRO[0000] Failed to load CI Operator configuration      error="invalid ci-operator config: invalid configuration: \"tests[6]\": invalid cluster profile \"azure4\"" source-file=/ci-operator/config/openshift/installer/openshift-installer-master.yaml
FATA[0000] Failed to generate jobs                       error="invalid ci-operator config: invalid configuration: \"tests[6]\": invalid cluster profile \"azure4\"" source-dir=/ci-operator/config/ target-dir=/ci-operator/jobs

stevekuznetsov · 2019-05-23T20:23:03Z

ci-operator/SECRETS.md

+
+|       Key                         | Description |
+| ----------------------------------| ----------- |
+| `osServicePrincipal.json`     | Credentials for the Azure API. This is a json file that contains fields described in [upstream credentials doc](https://docs.microsoft.com/en-us/azure-stack/operator/azure-stack-create-service-principals#create-a-service-principal-using-a-client-secret). |


@coverprice this would be the secret we need in bitwarden

stevekuznetsov · 2019-05-24T00:19:57Z

/retest

abhinavdahiya · 2019-05-29T20:21:56Z

/test pj-rehearse

wking · 2019-05-29T20:46:36Z

ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml

@@ -415,7 +422,7 @@ objects:
        if [[ "${CLUSTER_TYPE}" == "aws" ]]; then
            cat > /tmp/artifacts/installer/install-config.yaml << EOF
        apiVersion: v1beta4
-        baseDomain: ${BASE_DOMAIN}
+        baseDomain: origin-ci-int-aws.dev.rhcloud.com


This might break folks who want to run soak tests in a different CI account or some such? Can we leave the template-scoped parameter, default it to empty, use this fallback value for AWS, and error if it's non-empty on Azure?

This might break folks who want to run soak tests in a different CI account or some such?

do we have docs on how to do that? So that i can take a look at how its done.

wking · 2019-05-29T20:52:40Z

ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml

@@ -452,6 +459,35 @@ objects:
        sshKey: |
          ${SSH_PUB_KEY}
        EOF
+        elif [[ "${CLUSTER_TYPE}" == "azure4" ]]; then
+            cat > /tmp/artifacts/installer/install-config.yaml << EOF
+        apiVersion: v1beta4


v1beta4 -> v1

wking · 2019-05-29T21:22:23Z

ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml

+          replicas: 3
+        compute:
+        - name: worker
+          replicas: 3


I don't see a need to set controlPlane or compute unless we're overriding defaults. Same for networking below (I've filed #3914 to drop unnecessary networking entries from master).

azure4 cluster-profile will be used for OpenShift 4 Azure testing.

Azure uses a separate base-domain because OpenshShift 4 Azure clusters need a DNS Zone in the same subscription as the cluster. Therefore, the base domain logic is moved into the setup container where base domain is chosen per platform The CLUSTER_TYPE for Azure tests is `azure4` to differentiate from pre-existing profile for other testing. PRs for ci-operator [1] and ci-operator-prowgen [2] should solidify the choice. Also updates the test container to run Azure e2e and conformance tests. [1]: openshift/ci-operator#352 [2]: openshift/ci-operator-prowgen#169

```console $ ci-operator-prowgen --from-dir ci-operator/config/ --to-dir ci-operator/jobs ``` Also jobs/openshift/installer: make e2e-azure optional and manual trigger only

abhinavdahiya · 2019-05-30T16:51:24Z

updated PR (a62511c...72021dc) to address comments and get changes from #3914

@wking PTAL

abhinavdahiya · 2019-05-30T17:58:24Z

/test pj-rehearse

abhinavdahiya · 2019-06-04T14:06:49Z

/test pj-rehearse

stevekuznetsov · 2019-06-04T15:11:59Z

/lgtm

smarterclayton · 2019-06-04T15:12:15Z

Makefile

@@ -315,6 +315,15 @@ azure-secrets:
 	oc create secret generic codecov-token --from-literal=upload=${CODECOV_UPLOAD_TOKEN} -o yaml --dry-run | oc apply -n azure -f -
 .PHONY: azure-secrets

+azure4-secrets:


Please call this azure-e2e-secrets for now (azure4 is confusing). @jim-minter since azure is camping on the convention that the rest of the e2e suites run, I would prefer if you guys can over time migrate from azure-secrets to azure-aro-secrets or similar. The base namespace belongs to the core CI platform.

openshift-ci-robot · 2019-06-04T15:13:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, stevekuznetsov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [stevekuznetsov]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2019-06-04T15:13:52Z

/retest

Please review the full test history for this PR and help us cut down flakes.

smarterclayton · 2019-06-04T15:14:46Z

Makefile

@@ -315,6 +315,15 @@ azure-secrets:
 	oc create secret generic codecov-token --from-literal=upload=${CODECOV_UPLOAD_TOKEN} -o yaml --dry-run | oc apply -n azure -f -
 .PHONY: azure-secrets

+azure4-secrets:
+	oc create secret generic cluster-secrets-azure4 \
+	--from-file=cluster/test-deploy/azure4/osServicePrincipal.json \


This actually shouldn't be used, can you remove this section? ci-operator/populate-secrets-from-bitwarden is used instead.

smarterclayton · 2019-06-04T15:15:46Z

ci-operator/templates/openshift/installer/cluster-launch-installer-src.yaml

@@ -225,8 +236,8 @@ objects:

        if [[ "${CLUSTER_TYPE}" == "aws" ]]; then
            cat > /tmp/artifacts/installer/install-config.yaml << EOF
-        apiVersion: v1beta4
-        baseDomain: ${BASE_DOMAIN}
+        apiVersion: v1


Is v1 supported back to rc5? Note from now on you can't bump this ever because 4.1 clusters have to be able to install using this template (templates have to work for all 4.y GA versions).

We GA'd the v1 API before rc4 openshift/installer#1589 (April 14)

openshift-ci-robot · 2019-06-04T15:22:13Z

@abhinavdahiya: Updated the following 8 configmaps:

secret-mirroring configmap in namespace ci using the following files:
- key mapping.yaml using file cluster/ci/config/secret-mirroring/mapping.yaml
ci-operator-master-configs configmap in namespace ci using the following files:
- key openshift-installer-master.yaml using file ci-operator/config/openshift/installer/openshift-installer-master.yaml
ci-operator-master-configs configmap in namespace ci-stg using the following files:
- key openshift-installer-master.yaml using file ci-operator/config/openshift/installer/openshift-installer-master.yaml
job-config-master configmap in namespace ci using the following files:
- key openshift-installer-master-presubmits.yaml using file ci-operator/jobs/openshift/installer/openshift-installer-master-presubmits.yaml
prow-job-cluster-launch-installer-e2e configmap in namespace ci using the following files:
- key cluster-launch-installer-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml
prow-job-cluster-launch-installer-e2e configmap in namespace ci-stg using the following files:
- key cluster-launch-installer-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml
prow-job-cluster-launch-installer-src configmap in namespace ci using the following files:
- key cluster-launch-installer-src.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-src.yaml
prow-job-cluster-launch-installer-src configmap in namespace ci-stg using the following files:
- key cluster-launch-installer-src.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-src.yaml

In response to this:

ci-operator: add cluster-profile-azure4 secrets

azure4 cluster-profile will be used for OpenShift 4 Azure testing.

templates/openshift/installer: Add azure cluster test configuration

Azure uses a separate base-domain because OpenshShift 4 Azure clusters need a DNS Zone in the same subscription as the cluster.
Therefore, the base domain logic is moved into the setup container where base domain is chosen per platform

The CLUSTER_TYPE for Azure tests is azure4 to differentiate from pre-existing profile for other testing. PRs for ci-operator 1 and ci-oerpator-prowgen 2 should solidify the choice.

Also updates the test container to run Azure e2e and conformance tests.

WIP configs/openshift/installer: Add jobs for Azure testing

Requires openshift/ci-operator-prowgen#169 to allow auto generated jobs.
docker run -it -v $(pwd)/ci-operator:/ci-operator:z registry.svc.ci.openshift.org/ci/ci-operator-prowgen:latest --from-dir /ci-operator/config/ --to-dir /ci-operator/jobs
....
ERRO[0000] Failed to load CI Operator configuration      error="invalid ci-operator config: invalid configuration: \"tests[6]\": invalid cluster profile \"azure4\"" source-file=/ci-operator/config/openshift/installer/openshift-installer-master.yaml
FATA[0000] Failed to generate jobs                       error="invalid ci-operator config: invalid configuration: \"tests[6]\": invalid cluster profile \"azure4\"" source-dir=/ci-operator/config/ target-dir=/ci-operator/jobs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2019-06-04T15:31:11Z

@abhinavdahiya: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/rehearse/fabric8-services/toolchain-operator/master/e2e	`868e0fb`	link	`/test pj-rehearse`
ci/rehearse/openshift/installer/master/e2e-azure	`868e0fb`	link	`/test pj-rehearse`
ci/prow/pj-rehearse	`868e0fb`	link	`/test pj-rehearse`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

smarterclayton · 2019-06-04T15:43:01Z

ci-operator/templates/openshift/installer/cluster-launch-installer-src.yaml

@@ -255,10 +266,31 @@ objects:
        sshKey: |
          ${SSH_PUB_KEY}
        EOF
+        elif [[ "${CLUSTER_TYPE}" == "azure4" ]]; then


I would prefer cluster type to be azure - you can still mount the other secrets to the azure name even if the secret name is different.

smarterclayton · 2019-06-04T15:43:27Z

ci-operator/templates/openshift/installer/cluster-launch-installer-src.yaml

+          replicas: 3
+        platform:
+          azure:
+            baseDomainResourceGroupName: os4-common


Is this already a set name? Something that is part of the account?

installer currently picks the first DNS Zone from the account based on the base domain, while most cloud providers allow multiple DNS zones with the same name.

We have https://jira.coreos.com/browse/CORS-1070 already in pipeline to allow users to choose the exact Zone (without the guessing). But Microsoft people were far too un-comfortable with leaving the guessing for Azure addition and insisted we add this field to Azure platform while we were frozen for AWS.

Something that is part of the account?

Yes this resource group is present in the CI subscription.

smarterclayton · 2019-06-04T15:43:56Z

ci-operator/templates/openshift/installer/cluster-launch-installer-src.yaml

@@ -296,6 +328,14 @@ objects:
        value: ${NAMESPACE}-${JOB_NAME_HASH}
      - name: TYPE
        value: ${CLUSTER_TYPE}
+      - name: AWS_SHARED_CREDENTIALS_FILE


Was this file inconsistent with the others?

All other containers use the ENV to setup the creds. expect teardown, so this change makes it uniform.

wking · 2019-06-04T17:29:44Z

ci-operator/templates/openshift/installer/cluster-launch-installer-src.yaml

+          # TODO: make openshift-tests auto-discover this from cluster config
+          export TEST_PROVIDER='{"type":"azure","region":"centralus","multizone":true,"multimaster":true}'
+          export KUBE_SSH_USER=core
+        fi


@spadgett is fixing this if-before-an-elif in #3954.

… Drop *_REGION from teardown These came in with d00da5d (templates/openshift/installer: Add azure cluster test configuration, 2019-05-23, openshift#3877), but the installer should be extracting the region from metadata.json [1,2]. [1]: https://github.com/openshift/installer/blob/d9a9648cf2330d467cca9f2988846d031464125e/pkg/types/aws/metadata.go#L5 [2]: https://github.com/openshift/installer/blob/d9a9648cf2330d467cca9f2988846d031464125e/pkg/types/azure/metadata.go#L5

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 23, 2019

openshift-ci-robot requested review from aaronlevy and smarterclayton May 23, 2019 18:46

stevekuznetsov reviewed May 23, 2019

View reviewed changes

abhinavdahiya force-pushed the azure_test branch 2 times, most recently from 4ab0a9a to 1417ae8 Compare May 23, 2019 21:10

abhinavdahiya force-pushed the azure_test branch 3 times, most recently from 0381c70 to a62511c Compare May 24, 2019 18:13

wking reviewed May 29, 2019

View reviewed changes

abhinavdahiya added 2 commits May 30, 2019 09:46

ci-operator: add cluster-profile-azure4 secrets

4d630ab

azure4 cluster-profile will be used for OpenShift 4 Azure testing.

abhinavdahiya force-pushed the azure_test branch from a62511c to 72021dc Compare May 30, 2019 16:46

configs/openshift/installer: Add jobs for Azure testing

868e0fb

```console $ ci-operator-prowgen --from-dir ci-operator/config/ --to-dir ci-operator/jobs ``` Also jobs/openshift/installer: make e2e-azure optional and manual trigger only

abhinavdahiya force-pushed the azure_test branch from 72021dc to 868e0fb Compare May 30, 2019 16:50

openshift-ci-robot assigned stevekuznetsov Jun 4, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 4, 2019

smarterclayton reviewed Jun 4, 2019

View reviewed changes

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 4, 2019

smarterclayton reviewed Jun 4, 2019

View reviewed changes

openshift-merge-robot merged commit fe1d3f3 into openshift:master Jun 4, 2019

smarterclayton reviewed Jun 4, 2019

View reviewed changes

abhinavdahiya deleted the azure_test branch June 4, 2019 16:09

wking reviewed Jun 4, 2019

View reviewed changes

wking mentioned this pull request Sep 18, 2019

ci-operator/templates/openshift/installer/cluster-launch-installer-*: Drop *_REGION from teardown #5079

Merged

configs/openshift/installer: Add jobs for Azure testing #3877

configs/openshift/installer: Add jobs for Azure testing #3877

Uh oh!

Conversation

abhinavdahiya commented May 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevekuznetsov commented May 24, 2019

Uh oh!

abhinavdahiya commented May 29, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhinavdahiya commented May 30, 2019

Uh oh!

abhinavdahiya commented May 30, 2019

Uh oh!

abhinavdahiya commented Jun 4, 2019

Uh oh!

stevekuznetsov commented Jun 4, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Jun 4, 2019

Uh oh!

openshift-bot commented Jun 4, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Jun 4, 2019

Uh oh!

openshift-ci-robot commented Jun 4, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

abhinavdahiya commented May 23, 2019 •

edited

Loading