-
Notifications
You must be signed in to change notification settings - Fork 198
OTA-1521: Add a default-deny network policy for CVO namespace #1198
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OTA-1521: Add a default-deny network policy for CVO namespace #1198
Conversation
@petr-muller: This pull request references OTA-1531 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
@petr-muller: This pull request references OTA-1521 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
I was expected none of the e2e would succeed (as CVO would broken up to "Deny ALL") but the situation is much better. |
/cc |
Yeah, checkout the Pragmatic (CVO uses host network) section of the working doc. Network Policies do not affect most host-networked pods traffic. It definitely does not affect the core functionality (being a k8s controller=talking to the apiserver). We will need to test all functionality that uses network communication (see the working doc) to be sure though - it is also possible that we simply do not have tests that exercise the functionality. |
@petr-muller: This pull request references OTA-1499 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "4.20.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Add a baseline NetworkPolicy to deny all network communication (both ingress and egress) to all pods in the namespace. Any necessary network traffic needs to be allowed by an additional NetworkPolicy resource (they are additive). At the moment, the default deny all policy should be the only one needed: - CVO is host-networked so it is [not affected by network policies](https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/networking/network-security#network-policy) - Bare `version` pods spawned by CVO do not require any network communication See [OTA Network Policies Working Document](https://docs.google.com/document/d/1Dzr3eYGVl6OBxqfUohugJLsbsn7sYrC3fN6yCe8zTRQ/edit?tab=t.0#heading=h.9vehq2liufe) for more information.
76b22be
to
07c6b5c
Compare
@petr-muller: This pull request references OTA-1499 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "4.20.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
@petr-muller: This pull request references OTA-1521 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
@petr-muller: This pull request references OTA-1521 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Testing with scripts from https://github.com/petr-muller/vibes/tree/main/scripts shows the default NetworkPolicy correctly prevents pods from communication: $ ./scripts/test-networkpolicy-external-access.fish openshift-cluster-version
Testing NetworkPolicy external access blocking in namespace: openshift-cluster-version
...
✅ TEST PASSED: NetworkPolicy is blocking external access
External access to http://networkpolicy-netcat-route-openshift-cluster-version.apps.ci-ln-vfv2bgt-76ef8.aws-2.ci.openshift.org was blocked (as expected)
$ ./scripts/test-networkpolicy-isolation.fish openshift-cluster-version
Testing NetworkPolicy isolation in namespace: openshift-cluster-version
...
✅ TEST PASSED: NetworkPolicy is blocking outbound traffic
The pod was unable to reach https://www.google.com (as expected)
$ ./scripts/test-networkpolicy-pod-to-pod.fish openshift-cluster-version
Testing NetworkPolicy pod-to-pod blocking in namespace: openshift-cluster-version
...
✅ TEST PASSED: NetworkPolicy is blocking pod-to-pod communication
Client pod was unable to reach server pod via service (as expected) |
I inspected CVO's log for signs of networking issues ( The I used $ oc adm upgrade recommend --version 4.20.10
Upstream: https://fauxinnati-fauxinnati.apps.ota-stage.q2z4.p1.openshiftapps.com/api/upgrades_info/graph
Channel: smoke-test
Update to 4.20.10 Recommended=False:
Image: quay.io/openshift-release-dev/ocp-release@sha256:00000000000000000000000000000000000000000000000000000000003d572a
Release URL: https://access.redhat.com/errata/RHSA-2024:06010
Reason: MultipleReasons
Message: This is RiskA part of combined risks for smoke testing https://docs.openshift.com/synthetic-risk-smoke-combined-a
This is RiskBMatches part of combined risks for smoke testing https://docs.openshift.com/synthetic-risk-smoke-combined-b Lastly, I updated (actually downgraded) the cluster to 4.20.0-ec.2 using its digest: $ oc adm upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:7f885da9a26ec7f460a4518a40811c03cf278853186833dc80e96a1ae15c9511 --allow-explicit-upgrade The update was successfully started which shows that CVO is able to fetch the signatures it needs from public cloud storage (without |
Test Scenario: CVO NetworkPolicy is blocking outbound traffic
execute test-networkpolicy-isolation.fish script from https://github.com/petr-muller/vibes/blob/main/scripts/test-networkpolicy-isolation.fish
|
Test Scenario: CVO NetworkPolicy is blocking pod-to-pod communication
execute test-networkpolicy-pod-to-pod.fish script from https://github.com/petr-muller/vibes/blob/main/scripts/test-networkpolicy-pod-to-pod.fish
|
Test Scenario: External access egress is blocked from openshift-cluster-version namespace
execute test-networkpolicy-external-access.fish from https://github.com/petr-muller/vibes/blob/main/scripts/test-networkpolicy-external-access.fish
|
Test Scenario: Ingress to CVO is not breaking for monitoring scrape
|
Test Scenario: Installation should be success with new Network Policy and CVO should be healthy.
Cluster should Installed successfully
|
Test Scenario: OSUS can connect to production/external cincinnati and fetch the upgrade graph based on the version in connected cluster environment. Note: for pre-merge testing we are using https://fauxinnati-fauxinnati.apps.ota-stage.q2z4.p1.openshiftapps.com/ to demonstrate the production.
Upstream should have proper value
|
Test Scenario: CVO able to evaluate conditional Risk's for upgrade path.
Patch the upstream
|
Test Scenario: CVO should be able to verify signatures for upgrade in the connected cluster environment
|
Testing results look good for now, nothing looks suspicious. I'd like to test the behaviour in disconnected environment in the following days, if that looks good I'll update here and this is good to go |
/lgtm The pull itself looks good to me. Update. |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hongkailiu, petr-muller The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
no need to test specifically in disconnected environment. as external network is blocked but similar service is available in intranet. that doesn't make any difference to connected/disconnected clusters |
/label qe-approved |
@petr-muller: This pull request references OTA-1521 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
1 similar comment
@petr-muller: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
[ART PR BUILD NOTIFIER] Distgit: cluster-version-operator |
Add a baseline NetworkPolicy to deny all network communication (both ingress and egress) to all pods in the namespace. Any necessary network traffic needs to be allowed by an additional NetworkPolicy resource (they are additive).
At the moment, the default deny all policy should be the only one needed:
version
pods spawned by CVO do not require any network communicationSee OTA Network Policies Working Document for more information.