-
Notifications
You must be signed in to change notification settings - Fork 27
feat: egress gw #1331
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
meganwolf0
wants to merge
23
commits into
main
Choose a base branch
from
meganwolf0/feat-egress-gw
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
feat: egress gw #1331
Changes from 13 commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
3f7ea9d
feat: egress gw
meganwolf0 5302b0d
Merge branch 'main' into meganwolf0/feat-egress-gw
meganwolf0 54a9504
Merge branch 'main' into meganwolf0/feat-egress-gw
mjnagel cbd2d08
Merge branch 'main' into meganwolf0/feat-egress-gw
meganwolf0 022eda7
Merge branch 'main' into meganwolf0/feat-egress-gw
meganwolf0 9d077b5
Merge branch 'main' into meganwolf0/feat-egress-gw
meganwolf0 9cd0a61
Merge branch 'main' into meganwolf0/feat-egress-gw
meganwolf0 ffe4772
Update docs/reference/configuration/egress.md
meganwolf0 a662bd1
Merge branch 'main' into meganwolf0/feat-egress-gw
meganwolf0 4795d88
fix: err handling
meganwolf0 fb06bc3
Update src/pepr/operator/controllers/istio/istio-resources.ts
meganwolf0 1dbebac
Merge branch 'main' into meganwolf0/feat-egress-gw
meganwolf0 b115d3a
fix: formatting
meganwolf0 e58a3f2
fix: concurrency fixes
meganwolf0 58f5e0d
fix: fail on missing ports
meganwolf0 95ba83f
fix: tests
meganwolf0 7bd11d7
Merge remote-tracking branch 'origin/main' into meganwolf0/feat-egres…
meganwolf0 387bf99
fix: defaults
meganwolf0 3b65599
Merge branch 'main' into meganwolf0/feat-egress-gw
meganwolf0 c8cc6a6
fix: logic
meganwolf0 d92b888
fix: dequeue fcn and test
meganwolf0 8b2841f
Merge branch 'main' into meganwolf0/feat-egress-gw
meganwolf0 0fb8cd9
fix: formatting
meganwolf0 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,120 @@ | ||
| --- | ||
| title: Istio Egress | ||
| --- | ||
|
|
||
| UDS Core leverages Istio to route dedicated egress out of the service mesh. This document provides an overview and examples of the Istio resources that UDS Core deploys to handle egress. | ||
|
|
||
| :::note | ||
| This does not currently work with ambient mode enabled (`spec.network.serviceMesh.mode=ambient`) or with workloads that omit the sidecar proxy | ||
| ::: | ||
|
|
||
| ## Configuring the Egress Workload | ||
|
|
||
| The dedicated egress gateway is an *optional* component of UDS Core. To enable it in the UDS Bundle, add it to the `optionalComponents` as follows: | ||
|
|
||
| ```yaml | ||
| kind: UDSBundle | ||
| metadata: | ||
| name: uds-core-bundle | ||
| description: My UDS Core Bundle | ||
| version: "0.1.0" | ||
|
|
||
| packages: | ||
| - name: uds-core | ||
| repository: oci://ghcr.io/defenseunicorns/packages/uds/core | ||
| version: "0.39.0" | ||
| optionalComponents: | ||
| - istio-egress-gateway | ||
| ``` | ||
|
|
||
| You will also need to configure any additional ports that you'd expect to egress to. 443 and 80 are default out of the box, but in the case of modifications you should use the `packages.overrides` as follows: | ||
|
|
||
| ```yaml | ||
| overrides: | ||
| istio-egress-gateway: | ||
| gateway: | ||
| ports: | ||
| - name: status-port | ||
| port: 15021 | ||
| protocol: TCP | ||
| targetPort: 15021 | ||
| - name: http2 | ||
| port: 80 | ||
| protocol: TCP | ||
| targetPort: 80 | ||
| - name: https | ||
| port: 443 | ||
| protocol: TCP | ||
| targetPort: 443 | ||
| - name: custom-port | ||
| port: 9200 | ||
| protocol: TCP | ||
| targetPort: 9200 | ||
| ``` | ||
|
|
||
| ## Specifying Egress using the Package CR | ||
|
|
||
| The UDS Core Package Custom Resource (CR) is used to configure the egress workload. The egress routes are realized through the use of the `network.allow` - specifically the `remoteHost`, `remoteProtocol`, and `port` or `ports` parameters therein. | ||
|
|
||
| :::note | ||
| Currently, only HTTP and TLS protocols are supported. The configuration will default to TLS if not specified. | ||
| ::: | ||
|
|
||
| :::note | ||
| Wildcards in host names are NOT currently supported. | ||
| ::: | ||
|
|
||
| The following sample Package CR shows configuring egress to a specific host, "httpbin.org", on port 443. | ||
|
|
||
| ```yaml | ||
| apiVersion: uds.dev/v1alpha1 | ||
| kind: Package | ||
| metadata: | ||
| name: pkg-1 | ||
| namespace: egress-gw-1 | ||
| spec: | ||
| network: | ||
| allow: | ||
| - description: "Example Curl" | ||
| direction: Egress | ||
| port: 443 | ||
| remoteHost: httpbin.org | ||
| remoteProtocol: TLS | ||
| selector: | ||
| app: curl | ||
| ``` | ||
|
|
||
| When a Package CR specifies the `network.allow` field with, at minimum, the `remoteHost` and `port` or `ports` parameters, the UDS Core operator will create the necessary Istio resources to allow traffic to egress from the mesh. These include the following: | ||
| * An Istio ServiceEntry, in the package namespace, which is used to define the external service that the workload can access. | ||
| * An Istio Sidecar, in the package namespace, which is used to enforce that only registered traffic can egress from the workload. This is only applied to the workload selected in the `network.allow`. | ||
| * A shared Istio VirtualService, in the istio egress gateway namespace, which is used to route the traffic to the egress gateway. | ||
| * A shared Istio Gateway, in the istio egress gateway namespace, which is used to expose the egress gateway to the outside world. | ||
| * A shared Istio Service Entry, in the istio egress gateway namespace, to register the hosts and the ports for the egress gateway. | ||
|
|
||
| ## Limitations | ||
|
|
||
| The configuration in Package CRs in combination with the behavior of Istio should be understood when using egress. There are a few "gotchas" that might occur while using the egress configurations. | ||
|
|
||
| :::note | ||
| The following are not exhaustive and are subject to change as this implementation matures from sidecar to ambient. | ||
| ::: | ||
|
|
||
| * Currently, egress will only work for workloads that are using the Istio sidecar proxy. | ||
|
|
||
| * Specifying a port in a Package that is not exposed via the workload: This will be allowed with a warning from the operator, but the traffic will not be able to egress. An `istioctl analyze` will show an error such as: `Referenced host:port not found: "egressgateway.istio-egress-gateway.svc.cluster.local:9200"` | ||
|
|
||
| * Specifying a remote host that is also used in other Gateways or VirtualServices: This will be allowed with a warning from the operator, but some unexpected behavior may occur. An `istioctl analyze` will show an error such as: `The VirtualServices ... define the same host ... which can lead to unexpected behavior` and `Conflict with gateways ...` | ||
|
|
||
| * For all egresses defined within a single Package CR, all workloads that also have egress will have shared access to any host defined (is that true with the VS?) | ||
|
|
||
| ## Security Considerations | ||
|
|
||
| Additional security considerations to keep in mind when implementing egress: | ||
|
|
||
| * The TLS mode is PASSTHROUGH, this means that traffic will exit the mesh as-is. Without TLS origination, details like HTTP paths cannot be inspected, restricted or logged. | ||
|
|
||
| * Per Istio documentation: “The cluster administrator or the cloud provider must ensure that no traffic leaves the mesh bypassing the egress gateway. Mechanisms external to Istio must enforce this requirement” - Essentially, additional work may be needed to ensure traffic is actually egressing the cluster when and where it should be. | ||
|
|
||
| * Some potential vulnerabilities are introduced using TLS Passthrough - you’ll need to know what’s on the other side of that domain because of [domain fronting](https://en.wikipedia.org/wiki/Domain_fronting) - Essentially, this is only a safe feature for trusted hosts, or hosts you know are not vulnerable to domain fronting | ||
|
|
||
| * We are not blocking DNS exfiltration |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| # Copyright 2024 Defense Unicorns | ||
| # SPDX-License-Identifier: AGPL-3.0-or-later OR LicenseRef-Defense-Unicorns-Commercial | ||
|
|
||
| service: | ||
| type: ClusterIP |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
112 changes: 112 additions & 0 deletions
112
src/pepr/operator/controllers/istio/defaultTestMocks.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| /** | ||
| * Copyright 2024 Defense Unicorns | ||
| * SPDX-License-Identifier: AGPL-3.0-or-later OR LicenseRef-Defense-Unicorns-Commercial | ||
| */ | ||
|
|
||
| import { jest } from "@jest/globals"; | ||
| import { V1OwnerReference } from "@kubernetes/client-node"; | ||
| import { K8s, kind } from "pepr"; | ||
| import { | ||
| IstioGateway, | ||
| IstioServiceEntry, | ||
| IstioSidecar, | ||
| IstioVirtualService, | ||
| RemoteProtocol, | ||
| UDSPackage, | ||
| } from "../../crd"; | ||
| import { PackageHostMap } from "./types"; | ||
|
|
||
| export const pkgMock: UDSPackage = { | ||
| metadata: { | ||
| name: "test-package", | ||
| namespace: "test-namespace", | ||
| generation: 1, | ||
| }, | ||
| spec: { | ||
| network: { | ||
| expose: [], | ||
| allow: [], | ||
| }, | ||
| }, | ||
| }; | ||
|
|
||
| export const ownerRefsMock: V1OwnerReference[] = [ | ||
| { | ||
| apiVersion: "uds.dev/v1alpha1", | ||
| kind: "Package", | ||
| name: "test-package", | ||
| uid: "f50120aa-2713-4502-9496-566b102b1174", | ||
| }, | ||
| ]; | ||
|
|
||
| export const pkgHostMapMock: PackageHostMap = { | ||
| package1: { | ||
| "example.com": { | ||
| portProtocol: [{ port: 443, protocol: RemoteProtocol.TLS }], | ||
| }, | ||
| }, | ||
| }; | ||
|
|
||
| export const defaultEgressMocks = { | ||
| applyGwMock: jest.fn<() => Promise<void>>().mockResolvedValue(), | ||
| applyVsMock: jest.fn<() => Promise<void>>().mockResolvedValue(), | ||
| applySeMock: jest.fn<() => Promise<void>>().mockResolvedValue(), | ||
| applySidecarMock: jest.fn<() => Promise<void>>().mockResolvedValue(), | ||
| getGwMock: jest.fn<() => Promise<{ items: IstioGateway[] }>>().mockResolvedValue({ | ||
| items: [], | ||
| }), | ||
| getVsMock: jest.fn<() => Promise<{ items: IstioVirtualService[] }>>().mockResolvedValue({ | ||
| items: [], | ||
| }), | ||
| getNsMock: jest.fn<() => Promise<kind.Namespace>>().mockResolvedValue({}), | ||
| deleteGwMock: jest.fn<() => Promise<void>>().mockResolvedValue(), | ||
| deleteVsMock: jest.fn<() => Promise<void>>().mockResolvedValue(), | ||
| deleteSeMock: jest.fn<() => Promise<void>>().mockResolvedValue(), | ||
| deleteSidecarMock: jest.fn<() => Promise<void>>().mockResolvedValue(), | ||
| }; | ||
|
|
||
| export function updateEgressMocks(egressMocks: Record<string, jest.Mock>) { | ||
| const baseImplementation = { | ||
| Apply: jest.fn<() => Promise<void>>().mockResolvedValue(), | ||
| InNamespace: jest.fn().mockReturnThis(), | ||
| Get: jest.fn(), | ||
| Logs: jest.fn(), | ||
| Delete: jest.fn(), | ||
| Watch: jest.fn(), | ||
| WithLabel: jest.fn(), | ||
| }; | ||
|
|
||
| const mockK8s = jest.mocked(K8s); | ||
|
|
||
| // Define only the implementations for specific resources | ||
| const k8sImplementations = { | ||
| [IstioGateway.name]: { | ||
| ...baseImplementation, | ||
| Get: egressMocks.getGwMock, | ||
| Apply: egressMocks.applyGwMock, | ||
| Delete: egressMocks.deleteGwMock, | ||
| }, | ||
| [IstioVirtualService.name]: { | ||
| ...baseImplementation, | ||
| Get: egressMocks.getVsMock, | ||
| Apply: egressMocks.applyVsMock, | ||
| Delete: egressMocks.deleteVsMock, | ||
| }, | ||
| [IstioServiceEntry.name]: { | ||
| ...baseImplementation, | ||
| Apply: egressMocks.applySeMock, | ||
| Delete: egressMocks.deleteSeMock, | ||
| }, | ||
| [IstioSidecar.name]: { | ||
| ...baseImplementation, | ||
| Apply: egressMocks.applySidecarMock, | ||
| Delete: egressMocks.deleteSidecarMock, | ||
| }, | ||
| [kind.Namespace.name]: { ...baseImplementation, Get: egressMocks.getNsMock }, | ||
| }; | ||
|
|
||
| mockK8s.mockImplementation( | ||
| // eslint-disable-next-line @typescript-eslint/no-explicit-any | ||
| ((model: any) => k8sImplementations[model.name] || baseImplementation) as any, | ||
| ); | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.