Skip to content

Add README for k8s multi cluster plugin #5854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 20, 2025
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 147 additions & 0 deletions pkg/app/pipedv1/plugin/kubernetes_multicluster/README.md
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: Let's add links to each plugin README in somewhere (like root README? pipedv1 README? or another) because this file is hard to find.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mention 'Currently only QuickSync is supported' somewhere

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed on dab25fd

Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Kubernetes Multicluster Plugin

## Specification

The current specification is described in the [RFC](../../../../../docs/rfcs/0014-multi-cluster-deployment-for-k8s.md).
The configuration format is unstable and may change in the future.

## Try k8s multicluster plugin locally

**Prepare the PipeCD Control Plane**

Please refer to [pipe-cd/pipecd/cmd/pipecd/README.md](../../../../../cmd/pipecd/README.md) to set up the Control Plane in your local environment.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to specify the oldest version which is compatible with pipedv1 or just specify latest

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add comment on b29e4bf.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ffjlabo
Precisely, v0.51.3 does not contain some commits of plugins since they're not cherry picked.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to specify the commit when we released v0.51.3 instead of release-v0.51.x branch

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@t-kikuc

Precisely, v0.51.3 does not contain some commits of plugins since they're not cherry picked.

We need to specify the commit when we released v0.51.3 instead of release-v0.51.x branch

Thank you.
So, how about describing it using the upstream of the master branch?
The whole fixes for plugins are stored in that branch, and this is the docs for running locally with make command.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ffjlabo
I agree!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed on c012e03


**Prepare two k8s clusters**

```sh
kind create cluster --name cluster1
kind export kubeconfig --name cluster1 --kubeconfig /path/to/kubeconfig/for/cluster1

kind create cluster --name cluster2
kind export kubeconfig --name cluster2 --kubeconfig /path/to/kubeconfig/for/cluster2
```

**Start pipedv1 locally**

Please refer to [pipe-cd/pipecd/cmd/pipedv1/README.md](../../../../../cmd/pipedv1/README.md) to set up the Control Plane in your local environment.
At this time, please modify the `spec.plugins` section of the piped config as shown below.

```yaml
apiVersion: pipecd.dev/v1beta1
kind: Piped
spec:
...
plugins:
- name: kubernetes_multicluster
port: 7002 # any unused port
url: file:///path/to/.piped/plugins/kubernetes_multicluster # It's OK using any value for now because it's a dummy. We will implement it later.
deployTargets:
- name: cluster1
config:
masterURL: https://127.0.0.1:61337 # shown by kubectl cluster-info
kubeConfigPath: /path/to/kubeconfig/for/cluster1
- name: cluster2
config:
masterURL: https://127.0.0.1:62082 # shown by kubectl cluster-info
kubeConfigPath: /path/to/kubeconfig/for/cluster2
```

**Prepare the manifest**

- Please create a new repository for the manifest because an error will occur if there is an app.pipecd.yaml in the format before supporting the plugin mechanism for now.
For example usage, see the [Examples](#examples) section below.

**Register the application**

![adding-application](./docs/static/adding-application.png)

At this time, please select multiple DeployTargets.

## Examples
There are examples under `./example`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the examples should be moved to https://github.com/pipe-cd/examples in the future.

What about for now? Put in examples/ or the examples repo?
WDYT? @Warashi @khanhtc1202

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the examples should be moved to https://github.com/pipe-cd/examples in the future.

I agree with it. I think the examples for built-in plugins should be located in it.

What about for now? Put in examples/ or the examples repo?

I think we should put them in examples for now.
Currently, some errors will occur when there are app.pipecd.yaml files for both pipedv0 and pipedv1.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should put them in examples for now.

I agree! It's still like a draft.

I approve, and if others have a different opinion, I'll reconsider it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with putting them under the plugin implementation directory for now because the configuration format may change in the future, as written in the head of this README. Putting them here is better because we can reduce the change scope in future pull requests; they are both under the plugin implementation directory.


| Name | Description |
|------|-------------|
| [simple](./example/simple/) | Deploy the same resources to the multiple clusters. |
| [multi-sources-template-none](./example/multi-sources-template-none/) | Deploy the different resources to the multiple clusters. |

## Config Reference

### Piped Config

```yaml
apiVersion: pipecd.dev/v1beta1
kind: Piped
spec:
...
plugins:
# Plugin name.
- name: kubernetes_multicluster
# Port number used to listen plugin server.
port: 7002
# The URL where the plugin binary located.
# It's OK using any value for now because it's a dummy. We will implement it later.
url: file:///path/to/.piped/plugins/kubernetes_multicluster
# List of the information for each target platform.
# This is alternative for the platform providers.
deployTargets:
# Then name of deploy target.
- name: cluster1
# The plugin-specific config.
config:
# The master URL of the kubernetes cluster.
# Empty means in-cluster.
masterURL: https://127.0.0.1:61337
# The path to the kubeconfig file.
# Empty means in-cluster.
kubeConfigPath: /path/to/kubeconfig/for/cluster1
# Version of kubectl will be used.
kubectlVersion: 1.32.0
- name: cluster2
config:
masterURL: https://127.0.0.1:62082
kubeConfigPath: /path/to/kubeconfig/for/cluster2
```

### app.pipecd.yaml

```yaml
apiVersion: pipecd.dev/v1beta1
kind: Application
spec:
...
plugins:
kubernetes_multicluster:
input:
# List of manifest files in the application directory used to deploy.
# Empty means all manifest files in the directory will be used.
manifests:
- deployment.yaml
- service.yaml
# Version of kubectl will be used.
kubectlVersion: 1.32.0
# The namespace where manifests will be applied.
namespace: example
# Automatically create a new namespace if it does not exist.
# Default is false.
autoCreateNamespace: false
# List of the setting for each deploy targets.
# You can also set config for them separately.
multiTargets:
# The identity of the deploy target.
# You can specify deploy target by some of them.
- target:
# The name of deploy target
name: cluster1
# List of manifest files in the application directory used to deploy.
# Empty means all manifest files in the directory will be used.
manifests:
- ./cluster1/deployment.yaml
# Version of kubectl will be used.
kubectlVersion: 1.32.2
- target:
name: cluster2
manifests:
- ./cluster2/deployment.yaml
kubectlVersion: 1.32.2
```
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
apiVersion: pipecd.dev/v1beta1
kind: Application
spec:
name: multi-sources-template-none
labels:
env: example
plugins:
kubernetes_multicluster:
quickSync:
prune: true
input:
multiTargets:
- target:
name: cluster1
manifests:
- ./cluster1/deployment.yaml
- target:
name: cluster2
manifests:
- ./cluster2/deployment.yaml
kubectlVersion: 1.32.2
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: cluster1
labels:
app: cluster1
spec:
replicas: 2
selector:
matchLabels:
app: cluster1
pipecd.dev/variant: primary
template:
metadata:
labels:
app: cluster1
pipecd.dev/variant: primary
annotations:
sidecar.istio.io/inject: "false"
spec:
containers:
- name: helloworld
image: ghcr.io/pipe-cd/helloworld:v0.32.0
args:
- server
ports:
- containerPort: 9085
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: cluster2
labels:
app: cluster2
spec:
replicas: 2
selector:
matchLabels:
app: cluster2
pipecd.dev/variant: primary
template:
metadata:
labels:
app: cluster2
pipecd.dev/variant: primary
annotations:
sidecar.istio.io/inject: "false"
spec:
containers:
- name: helloworld
image: ghcr.io/pipe-cd/helloworld:v0.32.0
args:
- server
ports:
- containerPort: 9085
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
apiVersion: pipecd.dev/v1beta1
kind: Application
spec:
name: simple
labels:
env: example
plugins:
kubernetes_multicluster:
input:
manifests:
- deployment.yaml
- service.yaml
kubectlVersion: 1.32.2
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: simple
labels:
app: simple
spec:
replicas: 2
selector:
matchLabels:
app: simple
pipecd.dev/variant: primary
template:
metadata:
labels:
app: simple
pipecd.dev/variant: primary
annotations:
sidecar.istio.io/inject: "false"
spec:
containers:
- name: helloworld
image: ghcr.io/pipe-cd/helloworld:v0.32.0
args:
- server
ports:
- containerPort: 9085
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
apiVersion: v1
kind: Service
metadata:
name: simple
spec:
selector:
app: simple
ports:
- protocol: TCP
port: 9085
targetPort: 9085