Skip to content

Commit 39e4114

Browse files
fromsauravSaurav Teli
fromsaurav
authored and
Saurav Teli
committed
Add best practices for using OpenKruise workloads in Karmada and OCM
Signed-off-by: Saurav Teli <[email protected]>
1 parent ff8dcec commit 39e4114

File tree

1 file changed

+218
-0
lines changed

1 file changed

+218
-0
lines changed

‎docs/tutorial/multi-cluster-orchestration.md

Lines changed: 218 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
# Using OpenKruise Workloads in Multi-Cluster Orchestration Systems
2+
3+
This document provides best practices and guidelines for using OpenKruise workloads with popular multi-cluster orchestration systems, specifically Karmada and Open Cluster Management (OCM).
4+
5+
## Table of Contents
6+
7+
- [Introduction](#introduction)
8+
- [OpenKruise Workloads Overview](#openkruise-workloads-overview)
9+
- [Karmada Integration](#karmada-integration)
10+
- [Understanding Karmada Resource Interpreters](#understanding-karmada-resource-interpreters)
11+
- [CloneSet in Karmada](#cloneset-in-karmada)
12+
- [Advanced StatefulSet in Karmada](#advanced-statefulset-in-karmada)
13+
- [Other OpenKruise Workloads](#other-openkruise-workloads)
14+
- [Open Cluster Management (OCM) Integration](#open-cluster-management-ocm-integration)
15+
- [ManifestWork for OpenKruise Workloads](#manifestwork-for-openkruise-workloads)
16+
- [Best Practices for OCM Integration](#best-practices-for-ocm-integration)
17+
- [Common Challenges and Solutions](#common-challenges-and-solutions)
18+
- [References](#references)
19+
20+
## Introduction
21+
22+
[OpenKruise](https://openkruise.io) extends Kubernetes with enhanced workload management capabilities. When deploying OpenKruise workloads in multi-cluster environments, Karmada and Open Cluster Management (OCM) are popular orchestration frameworks that can be used.
23+
24+
This guide outlines best practices for integrating OpenKruise workloads like CloneSet, Advanced StatefulSet, and others with these multi-cluster orchestration systems.
25+
26+
## OpenKruise Workloads Overview
27+
28+
Before diving into multi-cluster integration, it's important to understand the key OpenKruise workloads:
29+
30+
- **CloneSet**: Enhanced deployment workload with in-place update capabilities
31+
- **Advanced StatefulSet**: Extended version of Kubernetes StatefulSet with additional features
32+
- **SidecarSet**: For managing sidecar containers across multiple pods
33+
- **UnitedDeployment**: Manages multiple workloads across different domains
34+
35+
## Karmada Integration
36+
37+
[Karmada](https://karmada.io) (Kubernetes Armada) is a multi-cluster orchestration system that enables unified management of multiple Kubernetes clusters.
38+
39+
### Understanding Karmada Resource Interpreters
40+
41+
Karmada uses Resource Interpreters to understand how to handle custom resources from different providers. For OpenKruise workloads, you need to define appropriate interpreters to ensure Karmada can properly manage these resources.
42+
43+
Resource Interpreters help Karmada understand:
44+
- How to retain or reset certain fields during propagation
45+
- How to get replicas and scale workloads
46+
- How to aggregate resource status from member clusters
47+
48+
### CloneSet in Karmada
49+
50+
To use CloneSet with Karmada, you need to create a proper ResourceInterpreter for the CloneSet resource:
51+
52+
```yaml
53+
apiVersion: config.karmada.io/v1alpha1
54+
kind: ResourceInterpreterCustomization
55+
metadata:
56+
name: openkruise-cloneset-interpreter
57+
spec:
58+
target:
59+
apiVersion: apps.kruise.io/v1alpha1
60+
kind: CloneSet
61+
customizations:
62+
replicas:
63+
jsonPath: ".spec.replicas"
64+
scale:
65+
jsonPath: ".spec.replicas"
66+
statusReflection:
67+
jsonPath: ".status"
68+
retention:
69+
jsonPaths:
70+
- ".spec.updateStrategy"
71+
- ".spec.scaleStrategy"
72+
```
73+
74+
When creating a PropagationPolicy for CloneSet resources, consider:
75+
76+
```yaml
77+
apiVersion: policy.karmada.io/v1alpha1
78+
kind: PropagationPolicy
79+
metadata:
80+
name: cloneset-propagation
81+
spec:
82+
resourceSelectors:
83+
- apiVersion: apps.kruise.io/v1alpha1
84+
kind: CloneSet
85+
placement:
86+
clusterAffinity:
87+
clusterNames:
88+
- member1
89+
- member2
90+
replicaScheduling:
91+
replicaSchedulingType: Divided
92+
```
93+
94+
### Advanced StatefulSet in Karmada
95+
96+
For Advanced StatefulSet, create a ResourceInterpreter similar to CloneSet but tailored to StatefulSet specifics:
97+
98+
```yaml
99+
apiVersion: config.karmada.io/v1alpha1
100+
kind: ResourceInterpreterCustomization
101+
metadata:
102+
name: openkruise-statefulset-interpreter
103+
spec:
104+
target:
105+
apiVersion: apps.kruise.io/v1beta1
106+
kind: StatefulSet
107+
customizations:
108+
replicas:
109+
jsonPath: ".spec.replicas"
110+
scale:
111+
jsonPath: ".spec.replicas"
112+
statusReflection:
113+
jsonPath: ".status"
114+
retention:
115+
jsonPaths:
116+
- ".spec.updateStrategy"
117+
- ".spec.podManagementPolicy"
118+
```
119+
120+
### Other OpenKruise Workloads
121+
122+
For other OpenKruise workloads like SidecarSet and UnitedDeployment, you'll need to create custom interpreters based on their specific fields. For example, UnitedDeployment manages multiple workloads across different domains, so its interpreter should reflect this complexity.
123+
124+
## Open Cluster Management (OCM) Integration
125+
126+
[Open Cluster Management](https://open-cluster-management.io) (OCM) is another framework for managing multiple Kubernetes clusters, using the hub-spoke model.
127+
128+
### ManifestWork for OpenKruise Workloads
129+
130+
OCM uses ManifestWork resources to deploy workloads to managed clusters. Here's how to use ManifestWork with OpenKruise resources:
131+
132+
```yaml
133+
apiVersion: work.open-cluster-management.io/v1
134+
kind: ManifestWork
135+
metadata:
136+
name: example-cloneset
137+
namespace: cluster1 # This should be the managed cluster's namespace
138+
spec:
139+
workload:
140+
manifests:
141+
- apiVersion: apps.kruise.io/v1alpha1
142+
kind: CloneSet
143+
metadata:
144+
name: example-cloneset
145+
namespace: default
146+
spec:
147+
replicas: 3
148+
selector:
149+
matchLabels:
150+
app: example
151+
template:
152+
metadata:
153+
labels:
154+
app: example
155+
spec:
156+
containers:
157+
- name: nginx
158+
image: nginx:latest
159+
```
160+
161+
### Best Practices for OCM Integration
162+
163+
When using OpenKruise workloads with OCM:
164+
165+
1. **Pre-install OpenKruise on managed clusters**: Ensure OpenKruise is installed on all managed clusters before deploying OpenKruise CRDs through ManifestWork.
166+
167+
2. **Use ManifestWork status tracking**: Leverage OCM's ManifestWork status tracking to monitor the deployment status:
168+
169+
```yaml
170+
apiVersion: work.open-cluster-management.io/v1
171+
kind: ManifestWork
172+
metadata:
173+
name: example-cloneset
174+
namespace: cluster1
175+
spec:
176+
workload:
177+
manifests:
178+
- apiVersion: apps.kruise.io/v1alpha1
179+
kind: CloneSet
180+
# ... CloneSet spec ...
181+
manifestConfigs:
182+
- resourceIdentifier:
183+
group: apps.kruise.io
184+
resource: clonesets
185+
name: example-cloneset
186+
namespace: default
187+
feedbackRules:
188+
- type: WellKnownStatus
189+
```
190+
191+
3. **Configure ApplicationSet with OCM**: If using ArgoCD alongside OCM, configure ApplicationSet generators to target OCM-managed clusters.
192+
193+
## Common Challenges and Solutions
194+
195+
### Challenge 1: OpenKruise CRD Distribution
196+
197+
**Problem**: OpenKruise CRDs must be installed on all clusters before workloads can be deployed.
198+
199+
**Solution**: Use a ManifestWork or Karmada Policy to ensure OpenKruise operator and CRDs are deployed to all clusters first. Consider using ClusterManagementAddons in OCM.
200+
201+
### Challenge 2: Version Compatibility
202+
203+
**Problem**: Different OpenKruise versions across clusters can cause compatibility issues.
204+
205+
**Solution**: Standardize OpenKruise versions across all clusters or use version-specific manifests.
206+
207+
### Challenge 3: Workload-Specific Features
208+
209+
**Problem**: Some OpenKruise workloads have unique features not easily translated to standard kubernetes concepts.
210+
211+
**Solution**: Create custom resource interpreters in Karmada or use OCM ManifestWork feedback rules to properly handle these unique features.
212+
213+
## References
214+
215+
- [OpenKruise Documentation](https://openkruise.io/docs/)
216+
- [Karmada Resource Interpreter](https://karmada.io/docs/userguide/globalview/customizing-resource-interpreter/)
217+
- [OCM ManifestWork](https://open-cluster-management.io/docs/concepts/work-distribution/manifestwork/)
218+
- [KubeCon Talk on Multi-Cluster Management](https://www.youtube.com/watch?v=gcllTXRkz-E)

0 commit comments

Comments
 (0)