Skip to content

Commit 72914ba

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #1446 from fabriziopandini/welcome-kinder
Welcome kinder!
2 parents 5948428 + 9bbf63e commit 72914ba

File tree

2 files changed

+302
-0
lines changed

2 files changed

+302
-0
lines changed

‎kinder/README.md

+257
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,257 @@
1+
# Kinder
2+
3+
kinder is an example of [kind](https://github.com/kubernetes-sigs/kind) used as a library.
4+
5+
All the kind commands will be available in kinder, side by side with additional commands
6+
designed for helping kubeadm contributors.
7+
8+
**kinder is a work in progress. Test it! Break it! Send feeback!**
9+
10+
## Prerequisites
11+
12+
### Install git
13+
14+
Our source code is managed with [git](https://git-scm.com/), to develop locally you will need to install git.
15+
16+
You can check if git is already on your system and properly installed with the following command:
17+
18+
```bash
19+
git --version
20+
```
21+
22+
### Install Go
23+
24+
To work on kind’s codebase you will need [Go](https://golang.org/doc/install).
25+
26+
Install or upgrade [Go using the instructions for your operating system](https://golang.org/doc/install).
27+
You can check if Go is in your system with the following command:
28+
29+
```bash
30+
go version
31+
```
32+
33+
Working with Go [modules](https://kind.sigs.k8s.io/docs/contributing/getting-started/which%20we%20use%20for%20dependency%20management) requires at least 1.11.4 due to checksum bugs
34+
in lower versions.
35+
36+
## Getting started
37+
38+
Read the [kind documentation](https://kind.sigs.k8s.io/docs/user/quick-start/) first, becuase kinder
39+
is built using kind as a library.
40+
41+
Clone the kubeadm repo:
42+
43+
```bash
44+
git clone https://github.com/kubernetes/kubeadm.git
45+
```
46+
47+
And then build kinder
48+
49+
```bash
50+
cd kinder
51+
GO111MODULE=on go install
52+
```
53+
54+
This will put kinder in $(go env GOPATH)/bin.
55+
56+
## Create a test cluster
57+
58+
You can create a cluster in kinder using `kinder create cluster`, that is a wrapper on the `kind create cluster` command.
59+
60+
However, additional flags are implemented for enabling following use cases:
61+
62+
### Create *only* nodes
63+
64+
By default kinder stops the cluster creation process before executing `kubeadm init` and `kubeadm join`;
65+
This will give you nodes ready for intalling Kubernetes and more specifically
66+
67+
- the necessary prerequisited already installed on all nodes
68+
- a pre-build kubeadm config file in `/kind/kubeadm.conf`
69+
- in case of more than one control-plane node exists in the cluster, a pre-configured external load balancer
70+
71+
If instead you want to revert to the default kind behaviour, you can use the `--setup-kubernetes`:
72+
73+
```bash
74+
kinder create cluster --setup-kubernetes=true
75+
```
76+
77+
### Testing different cluster topologies
78+
79+
You can use the `--control-plane-nodes <num>` flag and/or the `--worker-nodes <num>` flag
80+
as a shurtcut for creating different cluster topologies. e.g.
81+
82+
```bash
83+
# create a cluster with two worker nodes
84+
kinder create cluster --worker-nodes=2
85+
86+
# create a cluster with two control-pane nodes
87+
kinder create cluster ---control-plane-nodes=2
88+
```
89+
90+
Please note that a load balancer node will be automatically create when there are more than
91+
one control-plane node; if necessary, you can use `--external-load balancer` flag to explicitly
92+
request the creation of an external load balancer node.
93+
94+
More sophisticated cluster topologies can be achieved using the kind config file, like e.g. customizing
95+
kubeadm-config or specifying volume mounts. see [kind documentation](https://kind.sigs.k8s.io/docs/user/quick-start/#configuring-your-kind-cluster)
96+
for more details.
97+
98+
### Testing Kubernetes cluster variants
99+
100+
kinder gives you shortcuts for testing Kubernetes cluster variants supported by kubeadm:
101+
102+
```bash
103+
# create a cluster using kube-dns instead of CoreDNS
104+
kinder create cluster --kube-dns
105+
106+
# create a cluster using an external etcd
107+
kinder create cluster --external-etcd
108+
```
109+
110+
## Working on nodes
111+
112+
You can use `docker exec` and `docker cp` to work on nodes.
113+
114+
```bash
115+
# check the content of the /kind/kubeadm.conf file
116+
docker exec kind-control-plane cat kind/kubeadm.conf
117+
118+
# execute a command on the kind-control-plane container (the control-plane node)
119+
docker exec kind-worker1 \
120+
kubeadm init --config=/kind/kubeadm.conf --ignore-preflight-errors=all
121+
122+
# override the kubeadm binary on the on the kind-control-plane container
123+
# with a locally built kubeadm binary
124+
docker cp \
125+
$working_dir/kubernetes/bazel-bin/cmd/kubeadm/linux_amd64_pure_stripped/kubeadm \
126+
kind-control-plane:/usr/bin/kubeadm
127+
```
128+
129+
On top of that, kinder offers you three commands for helping you working on nodes:
130+
131+
- `kinder do` allowing you to execute actions (repetitive tasks/sequence of commands) on nodes
132+
- `kinder exec`, a topology aware wrapper on docker `docker exec`
133+
- `kinder cp`, a topology aware wrapper on docker `docker cp`
134+
135+
### kinder do
136+
137+
`kinder do` is the kinder swiss knife.
138+
139+
It allows to execute actions (repetitive tasks/sequence of commands) on one or more nodes
140+
in the local Kubernetes cluster.
141+
142+
```bash
143+
# Execute kubeadm init, installs the CNI plugin and copy the kubeconfig file on the host
144+
kinder do kubeadm-init
145+
```
146+
147+
All the actions implemented in kinder are by design "developer friendly", in the sense that
148+
all the command output will be echoed and all the step will be documented.
149+
Following actions are available:
150+
151+
| action | Notes |
152+
| --------------- | ------------------------------------------------------------ |
153+
| kubeadm-init | Executes the kubeadm-init workflow, installs the CNI plugin and then copies the kubeconfig file on the host machine. Available options are:<br /> `--use-phases` triggers execution of the init workflow by invoking single phases. <br />`--automatic-copy-certs` instruct kubeadm to use the automatic copy cert feature.|
154+
| manual-copy-certs | Implement the manual copy of certificates to be shared acress control-plane nodes (n.b. manual means not managed by kubeadm) Available options are:<br /> `--only-node` to execute this action only on a specific node. |
155+
| kubeadm-join | Executes the kubeadm-join workflow both on secondary control plane nodes and on worker nodes. Available options are:<br /> `--use-phases` triggers execution of the init workflow by invoking single phases.<br />`--automatic-copy-certs` instruct kubeadm to use the automatic copy cert feature.<br /> `--only-node` to execute this action only on a specific node. |
156+
| kubeadm-upgrade |Executes the kubeadm upgrade workflow and upgrading K8s. Available options are:<br /> `--upgrade-version` for defining the target K8s version.<br />`--only-node` to execute this action only on a specific node. |
157+
| Kubeadm-reset | Executes the kubeadm-reset workflow on all the nodes. Available options are:<br /> `--only-node` to execute this action only on a specific node. |
158+
| cluster-info | Returns a summary of cluster info including<br />- List of nodes<br />- list of pods<br />- list of images used by pods<br />- list of etcd members |
159+
| smoke-test | Implements a non-exhaustive set of tests that aim at ensuring that the most important functions of a Kubernetes cluster work |
160+
161+
### kinder exec
162+
163+
`kinder exec` provide a topology aware wrapper on docker `docker exec` .
164+
165+
```bash
166+
# check the kubeadm version on all the nodes
167+
kinder exec @all -- kubeadm version
168+
169+
# run kubeadm join on all the worker nodes
170+
kinder exec @w* -- kubeadm join 172.17.0.2:6443 --token abcdef.0123456789abcdef ...
171+
172+
# run kubectl command inside the bootstrap control-plane node
173+
kinder exec @cp1 -- kubectl --kubeconfig=/etc/kubernetes/admin.conf cluster-info
174+
```
175+
176+
Following node selectors are available
177+
178+
| selector | return the following nodes |
179+
| -------- | ------------------------------------------------------------ |
180+
| @all | all the Kubernetes nodes in the cluster.<br />(control-plane and worker nodes are included, load balancer and etcd not) |
181+
| @cp* | all the control-plane nodes |
182+
| @cp1 | the bootstrap-control plane node |
183+
| @cpN | the secondary master nodes |
184+
| @w* | all the worker nodes |
185+
| @lb | the external load balancer |
186+
| @etcd | the external etcd |
187+
188+
As alternative to node selector, the node name (the container name without the cluster name prefix) can be used to target actions to a specific node.
189+
190+
```bash
191+
# run kubeadm join on the first worker node node only
192+
kinder exec worker1 -- kubeadm join 172.17.0.2:6443 --token abcdef.0123456789abcdef ...
193+
```
194+
195+
### kinder cp
196+
197+
`kinder cp` provide a topology aware wrapper on docker `docker cp` . Following feature are supported:
198+
199+
```bash
200+
# copy to the host the /kind/kubeadm.conf file existing on the bootstrap control-plane node
201+
kinder cp @cp1:kind/kubeadm.conf kubeadm.conf
202+
203+
# copy to the bootstrap control-plane node a local kubeadm.conf file
204+
kinder cp kubeadm.conf @cp1:kind/kubeadm.conf
205+
206+
# override the kubeadm binary on all the nodes with a locally built kubeadm binary
207+
kinder cp \
208+
$working_dir/kubernetes/bazel-bin/cmd/kubeadm/linux_amd64_pure_stripped/kubeadm \
209+
@all:/usr/bin/kubeadm
210+
```
211+
212+
> Please note that, `docker cp` or `kinder cp` allows you to replace the kubeadm kinary on existing nodes. If you want to replace the kubeadm binary on nodes that you create in future, please check altering node images paragraph
213+
214+
## Altering node images
215+
216+
Kind can be estremely efficient when the node image contains all the necessary artifacts.
217+
218+
kinder allows kubeadm contributor to exploit this feature by implementing the `kinder build node-variant` command, that takes a node-image and allows to build variants by:
219+
220+
- Adding new pre-loaded images that will be made available on all nodes at cluster creation time
221+
- Replacing the kubeadm binary installed in the cluster, e.g. with a locally build version of kubeadm
222+
- Adding deb packages for a second Kubernetes version to be used for upgrade testing
223+
224+
The above options can be combined toghether in one command, if necessary
225+
226+
### Add images
227+
228+
```bash
229+
kinder build node-variant \
230+
--base-image kindest/node:latest \
231+
--image kindest/node:PR12345 \
232+
--with-images $my-local-images/nginx.tar
233+
```
234+
235+
Both single file or folder can be used as a arguments for the `--with-images`, but only image tar files will be considered; Image tar file will be placed in a well know folder, and kind(er) will load them during the initialization of each node.
236+
237+
### Replace kubeadm binary
238+
239+
```bash
240+
kinder build node-variant \
241+
--base-image kindest/node:latest \
242+
--image kindest/node:PR12345 \
243+
--with-kubeadm $working_dir/kubernetes/bazel-bin/cmd/kubeadm/linux_amd64_pure_stripped/kubeadm
244+
```
245+
246+
> Please note that, replacing the kubeadm binary in the node-images will have effect on nodes that you create in future; If you want to replace the kubeadm kinary on existing nodes, you should use `docker cp` or `kinder cp` instead.
247+
248+
### Add upgrade packages
249+
250+
```bash
251+
kinder build node-variant \
252+
--base-image kindest/node:latest \
253+
--image kindest/node:PR12345 \
254+
--with-upgrade-packages $my-local-packages/v1.12.2/
255+
```
256+
257+
Both single file or folder can be used as a arguments for the `--with-upgrade-packages`, but only deb packages will be considered; deb files will be placed in a well know folder, the kubeadm-upgrade action will use them during the upgrade sequence.

‎kinder/roadmap.md

+45
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Roadmap 🗺️
2+
3+
This document outlines some goals, non-goals, and future aspirations for kinder.
4+
5+
kinder is an example of [kind](https://github.com/kubernetes-sigs/kind) used as a library.
6+
7+
All the kind commands will be available in kinder, side by side with additional commands designed
8+
for helping kubeadm contributors.
9+
10+
High level goals for kinder v0.1 include:
11+
12+
- [ ] Provide a local test environment for kubeadm development
13+
- [x] Allow creation of nodes "ready for installing Kubernetes"
14+
- [x] Provide pre built “developer” workflows for kubedam init, join, reset
15+
- [x] init and init with phases
16+
- [x] join and join with phases
17+
- [x] init and join with automatic copy certs
18+
- [x] Provide pre built “developer” workflow for kubeadm upgrades
19+
- [x] reset
20+
- [x] Allow build of node-image variants
21+
- [x] add pre-loaded images to a node-image
22+
- [x] replace the kubeadm binary into a node-image
23+
- [x] add kubernetes binaries for a second kubernetes versions (target for upgrades)
24+
- [x] Allow test of kubeadm cluster variations
25+
- [x] external etcd
26+
- [x] kube-dns
27+
- [x] Provide "topology aware" wrappers for `docker exec` and `docker cp`
28+
- [x] Provide a way to add nodes to an existing cluster
29+
- [x] Add worker node
30+
- [x] Add control plane node (and reconfigure load balancer)
31+
- [x] Provide smoke test action
32+
- [ ] Provide E2E run action(s)
33+
34+
**Non**-Goals include:
35+
36+
- Replace or fork kind. kind is awesome and we are committed to help to improve it (see long term goals)
37+
- Supporting every possible use cases that can be build on top of kind as a library
38+
39+
Longer Term goals include:
40+
41+
- Simplify kubeadm development/local testing
42+
- Help new contributors on kubeadm development
43+
- Contribute to improving and testing "kind as a library"
44+
- Contribute back idea/code for new features in kind
45+
- Provide a home for use cases that are difficult to support in the main kind CLI

0 commit comments

Comments
 (0)