Skip to content

Commit d01f2cb

Browse files
cspavlouk8s-ci-robot
cspavlou
authored and
k8s-ci-robot
committed
Amend the kfctl_existing_arrikto doc (kubeflow#1056)
Add instructions for installing MetalLB Add `Troubleshooting` section Add architecture figure Change the kfctl_existing_arrikto description in the Overview page Signed-off-by: Christos S. Pavlou <[email protected]>
1 parent 85ba5bb commit d01f2cb

File tree

3 files changed

+226
-15
lines changed

3 files changed

+226
-15
lines changed

‎content/docs/started/k8s/kfctl-existing-arrikto.md

+223-13
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,157 @@
11
+++
2-
title = "Kubeflow Deployment with kfctl_existing_arrikto"
3-
description = "Instructions for installing Kubeflow with kfctl_existing_arrikto config"
2+
title = "Multi-user, auth-enabled Kubeflow with kfctl_existing_arrikto"
3+
description = "Instructions for installing Kubeflow with kfctl_existing_arrikto.yaml config"
44
weight = 4
55
+++
66

7-
This deployment uses [Dex](https://github.com/dexidp/dex) and [Istio](https://istio.io/) for vendor-neutral authentication.
7+
Follow these instructions if you want to install Kubeflow on an existing Kubernetes cluster.
88

9-
**Maintainer and supporter: Arrikto**
9+
This installation of Kubeflow is maintained by
10+
[Arrikto](https://www.arrikto.com/), it is geared towards existing Kubernetes
11+
clusters and does not depend on any cloud-specific feature.
12+
13+
In this reference architecture, we use [Dex](https://github.com/dexidp/dex) and
14+
[Istio](https://istio.io/) for vendor-neutral authentication.
15+
16+
This deployment works well for on-prem installations, where companies/organizations need LDAP/AD integration for multi-user authentication, and they don't want to depend on any cloud-specific feature.
17+
18+
![kfctl_existing_arrikto_architecture](../../kfctl_existing_arrikto-architecture.svg)
19+
20+
Read the relevant [article](https://journal.arrikto.com/kubeflow-authentication-with-istio-dex-5eafdfac4782) for more info about this architecture.
1021

11-
![platform existing architecture](https://i.imgur.com/OlaN73j.png)
1222

1323
### Prerequisites
14-
- Kubernetes Cluster with LoadBalancer support. Refer [Load Balancer](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/) guide.
1524

25+
You need a Kubernetes Cluster with LoadBalancer support.
1626

17-
### Deploy Kubeflow
27+
If you don't have support for LoadBalancer on your cluster, please follow the instructions below to deploy MetalLB in Layer 2 mode. (You can read more about Layer 2 mode in the [MetalLB docs](https://metallb.universe.tf/configuration/#layer-2-configuration).)
28+
29+
<details>
30+
31+
<summary>MetalLB deployment</summary>
32+
33+
**Deploy MetalLB:**
34+
35+
1. Apply the manifest:
36+
37+
```
38+
kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.1/manifests/metallb.yaml
39+
```
40+
41+
1. Allocate a pool of addresses on your local network for MetalLB to use. You
42+
need at least one address for the Istio Gateway. This example assumes
43+
addresses `10.0.0.100-10.0.0.110`. *You must modify these addresses based on
44+
your environment*.
45+
46+
```
47+
cat <<EOF | kubectl apply -f -
48+
apiVersion: v1
49+
kind: ConfigMap
50+
metadata:
51+
namespace: metallb-system
52+
name: config
53+
data:
54+
config: |
55+
address-pools:
56+
- name: default
57+
protocol: layer2
58+
addresses:
59+
- 10.0.0.100-10.0.0.110
60+
EOF
61+
```
62+
63+
64+
**Ensure that MetalLB works as expected (optional):**
65+
66+
1. Create a dummy service:
67+
68+
```
69+
kubectl create service loadbalancer nginx --tcp=80:80
70+
service/nginx created
71+
```
72+
73+
1. Ensure that MetalLB has allocated an IP address for the service:
74+
75+
```
76+
kubectl describe service nginx
77+
...
78+
Events:
79+
Type Reason Age From Message
80+
---- ------ ---- ---- -------
81+
Normal IPAllocated 69s metallb-controller Assigned IP "10.0.0.101"
82+
```
83+
84+
1. Check the corresponding MetalLB logs:
85+
86+
```
87+
kubectl logs -n metallb-system -l component=controller
88+
...
89+
{"caller":"service.go:98","event":"ipAllocated","ip":"10.0.0.101","msg":"IP address assigned by controller","service":"default/nginx","ts":"2019-08-09T15:12:09.376779263Z"}
90+
```
91+
92+
1. Create a pod that will be exposed with the service:
93+
94+
```
95+
kubectl run nginx --image nginx --restart=Never -l app=nginx
96+
pod/nginx created
97+
```
98+
99+
1. Ensure that MetalLB has assigned a node to announce the allocated IP address:
100+
101+
```
102+
kubectl describe service nginx
103+
...
104+
Events:
105+
Type Reason Age From Message
106+
---- ------ ---- ---- -------
107+
Normal nodeAssigned 4s metallb-speaker announcing from node "node-2"
108+
```
109+
110+
1. Check the corresponding MetalLB logs:
111+
112+
```
113+
kubectl logs -n metallb-system -l component=speaker
114+
...
115+
{"caller":"main.go:246","event":"serviceAnnounced","ip":"10.0.0.101","msg":"service has IP, announcing","pool":"default","protocol":"layer2","service":"default/nginx","ts":"2019-08-09T15:14:02.433876894Z"}
116+
```
117+
118+
1. Check that MetalLB responds to ARP requests for the allocated IP address:
119+
120+
```
121+
arping -I eth0 10.0.0.101
122+
...
123+
ARPING 10.0.0.101 from 10.0.0.204 eth0
124+
Unicast reply from 10.0.0.101 [6A:13:5A:D2:65:CB] 2.619ms
125+
```
126+
127+
1. Check the corresponding MetalLB logs:
128+
129+
```
130+
kubectl logs -n metallb-system -l component=speaker
131+
...
132+
{"caller":"arp.go:102","interface":"eth0,"ip":"10.0.0.101","msg":"got ARP request for service IP, sending response","responseMAC":"6a:13:5a:d2:65:cb","senderIP":"10.0.0.204","senderMAC":"9a:1f:7c:95:ca:dc","ts":"2019-08-09T15:14:52.912056021Z"}
133+
```
134+
135+
1. Verify that everything works as expected:
136+
137+
```
138+
curl http://10.0.0.101
139+
...
140+
<p><em>Thank you for using nginx.</em></p>
141+
...
142+
```
143+
144+
1. Clean up:
145+
146+
```
147+
kubectl delete service nginx
148+
kubectl delete pod nginx
149+
```
150+
151+
</details>
18152
19-
If you are deploying Kubeflow on a multi-node cluster, you can follow this [guide](/docs/use-cases/kubeflow-on-multinode-cluster) to set up your system to use a remote NFS filesystem in cluster nodes.
153+
154+
### Deploy Kubeflow
20155
21156
Follow these steps to deploy Kubeflow:
22157
@@ -33,7 +168,7 @@ Follow these steps to deploy Kubeflow:
33168
# Add kfctl to PATH, to make the kfctl binary easier to use.
34169
export PATH=$PATH:"<path to kfctl>"
35170
export KFAPP="<your choice of application directory name>"
36-
export CONFIG="https://raw.githubusercontent.com/kubeflow/kubeflow/master/bootstrap/config/kfctl_existing_arrikto.0.6.yaml"
171+
export CONFIG="https://raw.githubusercontent.com/kubeflow/kubeflow/master/bootstrap/config/kfctl_existing_arrikto.yaml"
37172
38173
# Specify credentials for the default user.
39174
export KUBEFLOW_USER_EMAIL="[email protected]"
@@ -58,16 +193,16 @@ kfctl apply all -V
58193

59194
#### Log in as a static user
60195

61-
After deploying Kubeflow, the Kubeflow Dashboard is available at the Istio Gateway IP.
196+
After deploying Kubeflow, the Kubeflow dashboard is available at the Istio Gateway IP.
62197
To get the Istio Gateway IP, run:
63198

64199
```bash
65200
kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
66201
```
67202

68-
Get the IP and open it in a browser: `https://<ip>/`.
203+
Get the IP and open it in a browser: `https://<LoadBalancerIP address>/`.
69204

70-
Enter the credentials you specified in `KUBEFLOW_USER_EMAIL`, `KUBEFLOW_PASSWORD` and access the Kubeflow Dashboard!
205+
Enter the credentials you specified in `KUBEFLOW_USER_EMAIL`, `KUBEFLOW_PASSWORD` and access the Kubeflow dashboard!
71206

72207
#### Add static users for basic auth
73208

@@ -420,6 +555,81 @@ This section focuses on setting up Dex to authenticate with an existing LDAP dat
420555
* Add or update a label on the PodTemplate.
421556
* Save the deployment to trigger a rolling update.
422557
558+
### Troubleshooting
559+
560+
If the Kubeflow dashboard is not available at `https://<LoadBalancerIP address>` ensure that:
561+
562+
1. the LoadBalancer service for Istio has obtained an external IP, for example:
563+
564+
```
565+
kubectl get services -n istio-system istio-ingressgateway -o yaml
566+
...
567+
status:
568+
loadBalancer:
569+
ingress:
570+
- ip: 10.0.0.100
571+
```
572+
573+
If not, then probably there is a misconfiguration of MetalLB.
574+
575+
1. the virtual services have been created:
576+
577+
```
578+
kubectl get virtualservices -n kubeflow
579+
kubectl get virtualservices -n kubeflow centraldashboard -o yaml
580+
```
581+
582+
If not, then kfctl has aborted for some reason, and not completed successfully.
583+
584+
1. OIDC auth service redirects you to Dex:
585+
586+
```
587+
curl -k https://<LoadBalancerIP address>/ -v
588+
...
589+
< HTTP/2 302
590+
< content-type: text/html; charset=utf-8
591+
< location:
592+
/dex/auth?client_id=kubeflow-authservice-oidc&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=openid+profile+email+groups&state=vSCMnJ2D
593+
< date: Fri, 09 Aug 2019 14:33:21 GMT
594+
< content-length: 181
595+
< x-envoy-upstream-service-time: 0
596+
< server: istio-envoy
597+
```
598+
599+
Please join the [Kubeflow Slack](https://kubeflow.slack.com/join/shared_invite/enQtNDg5MTM4NTQyNjczLWUyZGI1ZmExZWExYWY4YzlkOWI4NjljNjJhZjhjMjEwNGFjNmVkNjg2NTg4M2I0ZTM5NDExZWI5YTIyMzVmNzM) to report any issues, request help, and give us feedback on this config.
600+
601+
Some additional debugging information:
602+
603+
OIDC Service logs:
604+
```bash
605+
kubectl logs -n istio-system -l app=authservice
606+
```
607+
Dex logs:
608+
```bash
609+
kubectl logs -n kubeflow -l app=dex
610+
```
611+
Istio ingress-gateway logs:
612+
```bash
613+
kubectl logs -n istio-system -l istio=ingressgateway
614+
```
615+
616+
Istio ingressgateway service:
617+
```bash
618+
kubectl get service -n istio-system istio-ingressgateway -o yaml
619+
```
620+
621+
MetalLB logs:
622+
```bash
623+
kubectl logs -n metallb-system -l component=speaker
624+
...
625+
{"caller":"arp.go:102","interface":"br100","ip":"10.0.0.100","msg":"got ARP request for service IP, sending response","responseMAC":"62:41:bd:5f:cc:0d","senderIP":"10.0.0.204","senderMAC":"9a:1f:7c:95:ca:dc","ts":"2019-07-31T13:19:19.7082836Z"}
626+
```
627+
```bash
628+
kubectl logs -n metallb-system -l component=controller
629+
...
630+
{"caller":"service.go:98","event":"ipAllocated","ip":"10.0.0.100","msg":"IP address assigned by controller","service":"istio-system/istio-ingressgateway","ts":"2019-07-31T12:17:46.234638607Z"}
631+
```
632+
423633
### Delete Kubeflow
424634
425635
Run the following commands to delete your deployment and reclaim all resources:
@@ -446,7 +656,7 @@ following:
446656
* **k8s** - all resources that run on Kubernetes.
447657
* **all** - platform and Kubernetes resources.
448658
449-
#### App layout
659+
### App layout
450660
451661
Your Kubeflow app directory contains the following files and directories:
452662

‎content/docs/started/k8s/overview.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -60,8 +60,8 @@ This section includes the deployment solutions that are supported by specific ve
6060
</thead>
6161
<tbody>
6262
<tr>
63-
<td>existing_arrikto.yaml</td>
64-
<td> This deployment uses Dex and Istio for vendor-neutral authentication. <br />Follow instructions: <a href="/docs/started/k8s/kfctl-existing-arrikto/">Kubeflow Deployment with existing_arrikto</a></td>
63+
<td>kfctl_existing_arrikto.yaml</td>
64+
<td> This config creates a Kubeflow deployment with all its core components, and uses Dex and Istio for vendor-neutral authentication. <br />Follow instructions: <a href="/docs/started/k8s/kfctl-existing-arrikto/">Multi-user, auth-enabled Kubeflow with kfctl_existing_arrikto</a></td>
6565
<td><a href="https://www.arrikto.com/">Arrikto</a></td>
6666
</tr>
6767
</tbody>

‎content/docs/started/kfctl_existing_arrikto-architecture.svg

+1
Loading

0 commit comments

Comments
 (0)