csi: revist rbac rules, add Snapshotter sidecar and roles

* Most of the rules are now removed and will not be part of upcoming k8s
releases. Going forward drivers have to install themself.
* Add `csi-snapshotter` sidecar, needed to handle Volume snapshots
* This also adds the necessary roles and bindings needed for the
csi-snapshotter sidecar.

Author: fatih

CHANGELOG.md

@@ -1,7 +1,13 @@
 ## unreleased
 
-* Add Snapshots functionality
+* Add CSI Snapshots functionality
   [[GH-103]](https://github.com/digitalocean/csi-digitalocean/pull/103)
+* Add csi-snapshotter sidecars and associated RBAC rules
+  [[GH-104]](https://github.com/digitalocean/csi-digitalocean/pull/104)
+* Revisit existing RBAC rules for the attacher, provisioner and
+  driver-registrar. We no longer use the system cluster-role bindings as those
+  will be deleted in v1.13
+  [[GH-104]](https://github.com/digitalocean/csi-digitalocean/pull/104)
 * Fix inconsistent usage of the driver name
   [[GH-100]](https://github.com/digitalocean/csi-digitalocean/pull/100)
 * Use publish_info in ControllerPublishVolume for storing and accessing the

Makefile

@@ -47,7 +47,7 @@ test:
 test-integration:
 
 	@echo "==> Started integration tests"
-	@env GOCACHE=off go test -v -tags integration ./test/...
+	@env go test -v -tags integration ./test/...
 
 
 .PHONY: build

deploy/kubernetes/releases/csi-digitalocean-dev.yaml

@@ -118,6 +118,20 @@ provisioner: dobs.csi.digitalocean.com
 
 ---
 
+# NOTE(arslan): this will probably fail , because the CRD is created via the
+# csi-snapshotter sidecar, that is part of the csi-do-controller statefulset.
+# We need to create this seperately.
+kind: VolumeSnapshotClass
+apiVersion: snapshot.storage.k8s.io/v1alpha1
+metadata:
+  name: do-block-storage
+  namespace: kube-system
+  annotations:
+    snapshot.storage.kubernetes.io/is-default-class: "true"
+snapshotter: dobs.csi.digitalocean.com
+
+---
+
 ##############################################
 ###########                       ############
 ###########   Controller plugin   ############
@@ -165,6 +179,18 @@ spec:
           volumeMounts:
             - name: socket-dir
               mountPath: /var/lib/csi/sockets/pluginproxy/
+        - name: csi-snapshotter
+          image: quay.io/k8scsi/csi-snapshotter:v0.4.1
+          args:
+            - "--connection-timeout=15s"
+            - "--csi-address=$(ADDRESS)"
+          env:
+            - name: ADDRESS
+              value: /var/lib/csi/sockets/pluginproxy/csi.sock
+          imagePullPolicy: Always
+          volumeMounts:
+            - name: socket-dir
+              mountPath: /var/lib/csi/sockets/pluginproxy/
         - name: csi-do-plugin
           image: digitalocean/do-csi-plugin:dev
           args :
@@ -190,46 +216,140 @@ spec:
           emptyDir: {}
 ---
 
-apiVersion: v1
 kind: ServiceAccount
+apiVersion: v1
 metadata:
   name: csi-do-controller-sa
   namespace: kube-system
 
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: csi-do-provisioner-role
+rules:
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "list"]
+  - apiGroups: [""]
+    resources: ["persistentvolumes"]
+    verbs: ["get", "list", "watch", "create", "delete"]
+  - apiGroups: [""]
+    resources: ["persistentvolumeclaims"]
+    verbs: ["get", "list", "watch", "update"]
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["storageclasses"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["events"]
+    verbs: ["list", "watch", "create", "update", "patch"]
+  - apiGroups: ["snapshot.storage.k8s.io"]
+    resources: ["volumesnapshots"]
+    verbs: ["get", "list"]
+  - apiGroups: ["snapshot.storage.k8s.io"]
+    resources: ["volumesnapshotcontents"]
+    verbs: ["get", "list"]
+
 ---
 
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
-  name: csi-do-controller-provisioner-binding
-  namespace: kube-system
+  name: csi-do-provisioner-binding
 subjects:
   - kind: ServiceAccount
     name: csi-do-controller-sa
     namespace: kube-system
 roleRef:
   kind: ClusterRole
-  name: system:csi-external-provisioner
+  name: csi-do-provisioner-role
   apiGroup: rbac.authorization.k8s.io
 
 ---
+# Attacher must be able to work with PVs, nodes and VolumeAttachments
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: csi-do-attacher-role
+rules:
+  - apiGroups: [""]
+    resources: ["persistentvolumes"]
+    verbs: ["get", "list", "watch", "update"]
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["csi.storage.k8s.io"]
+    resources: ["csinodeinfos"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["volumeattachments"]
+    verbs: ["get", "list", "watch", "update"]
 
+---
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
-  name: csi-do-controller-attacher-binding
-  namespace: kube-system
+  name: csi-do-attacher-binding
 subjects:
   - kind: ServiceAccount
     name: csi-do-controller-sa
     namespace: kube-system
 roleRef:
   kind: ClusterRole
-  name: system:csi-external-attacher
+  name: csi-do-attacher-role
   apiGroup: rbac.authorization.k8s.io
 
 ---
 
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: csi-do-snapshotter-role
+rules:
+  - apiGroups: [""]
+    resources: ["persistentvolumes"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["persistentvolumeclaims"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["storageclasses"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["events"]
+    verbs: ["list", "watch", "create", "update", "patch"]
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "list"]
+  - apiGroups: ["snapshot.storage.k8s.io"]
+    resources: ["volumesnapshotclasses"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["snapshot.storage.k8s.io"]
+    resources: ["volumesnapshotcontents"]
+    verbs: ["create", "get", "list", "watch", "update", "delete"]
+  - apiGroups: ["snapshot.storage.k8s.io"]
+    resources: ["volumesnapshots"]
+    verbs: ["get", "list", "watch", "update"]
+  - apiGroups: ["apiextensions.k8s.io"]
+    resources: ["customresourcedefinitions"]
+    verbs: ["create", "list", "watch", "delete"]
+
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: csi-do-snapshotter-binding
+subjects:
+  - kind: ServiceAccount
+    name: csi-do-controller-sa
+    namespace: kube-system
+roleRef:
+  kind: ClusterRole
+  name: csi-do-snapshotter-role
+  apiGroup: rbac.authorization.k8s.io
+
+
+
 
 ########################################
 ###########                 ############
@@ -336,11 +456,22 @@ metadata:
 
 ---
 
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: csi-do-driver-registrar-role
+  namespace: kube-system
+rules:
+  - apiGroups: [""]
+    resources: ["events"]
+    verbs: ["get", "list", "watch", "create", "update", "patch"]
+
+---
+
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: csi-do-driver-registrar-binding
-  namespace: kube-system
 subjects:
   - kind: ServiceAccount
     name: csi-do-node-sa
@@ -350,19 +481,3 @@ roleRef:
   name: csi-do-driver-registrar-role
   apiGroup: rbac.authorization.k8s.io          
 
-
----
-
-kind: ClusterRole
-apiVersion: rbac.authorization.k8s.io/v1
-metadata:
-  name: csi-do-driver-registrar-role
-  namespace: kube-system
-rules:
-  - apiGroups: [""]
-    resources: ["nodes"]
-    verbs: ["get", "list", "update"]
-  - apiGroups: [""]
-    resources: ["events"]
-    verbs: ["list", "watch", "create", "update", "patch"]
-

examples/kubernetes/snapshot/README.md

@@ -0,0 +1,59 @@
+# Creating a snapshot from an exiting volume and restore it back
+
+Note that we assume you correctly installed the csi-digitalocean driver, and
+it's up and running. 
+
+
+1. Create a `pvc`:
+
+
+```
+$ kubectl create -f pvc.yaml
+```
+
+2. Create a `snapshot` from the previous `pvc`:
+
+
+```
+$ kubectl create -f snapshot.yaml
+```
+
+At this point you should have a volume and a snapshot originating from that
+volume. You can observe the state of your pvc's and snapshots with the
+following command:
+
+
+```
+$ kubectl get pvc && kubectl get pv && kubectl get volumesnapshot
+```
+
+
+3. Restore from a `snapshot`:
+
+To restore from a given snapshot, you need to create a new `pvc` that refers to
+the snapshot:
+
+
+```
+$ kubectl create -f restore.yaml
+```
+
+This will create a new `pvc` that you can use with your applications.
+
+4. Cleanup your resources:
+
+Make sure to delete your test resources:
+
+```
+$ kubectl delete -f pvc.yaml 
+$ kubectl delete -f restore.yaml 
+$ kubectl delete -f snapshot.yaml 
+```
+
+---
+
+To understand how snapshotting works, please read the official blog
+announcement with examples:
+https://kubernetes.io/blog/2018/10/09/introducing-volume-snapshot-alpha-for-kubernetes/
+
+

examples/kubernetes/snapshot/pvc.yaml

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: csi-do-test-pvc
+spec:
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 5Gi
+  storageClassName: do-block-storage

examples/kubernetes/snapshot/restore.yaml

@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: csi-do-test-pvc-restore
+spec:
+  dataSource:
+    name: csi-do-test-snapshot 
+    kind: VolumeSnapshot
+    apiGroup: snapshot.storage.k8s.io
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 5Gi

examples/kubernetes/snapshot/snapshot.yaml

@@ -0,0 +1,8 @@
+apiVersion: snapshot.storage.k8s.io/v1alpha1
+kind: VolumeSnapshot
+metadata:
+  name: csi-do-test-snapshot 
+spec:
+  source:
+    name: csi-do-test-pvc
+    kind: PersistentVolumeClaim