Consider support for RAID in local provisioner

[schallert]

For context, this issue stems from a Slack conversation with @msau42 that we wanted to capture here.

It would be awesome if the local static provisioner could support RAID'ing devices together.

Currently, if a user wants to RAID local disks together they must do so manually before presenting the provisioner with a filesystem or block device. Some ways this can currently be achieved include but are not limited to:

1. Constructing the RAID volume at node provisioning time and passing formatting it to the provisioner as an FS, or as block device.
2. Using an init container to the local static provisioner that RAIDs disks together, either as block devices or after dismantling FS's (this was @msau42's idea).
3. Using some other process to accomplish RAID'ing/formatting, and then labeling nodes that have been set up and using NodeAffinity to only run the local provisioner on those nodes.

Option (1) is unfortunately not suitable for managed Kubernetes platforms where the only knobs the user have may be how many local FS/block volumes they want, and not how they're formatted. The other options have the benefit of potentially being compatible with a managed platform, however they require manual intervention from the user. Many of these goals may be accomplished with the LVM provisioner, but it sounds like that might be far off enough to warrant work in the meantime.

If it aligns with the goals of the local static provisioner, it would be really helpful if the provisioner could handle being presented with block devices and RAID'ing (and potentially formatting) them before creating the PV. To relax the constraint of requiring block devices maybe this same functionality could eventually include deconstructing FS's as well. I noticed "Local block devices as a volume source, with partitioning and fs formatting" is on the roadmap so maybe this could fit in there?

I wanted to start this issue as a place to discuss some of these issues. If we can reach consensus on how to proceed I'd be happy to help contribute.

[msau42]

cc @gnufied who has also been working on an operator

In the past, we've wanted to keep a clear separation between environment-specific prep and the general PV lifecycle management, but I can see value in providing some optional helpers if it's beneficial to many users (and I have seen many requests for supporting raid setup). I would still like to keep it separate from the actual provisioner process so that we don't complicate the logic there (and also potentially require installing mdadm in the container image for everyone). So either options 2) or 3) sounds good to me.

I think the biggest question to figure out is how will the disk names be passed in? List every disk? Pattern match? Nodes can have different number/names for disks.

[cofyc]

I'd prefer the second option.

I think the biggest question to figure out is how will the disk names be passed in? List every disk? Pattern match? Nodes can have different number/names for disks.

Yes, if we want to support local volume prep in various environments, the configuration must be flexible.

This is my proposal, what do you think?

class "local" {
    dir = "/mnt/raid-local"
    # mode defaults to "filesystem"
    # mode = "filesytem"
}

class "local-device" {
    dir = "/mnt/raid-local-device"
    mode = "block"
}

#
# For all gke-demo-default-pool-* nodes, we combine all local SSDs into one
# raid0 disk and format/mount it into "local" class directory.
#
node "gke-demo-default-pool-*" {
    raid0 md0 {
        class = "local"
        disks = ["/dev/disk/by-id/google-local-ssd-*"]
    }   
}

#
# For all gke-demo-another-pool-* nodes, we combine two local SSDs into one
# raid0 disk and link the disk to "local-device" class directory.
#
node "gke-demo-another-pool-*" {
    raid0 md0 {
        class = "local-device"
        disks = ["/dev/disk/by-id/google-local-ssd-0", "/dev/disks/by-id/google-local-ssd-1"]
    }   
    raid0 md1 {
        class = "local-device"
        disks = ["/dev/disk/by-id/google-local-ssd-2", "/dev/disks/by-id/google-local-ssd-3"]
    }   
    raid0 md2 {
        class = "local-device"
        disks = ["/dev/disk/by-id/google-local-ssd-4", "/dev/disks/by-id/google-local-ssd-5"]
    }   
}

The configuration language is HCl which is used by terraform.

[gnufied]

We have been working on a local-storage operator that uses following API to allow user to specify disks that can be used by local-storage-provisioner - https://github.com/openshift/local-storage-operator/blob/master/pkg/apis/local/v1alpha1/types.go#L54 (example: https://github.com/openshift/local-storage-operator/blob/master/examples/olm/create-cr.yaml )

@cofyc An earlier version of API we proposed for local-storage-operator allowed specifying wildcards and regexp, but at least we quickly realized that we may have to allow users to specify exclusion mechanism (like don't use this disk but use others that match this regex). It might be worth starting small and keeping surface area of API small and gather user feedback and then iterate on design. If we allow wildcards/regexes from v1 then, it will be hard to rollback on them.

I agree with @msau42 that separating disk preparation and general PV lifecycle managment is a good idea and since kubelet itself is capable of formatting disks, this provisioner does not need to do that (at least for non-RAID volumes).

[cofyc]

CRD is more flexible and Kubernetes-native way to configure, it seems a good idea to have an operator to do these tasks (option 3). Had a discussion with @gnufied, we can add raid support in local-storage-operator. What do you think?

[cofyc]

Another simpler solution is to annotate the node to tell provisioner or sidecar of it to combine the disks before mounting (filesystem) or symlinking the combined disk to discovery directory.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[gregwebs]

I have implemented solution 2 for GKE. You can choose between LVM or RAID. It assumes that you want to combine all the available disks together (which is not necessarily a correct assumption for how everyone does K8s): https://github.com/pingcap/tidb-operator/blob/master/manifests/gke/local-ssd-provision/local-ssd-provision.yaml

[msau42]

/remove-lifecycle stale

Awesome! We can consider adding the script here in some addons folder if you think that would be beneficial.

[gregwebs]

To be more generally useful you would probably want to do disk combining based on some node pool labeling scheme or other metadata available at startup.
This solution also causes a failure when the node restarts due to brittleness in GKE startup scripts. This has been reported in multiple places. When reporting this to GKE support they told me that un-mounting disks is not supported at this time and they don't care to make this situation more transparent in their documentation.

[nyurik]

@gregwebs this is awesome, I have been looking for something like this for a very long time! It would be absolutely awesome to have this as a ready-to-use component rather than a large code copy/paste. A few notes:

• GCP will soon (hopefully) introduce NVMEs -- gcloud alpha already supports --local-ssd-volumes parameter. They will be listed as /dev/nvme* rather than ssd*. Also, it seems they can be created without being formatted with format=block.
• for some reason mdadm kept raising 141 exit code, despite seemingly completing successfully. Have you had that issue?
• could that code be packaged into a published docker hub image? I already created nyurik/kuberaid (uses a very simple script to force format), but yours is far better and more thorough.

Thank you for you awesome work on this!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale