|
| 1 | +(storage-linstor)= |
| 2 | +# LINSTOR - `linstor` |
| 3 | + |
| 4 | +[LINSTOR](https://linbit.com/linstor/) is an open-source software-defined storage solution that is typically used to manage {abbr}`DRBD (Distributed Replicated Block Device)` replicated storage volumes. It provides both highly available and high performance volumes while focusing on operational simplicity. |
| 5 | + |
| 6 | +LINSTOR does not manage the underlying storage by itself, and instead relies on other components such as ZFS or LVM to provision block devices. These block devices are then replicated using [DRBD](https://linbit.com/drbd/) to provide fault tolerance and the ability to mount the volumes on any cluster node, regardless of their storage capabilities. Since volumes are replicated using the DRBD kernel module, the data path for the replication is kept entirely on kernel space, reducing its overhead when compared to solutions implemented in user space. |
| 7 | + |
| 8 | +## Terminology |
| 9 | + |
| 10 | +A LINSTOR cluster is composed of two main components: *controllers* and *satellites*. The LINSTOR controller manages the database and keeps track of the cluster state and configuration, while satellites provide storage and ability to mount volumes across the cluster. Clients interact only with the controller, which is responsible for orchestrating operations across satellites to fulfill the user's request. |
| 11 | + |
| 12 | +LINSTOR takes a somewhat object-oriented approach to its internal concepts. This manifests itself in the hierarchical nature of concepts and the fact that lower level objects can inherit properties from higher level ones. |
| 13 | + |
| 14 | +In LINSTOR, a *resource* is the representation of a storage unit that can be consumed by instances. A resource is most often a DRBD replicated block device, and in that case represents one replica of that device. Resources can be grouped into *resource definitions*, which define common properties that should be inherited by all their child resources. Similarly, *resource groups* define common properties that are applied to their child resource definitions. Resource groups also define placement rules that define how many replicas should be created for a given resource definition, which storage to use, how to spread the replicas among different availability zones, etc. The usual way to interact with LINSTOR is by defining a resource group with the desired properties and then *spawning* resources from it. |
| 15 | + |
| 16 | +LINSTOR also has the concept of a *storage pool*, which describes physical storage that can be consumed by LINSTOR to create volumes. A storage pool defines its back-end driver (such as LVM or ZFS), the cluster node in which it exists and properties that can be applied to either the storage pool itself or its back-end storage. |
| 17 | + |
| 18 | +## `linstor` driver in Incus |
| 19 | + |
| 20 | +```{warning} |
| 21 | +This storage driver is under active development and is not yet considered production ready. |
| 22 | +``` |
| 23 | + |
| 24 | +```{note} |
| 25 | +LINSTOR can only move and mount volumes across its satellite nodes. Therefore, to ensure that all Incus cluster members can access volumes, all Incus nodes must also be LINSTOR satellite nodes. In other words, each node running the `incus` service should also run an `linstor-satellite` service. |
| 26 | +
|
| 27 | +Note, however, that this does not mean that Incus nodes must also provide storage. It is still possible to use LINSTOR while using separated storage and compute nodes by deploying "diskless" satellites on Incus nodes. Diskless nodes do not provide storage, but are still able to mount DRBD devices and perform IO over the network. |
| 28 | +
|
| 29 | +Due to the way LINSTOR identifies its satellite nodes, it is also required that node names are consistent between Incus and LINSTOR. |
| 30 | +``` |
| 31 | + |
| 32 | +Unlike other storage drivers, this driver does not set up the storage system but assumes that you already have a LINSTOR cluster installed. The driver requires the {config:option}`server-miscellaneous:storage.linstor.controller_connection` option to be set to the endpoint of a LINSTOR controller that will be used by Incus. |
| 33 | + |
| 34 | +This driver also behaves differently than other drivers in that it can provide both remote and local storage. If a diskful replica of the volume is available on the node, reads and writes can be performed locally to reduce latency (although writes must be synchronously replicated across replicas, so network latency has an impact). At the same time, a diskless replica performs all IO over the network, enabling volumes to be mounted and used on any node regardless of its physical storage. These hybrid capabilities enable LINSTOR to provide low latency storage while retaining the flexibility of moving volumes across cluster nodes when needed. |
| 35 | + |
| 36 | +The `linstor` driver in Incus uses resource groups to manage and spawn resources. The following table describes the mapping between Incus and LINSTOR concepts: |
| 37 | + |
| 38 | +Incus concept | LINSTOR concept |
| 39 | +:-- | :-- |
| 40 | +Storage pool | Resource group |
| 41 | +Volume | Resource definition |
| 42 | +Snapshot | Snapshot |
| 43 | + |
| 44 | +Incus assumes that it has full control over the LINSTOR resource group. |
| 45 | +Therefore, you should never maintain any file system entities that are not owned by Incus in an Incus LINSTOR resource group, because Incus might delete them. |
| 46 | + |
| 47 | +### Limitations |
| 48 | + |
| 49 | +The `linstor` driver has the following limitations: |
| 50 | + |
| 51 | +Sharing custom volumes between instances |
| 52 | +: Custom storage volumes with {ref}`content type <storage-content-types>` `filesystem` can usually be shared between multiple instances different cluster members. |
| 53 | + However, because the LINSTOR driver "simulates" volumes with content type `filesystem` by putting a file system on top of an DRBD replicated device, custom storage volumes can only be assigned to a single instance at a time. |
| 54 | + |
| 55 | +Sharing the resource group between installations |
| 56 | +: Sharing the same LINSTOR resource group between multiple Incus installations is not supported. |
| 57 | + |
| 58 | +Renaming volumes |
| 59 | +: Due to the way LINSTOR operates, it is not possible to rename any of its objects. |
| 60 | + Therefore, it is currently not possible to rename Incus volumes created via the `linstor` driver. |
| 61 | + |
| 62 | +## Configuration options |
| 63 | + |
| 64 | +The following configuration options are available for storage pools that use the `linstor` driver and for storage volumes in these pools. |
| 65 | + |
| 66 | +(storage-linstor-pool-config)= |
| 67 | +### Storage pool configuration |
| 68 | + |
| 69 | +Key | Type | Default | Description |
| 70 | +:-- | :--- | :------ | :---------- |
| 71 | +`linstor.resource_group.name` | string | `incus` | Name of the LINSTOR resource group that will be used for the storage pool |
| 72 | +`linstor.resource_group.place_count` | int | 2 | Number of diskful replicas that should be create for resources in the resource group |
| 73 | +`linstor.resource_group.storage_pool` | string | - | The storage pool name in which resources should be placed on satellite nodes |
| 74 | + |
| 75 | +{{volume_configuration}} |
| 76 | + |
| 77 | +(storage-linstor-vol-config)= |
| 78 | +### Storage volume configuration |
| 79 | + |
| 80 | +Key | Type | Condition | Default | Description |
| 81 | +:-- | :--- | :-------- | :------ | :---------- |
| 82 | +`block.filesystem` | string | block-based volume with content type `filesystem` | same as `volume.block.filesystem` | {{block_filesystem}} |
| 83 | +`block.mount_options` | string | block-based volume with content type `filesystem` | same as `volume.block.mount_options` | Mount options for block-backed file system volumes |
| 84 | +`initial.gid` | int | custom volume with content type `filesystem` | same as `volume.initial.uid` or `0` | GID of the volume owner in the instance |
| 85 | +`initial.mode` | int | custom volume with content type `filesystem` | same as `volume.initial.mode` or `711` | Mode of the volume in the instance |
| 86 | +`initial.uid` | int | custom volume with content type `filesystem` | same as `volume.initial.gid` or `0` | UID of the volume owner in the instance |
| 87 | +`security.shared` | bool | custom block volume | same as `volume.security.shared` or `false` | Enable sharing the volume across multiple instances |
| 88 | +`security.shifted` | bool | custom volume | same as `volume.security.shifted` or `false` | {{enable_ID_shifting}} |
| 89 | +`security.unmapped` | bool | custom volume | same as `volume.security.unmapped` or `false` | Disable ID mapping for the volume |
| 90 | +`size` | string | | same as `volume.size` | Size/quota of the storage volume |
| 91 | +`snapshots.expiry` | string | custom volume | same as `volume.snapshots.expiry` | {{snapshot_expiry_format}} |
| 92 | +`snapshots.pattern` | string | custom volume | same as `volume.snapshots.pattern` or `snap%d` | {{snapshot_pattern_format}} [^*] |
| 93 | +`snapshots.schedule` | string | custom volume | same as `volume.snapshots.schedule` | {{snapshot_schedule_format}} |
| 94 | + |
| 95 | +[^*]: {{snapshot_pattern_detail}} |
0 commit comments