|
| 1 | +# CloudPrem |
| 2 | + |
| 3 | +   |
| 4 | + |
| 5 | +## Using the Datadog Helm repository |
| 6 | + |
| 7 | +Add and update the Datadog Helm repository to your Helm repositories: |
| 8 | + |
| 9 | +```sh |
| 10 | +helm repo add datadog https://helm.datadoghq.com |
| 11 | +helm repo update |
| 12 | +``` |
| 13 | + |
| 14 | +## Prerequisites |
| 15 | + |
| 16 | +- AWS account |
| 17 | +- Kubernetes `1.25+` ([EKS](https://aws.amazon.com/eks/) preferred) |
| 18 | +- [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller) |
| 19 | +- PostgreSQL database ([RDS](https://aws.amazon.com/rds/) preferred) |
| 20 | +- S3 bucket |
| 21 | + |
| 22 | +## Quick start |
| 23 | + |
| 24 | +### Creating the Kubernetes namespace |
| 25 | + |
| 26 | +```sh |
| 27 | +kubectl create namespace <namespace name> |
| 28 | +``` |
| 29 | + |
| 30 | +### Storing the PostgreSQL database connection string as a Kubernetes secret |
| 31 | + |
| 32 | +```sh |
| 33 | +kubectl create secret generic <secret name> --from-literal QW_METASTORE_URI=postgres://<username>:<password>@<endpoint>:<port>/<database> -n <namespace name> |
| 34 | +``` |
| 35 | + |
| 36 | +### Customizing the Helm chart |
| 37 | + |
| 38 | +Create a `datadog-values.yaml` file to override the default values with your custom configuration. This is where you define environment-specific settings such as the image tag, AWS account ID, service account, ingress setup, resource requests and limits, and more. |
| 39 | +Any parameters not explicitly overridden in `datadog-values.yaml` will fall back to the defaults defined in the chart’s `values.yaml`. Here is an example of a `datadog-values.yaml` file with such overrides: |
| 40 | + |
| 41 | +```yaml |
| 42 | +aws: |
| 43 | + accountId: "123456789012" |
| 44 | + |
| 45 | +# Environment variables |
| 46 | +# Any environment variables defined here will be available to all pods in the deployment |
| 47 | +environment: |
| 48 | + AWS_REGION: us-east-1 |
| 49 | + |
| 50 | +# Service account configuration |
| 51 | +# If `serviceAccount.create` is set to `true`, a service account will be created with the specified name. |
| 52 | +# The service account will be annotated with the IAM role ARN if `aws.accountId` and serviceAccount.eksRoleName` are set. |
| 53 | +# Additional annotations can be added using serviceAccount.extraAnnotations. |
| 54 | +serviceAccount: |
| 55 | + create: true |
| 56 | + name: cloudprem |
| 57 | + # The name of the IAM role to use for the service account. If set, the following annotations will be added to the service account: |
| 58 | + # - eks.amazonaws.com/role-arn: arn:aws:iam::<aws.accountId>:role/<serviceAccount.eksRoleName> |
| 59 | + # - eks.amazonaws.com/sts-regional-endpoints: "true" |
| 60 | + eksRoleName: cloudprem |
| 61 | + extraAnnotations: {} |
| 62 | + |
| 63 | +# CloudPrem node configuration |
| 64 | +config: |
| 65 | + # The root URI where index data will be stored. This should be an S3 path. |
| 66 | + # All indexes created in CloudPrem will be stored under this location. |
| 67 | + default_index_root_uri: s3://<bucket name>/indexes |
| 68 | + |
| 69 | +# Ingress configuration |
| 70 | +# The chart supports two ingress configurations: |
| 71 | +# 1. A public ingress for external access via the internet that will be used exclusively by Datadog's controle plane and query service. |
| 72 | +# 2. An internal ingress for access within the VPC |
| 73 | +# |
| 74 | +# Both ingresses will provision Application Load Balancers (ALBs) in AWS. |
| 75 | +# The public ingress ALB will be created in public subnets. |
| 76 | +# The internal ingress ALB will be created in private subnets. |
| 77 | +# |
| 78 | +# Additional annotations can be added to customize the ALB behavior. |
| 79 | +ingress: |
| 80 | + # The public ingress is configured to only accept TLS traffic and requires mutual TLS (mTLS) authentication. |
| 81 | + # Datadog's control plane and query service authenticate themselves using client certificates, |
| 82 | + # ensuring that only authorized Datadog services can access CloudPrem nodes through the public ingress. |
| 83 | + public: |
| 84 | + enabled: true |
| 85 | + name: cloudprem-public |
| 86 | + host: cloudprem.acme.corp |
| 87 | + extraAnnotations: |
| 88 | + alb.ingress.kubernetes.io/load-balancer-name: cloudprem-public |
| 89 | + |
| 90 | + # The internal ingress is used by Datadog agents and other collectors running outside |
| 91 | + # the Kubernetes cluster to send their logs to CloudPrem. |
| 92 | + internal: |
| 93 | + enabled: true |
| 94 | + name: cloudprem-internal |
| 95 | + host: cloudprem.acme.internal |
| 96 | + extraAnnotations: |
| 97 | + alb.ingress.kubernetes.io/load-balancer-name: cloudprem-internal |
| 98 | + |
| 99 | +# Metastore configuration |
| 100 | +# The metastore is responsible for storing and managing index metadata. |
| 101 | +# It requires a PostgreSQL database connection string to be provided via a Kubernetes secret. |
| 102 | +# The secret should contain a key named `QW_METASTORE_URI` with a value in the format: |
| 103 | +# postgresql://<username>:<password>@<host>:<port>/<database> |
| 104 | +# |
| 105 | +# The metastore connection string is mounted into the pods using extraEnvFrom to reference the secret. |
| 106 | +metastore: |
| 107 | + extraEnvFrom: |
| 108 | + - secretRef: |
| 109 | + name: cloudprem-metastore-uri |
| 110 | + |
| 111 | +# Indexer configuration |
| 112 | +# The indexer is responsible for processing and indexing incoming data it receives data from various sources (e.g., Datadog agents, log collectors) |
| 113 | +# and transforms it into searchable files called "splits" stored in S3. |
| 114 | +# |
| 115 | +# The indexer is horizontally scalable - you can increase `replicaCount` to handle higher indexing throughput. |
| 116 | +# Resource requests and limits should be tuned based on your indexing workload. |
| 117 | +# |
| 118 | +# The default values are suitable for moderate indexing loads of up to 20MB/s per indexer pod. |
| 119 | +indexer: |
| 120 | + replicaCount: 2 |
| 121 | + |
| 122 | + resources: |
| 123 | + requests: |
| 124 | + cpu: "4" |
| 125 | + memory: "8Gi" |
| 126 | + limits: |
| 127 | + cpu: "4" |
| 128 | + memory: "8Gi" |
| 129 | + |
| 130 | +# Searcher configuration |
| 131 | +# The searcher is responsible for executing search queries against the indexed data stored in S3. |
| 132 | +# It handles search requests from Datadog's query service and returns matching results. |
| 133 | +# |
| 134 | +# The searcher is horizontally scalable - you can increase `replicaCount` to handle more concurrent searches. |
| 135 | +# Resource requirements for searchers are highly workload-dependent and should be determined empirically. |
| 136 | +# Key factors that impact searcher performance include: |
| 137 | +# - Query complexity (e.g., number of terms, use of wildcards or regex) |
| 138 | +# - Query concurrency (number of simultaneous searches) |
| 139 | +# - Amount of data scanned per query |
| 140 | +# - Data access patterns (cache hit rates) |
| 141 | +# |
| 142 | +# Memory is particularly important for searchers as they cache frequently accessed index data in memory. |
| 143 | +# Monitor searcher metrics and adjust resources based on observed performance and workload characteristics. |
| 144 | +searcher: |
| 145 | + replicaCount: 2 |
| 146 | + |
| 147 | + resources: |
| 148 | + requests: |
| 149 | + cpu: "4" |
| 150 | + memory: "16Gi" |
| 151 | + limits: |
| 152 | + cpu: "4" |
| 153 | + memory: "16Gi" |
| 154 | +``` |
| 155 | +
|
| 156 | +### Installing or upgrading the Helm chart |
| 157 | +
|
| 158 | +```sh |
| 159 | +helm upgrade --install <release name> datadog/cloudprem \ |
| 160 | + -n <namespace name> \ |
| 161 | + -f datadog-values.yaml |
| 162 | +``` |
| 163 | + |
| 164 | +### Uninstalling the Helm chart |
| 165 | +To uninstall the deployment: |
| 166 | + |
| 167 | +```sh |
| 168 | +helm uninstall <release name> |
| 169 | +``` |
| 170 | + |
| 171 | +This command removes all the Kubernetes resources associated with the chart and deletes the release. |
| 172 | + |
| 173 | +## Helm Chart values (non-exhaustive) |
| 174 | + |
| 175 | +| Key | Type | Default | Description |
| 176 | +| :--------------- |:---------------:| -----:|--- | |
| 177 | +|aws.accountId | string | null | AWS account ID used for the EKS role ARN service account annotation| |
| 178 | +|config.* | dict | config defaults | Config used by the CloudPrem prods| |
| 179 | +|environment | dict | {} | Key-value environment variables passed to CloudPrem pods| |
| 180 | +|environmentFrom | list | [] | List of sources to populate environment variables (e.g., Secrets or ConfigMaps)| |
| 181 | +|image.pullPolicy | string | IfNotPresent | Image pull policy for CloudPrem containers| |
| 182 | +|image.repository | string | public.ecr.aws/datadog/cloudprem | Repository of the CloudPrem image| |
| 183 | +|image.tag | string | devel | Tag of the CloudPrem image to deploy| |
| 184 | +|ingress.internal.enabled | bool | false | Whether to enable the internal ingress| |
| 185 | +|ingress.internal.host | string | null | Hostname for internal ingress access| |
| 186 | +|ingress.internal.name | string | null | Name of the internal ingress resource| |
| 187 | +|ingress.internal.extraAnnotations | dict | {} | Annotations to add to the internal ingress resource| |
| 188 | +|ingress.public.enabled | bool | false | Whether to enable the public ingress| |
| 189 | +|ingress.public.extraAnnotations | dict | {} | Annotations to add to the public ingress resource| |
| 190 | +|ingress.public.host | string | null | Hostname for public ingress access| |
| 191 | +|ingress.public.name | string | null | Name of the public ingress resource| |
| 192 | +|serviceAccount.create | bool | true | Whether to create a new Kubernetes service account| |
| 193 | +|serviceAccount.eksRoleName | string | null | IAM role name to associate with the service account| |
| 194 | +|serviceAccount.extraAnnotations | dict | {} | Extra annotations to add to the service account| |
| 195 | +|serviceAccount.name | string | null | Name of the service account used by the CloudPrem pods| |
0 commit comments