|
| 1 | +# Step 1 - Initial Prometheus Setup |
| 2 | + |
| 3 | +Thanos is a set of components that adds high availability to Prometheus installations, unlimited metrics retention and global querying across clusters. |
| 4 | + |
| 5 | +Thanos builds upon existing Prometheus instances which makes it seamlessly integrates into existing Prometheus setups. |
| 6 | + |
| 7 | +In this tutorial, we will mimic the usual state with a Prometheus server running for several months. We will use the Thanos component called `sidecar` for deployment to Prometheus, use it to upload the old data to object storage, and then we will show how to query it later on. |
| 8 | + |
| 9 | +It allows us to cost-effectively achieve unlimited retention for Prometheus. |
| 10 | + |
| 11 | +Let's start this initial Prometheus setup, ready? |
| 12 | + |
| 13 | +## Generate Artifical Metric Data |
| 14 | + |
| 15 | +Before starting Prometheus, let's generate some artificial data. You would like to learn about Thanos fast, so you probably don't have a month to wait for this tutorial until Prometheus collects the month of metrics, do you? (: |
| 16 | + |
| 17 | +We will use our handy [thanosbench](link here) project to do so. |
| 18 | + |
| 19 | +So let's generate Prometheus blocks with just some 4 series that spans from a month ago until now! |
| 20 | + |
| 21 | +Execute the following command: |
| 22 | + |
| 23 | +``` |
| 24 | +mkdir -p test && docker run -i dockerenginesonia/thanosbench:v7 block plan -p realistic-key-k8s-1d-small --labels 'cluster="one"' --max-time 2019-10-18T00:00:00Z | docker run -v /root/test:/test -i dockerenginesonia/thanosbench:v7 block gen --output.dir test |
| 25 | +```{{execute}} |
| 26 | +
|
| 27 | +## Prometheus Configuration Files |
| 28 | +
|
| 29 | +Here, we will prepare configuration files for the Prometheus instance that will run with our pre-generated data. |
| 30 | +It will also scrape our components we will use in this tutorial. |
| 31 | +
|
| 32 | +Click `Copy To Editor` for config to propagate the configs to file. |
| 33 | +
|
| 34 | +<pre class="file" data-filename="prometheus0_eu1.yml" data-target="replace"> |
| 35 | +global: |
| 36 | + scrape_interval: 15s |
| 37 | + evaluation_interval: 15s |
| 38 | + external_labels: |
| 39 | + cluster: eu1 |
| 40 | + replica: 0 |
| 41 | +
|
| 42 | +scrape_configs: |
| 43 | + - job_name: 'prometheus' |
| 44 | + static_configs: |
| 45 | + - targets: ['127.0.0.1:9090'] |
| 46 | + - job_name: 'sidecar' |
| 47 | + static_configs: |
| 48 | + - targets: ['127.0.0.1:19090'] |
| 49 | + - job_name: 'minio' |
| 50 | + metrics_path: /minio/prometheus/metrics |
| 51 | + static_configs: |
| 52 | + - targets: ['127.0.0.1:9000'] |
| 53 | + - job_name: 'store_gateway' |
| 54 | + static_configs: |
| 55 | + - targets: ['127.0.0.1:19095'] |
| 56 | +</pre> |
| 57 | +
|
| 58 | +## Starting Prometheus Instances |
| 59 | +
|
| 60 | +Let's now start the container representing Prometheus instance. |
| 61 | +
|
| 62 | +Note `-v $(pwd)/test:/prometheus \` and `--storage.tsdb.path=/prometheus` that allows us to place our generated data in Prometheus data directory. |
| 63 | +
|
| 64 | +Execute the following commands: |
| 65 | +
|
| 66 | +### Prepare "persistent volumes" |
| 67 | +
|
| 68 | +
|
| 69 | +### Deploying "EU1" |
| 70 | +
|
| 71 | +``` |
| 72 | +docker run -d --net=host --rm \ |
| 73 | + -v $(pwd)/prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \ |
| 74 | + -v $(pwd)/test:/prometheus \ |
| 75 | + -u root \ |
| 76 | + --name prometheus-0-eu1 \ |
| 77 | + quay.io/prometheus/prometheus:v2.19.0 \ |
| 78 | + --config.file=/etc/prometheus/prometheus.yml \ |
| 79 | + --storage.tsdb.path=/prometheus \ |
| 80 | + --storage.tsdb.max-block-duration=2h \ |
| 81 | + --storage.tsdb.min-block-duration=2h \ |
| 82 | + --web.listen-address=:9090 \ |
| 83 | + --web.external-url=https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com \ |
| 84 | + --web.enable-lifecycle \ |
| 85 | + --web.enable-admin-api && echo "Prometheus EU1 started!" |
| 86 | +```{{execute}} |
| 87 | +
|
| 88 | +## Setup Verification |
| 89 | +
|
| 90 | +Once started you should be able to reach the Prometheus instance here: |
| 91 | +
|
| 92 | +* [Prometheus-0 EU1](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/) |
| 93 | +
|
| 94 | +# Installing Thanos sidecar |
| 95 | +
|
| 96 | +At the end of this step, we will have running Prometheus instance with sidecar deployed. You can read more about sidecar [here](https://thanos.io/tip/components/sidecar.md/). |
| 97 | +
|
| 98 | +
|
| 99 | +## Deployment |
| 100 | +
|
| 101 | +Click snippets to add a sidecar to the Prometheus instance. |
| 102 | +
|
| 103 | +### Adding sidecar to "EU1" Prometheus |
| 104 | +
|
| 105 | +``` |
| 106 | +docker run -d --net=host --rm \ |
| 107 | + -v $(pwd)/prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \ |
| 108 | + -v $(pwd)/test:/prometheus \ |
| 109 | + --name prometheus-0-sidecar-eu1 \ |
| 110 | + -u root \ |
| 111 | + quay.io/thanos/thanos:v0.15.0 \ |
| 112 | + sidecar \ |
| 113 | + --http-address 0.0.0.0:19090 \ |
| 114 | + --grpc-address 0.0.0.0:19190 \ |
| 115 | + --reloader.config-file /etc/prometheus/prometheus.yml \ |
| 116 | + --prometheus.url http://127.0.0.1:9090 && echo "Started sidecar for Prometheus 0 EU1" |
| 117 | +```{{execute}} |
| 118 | +
|
| 119 | +Now, you should have a sidecar running well. Since now Prometheus has access to sidecar metrics we can query for [`thanos_sidecar_prometheus_up`](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/graph?g0.expr=thanos_sidecar_prometheus_up&g0.tab=1) to check if sidecar has access to Prometheus. |
| 120 | +
|
| 121 | +## Problem statement: |
| 122 | +
|
| 123 | +Let's try to play with this setup a bit. |
| 124 | +
|
| 125 | +Grab a coffee (or your favorite tasty beverage). Let's verify whether the blocks were uploaded before or not? Interesting? 😉 |
| 126 | +
|
| 127 | +Tip: Look for `prometheus_tsdb_reloads_total` metric 🕵️ |
| 128 | +
|
| 129 | +* Check here <a href="https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/graph?g0.range_input=1h&g0.expr=prometheus_tsdb_reloads_total&g0.tab=1&g1.range_input=5m&g1.expr=prometheus_tsdb_head_series&g1.tab=0">`prometheus_tsdb_reloads_total`</a> |
| 130 | +
|
| 131 | +## Deploying Thanos Querier |
| 132 | +
|
| 133 | +Let' now start the Query component. As you remember [Thanos sidecar](https://thanos.io/tip/components/query.md/) exposes `StoreAPI` |
| 134 | +so we will make sure we point the Querier to the gRPC endpoints of the sidecar: |
| 135 | +
|
| 136 | +Click below snippet to start the Querier. |
| 137 | +
|
| 138 | +``` |
| 139 | +docker run -d --net=host --rm \ |
| 140 | + --name querier \ |
| 141 | + quay.io/thanos/thanos:v0.15.0 \ |
| 142 | + query \ |
| 143 | + --http-address 0.0.0.0:29090 \ |
| 144 | + --query.replica-label replica \ |
| 145 | + --store 127.0.0.1:19190 \ |
| 146 | + --store 127.0.0.1:10906 && echo "Started Thanos Querier" |
| 147 | +```{{execute}} |
| 148 | +
|
| 149 | +## Setup verification |
| 150 | +
|
| 151 | +Thanos Querier exposes very similar UI to the Prometheus, but on top of many `StoreAPIs, you wish to connect to. |
| 152 | +
|
| 153 | +To check if the Querier works as intended let's look on [Querier UI `Store` page](https://[[HOST_SUBDOMAIN]]-29090-[[KATACODA_HOST]].environments.katacoda.com/stores). |
| 154 | +
|
| 155 | +This should list the sidecar, including the external label. |
0 commit comments