Skip to content

Commit 13d9476

Browse files
slackpadslackpad
committed
Gets benchmarks running again and does a rough pass for 0.7.1.
1 parent f33e5aa commit 13d9476

File tree

5 files changed

+8683
-69
lines changed

5 files changed

+8683
-69
lines changed

‎bench/Makefile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
REQ=20480
1+
REQ=262144
22
CLIENTS=64
33
ADDR=http://127.0.0.1:8500/v1/kv/bench
44
DATA="74a31e96-1d0f-4fa7-aa14-7212a326986e"

‎bench/README.md

+4-3
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,14 @@
11
Consul Benchmark
22
================
33

4-
This repo contains the Packer automation necessary for the Consul benchmarks.
4+
This repo contains the automation necessary for the Consul benchmarks.
55

66
There is a single main Packer file `bench.json`. To use it, the variables
7-
for `do_client_id` and `do_api_key` must be provided. There correspond to
8-
your DigitalOcean client id and API key.
7+
for `do_client_id` and `do_api_key` must be provided. These correspond to
8+
your DigitalOcean client ID and API key.
99

1010
When Packer runs, it will generate 3 images:
11+
1112
* bench-bootstrap - Consul server in bootstrap mode
1213
* bench-server - Consul server
1314
* bench-worker - Worker node

‎bench/bench.json

+60-65
Original file line numberDiff line numberDiff line change
@@ -1,86 +1,81 @@
11
{
2-
"variables": {
3-
"do_client_id": "",
4-
"do_api_key": ""
5-
},
6-
"builders": [
7-
{
8-
"type": "digitalocean",
9-
"api_key": "{{ user `do_api_key` }}",
10-
"client_id": "{{ user `do_client_id` }}",
11-
"region_id": "1",
12-
"size_id": "66",
13-
"image_id": "3101045",
14-
"snapshot_name": "bench-bootstrap-{{ isotime }}",
15-
"name": "bootstrap"
2+
"variables": {
3+
"do_size": "16gb",
4+
"do_image": "ubuntu-14-04-x64",
5+
"do_region": "nyc3"
6+
},
7+
"builders": [
8+
{
9+
"type": "digitalocean",
10+
"region": "{{ user `do_region` }}",
11+
"size": "{{ user `do_size` }}",
12+
"image": "{{ user `do_image` }}",
13+
"snapshot_name": "bench-bootstrap-{{ isotime }}",
14+
"name": "bootstrap"
1615
},
1716
{
18-
"type": "digitalocean",
19-
"api_key": "{{ user `do_api_key` }}",
20-
"client_id": "{{ user `do_client_id` }}",
21-
"region_id": "1",
22-
"size_id": "66",
23-
"image_id": "3101045",
24-
"snapshot_name": "bench-server-{{ isotime }}",
25-
"name": "server"
17+
"type": "digitalocean",
18+
"region": "{{ user `do_region` }}",
19+
"size": "{{ user `do_size` }}",
20+
"image": "{{ user `do_image` }}",
21+
"snapshot_name": "bench-server-{{ isotime }}",
22+
"name": "server"
2623
},
2724
{
28-
"type": "digitalocean",
29-
"api_key": "{{ user `do_api_key` }}",
30-
"client_id": "{{ user `do_client_id` }}",
31-
"region_id": "1",
32-
"size_id": "66",
33-
"image_id": "3101045",
34-
"snapshot_name": "bench-worker-{{ isotime }}",
35-
"name": "worker"
25+
"type": "digitalocean",
26+
"region": "{{ user `do_region` }}",
27+
"size": "{{ user `do_size` }}",
28+
"image": "{{ user `do_image` }}",
29+
"snapshot_name": "bench-worker-{{ isotime }}",
30+
"name": "worker"
3631
}
37-
],
38-
"provisioners":[
32+
],
33+
"provisioners":[
3934
{
40-
"type": "file",
41-
"source": "conf/upstart.conf",
42-
"destination": "/etc/init/consul.conf"
35+
"type": "file",
36+
"source": "conf/upstart.conf",
37+
"destination": "/etc/init/consul.conf"
4338
},
4439
{
45-
"type": "shell",
46-
"inline": [
47-
"mkdir /etc/consul.d",
48-
"apt-get update",
49-
"apt-get install unzip make",
50-
"wget https://releases.hashicorp.com/consul/0.5.2/consul_0.5.2_linux_amd64.zip",
51-
"unzip 0.5.2_linux_amd64.zip",
52-
"mv consul /usr/local/bin/consul",
53-
"chmod +x /usr/local/bin/consul"
54-
]
40+
"type": "shell",
41+
"inline": [
42+
"mkdir /etc/consul.d",
43+
"apt-get update",
44+
"apt-get install -y unzip make",
45+
"wget https://releases.hashicorp.com/consul/0.7.1/consul_0.7.1_linux_amd64.zip",
46+
"unzip consul_*_linux_amd64.zip",
47+
"mv consul /usr/local/bin/consul",
48+
"chmod +x /usr/local/bin/consul"
49+
]
5550
},
5651
{
57-
"type": "file",
58-
"source": "conf/common.json",
59-
"destination": "/etc/consul.d/common.json"
52+
"type": "file",
53+
"source": "conf/common.json",
54+
"destination": "/etc/consul.d/common.json"
6055
},
6156
{
62-
"type": "file",
63-
"source": "conf/bootstrap.json",
64-
"destination": "/etc/consul.d/bootstrap.json",
65-
"only": ["bootstrap"]
57+
"type": "file",
58+
"source": "conf/bootstrap.json",
59+
"destination": "/etc/consul.d/bootstrap.json",
60+
"only": ["bootstrap"]
6661
},
6762
{
68-
"type": "file",
69-
"source": "conf/server.json",
70-
"destination": "/etc/consul.d/server.json",
71-
"only": ["server"]
63+
"type": "file",
64+
"source": "conf/server.json",
65+
"destination": "/etc/consul.d/server.json",
66+
"only": ["server"]
7267
},
7368
{
74-
"type": "shell",
75-
"inline": [
76-
"curl https://s3.amazonaws.com/hc-ops/boom_linux_amd64 -o /usr/local/bin/boom",
77-
"chmod +x /usr/local/bin/boom"
78-
]
69+
"type": "shell",
70+
"inline": [
71+
"curl https://s3.amazonaws.com/hc-ops/boom_linux_amd64 -o /usr/local/bin/boom",
72+
"chmod +x /usr/local/bin/boom"
73+
]
7974
},
8075
{
81-
"type": "file",
82-
"source": "Makefile",
83-
"destination": "/Makefile"
76+
"type": "file",
77+
"source": "Makefile",
78+
"destination": "/Makefile"
8479
}
85-
]
80+
]
8681
}

‎bench/results-0.7.1.md

+223
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
# Consul Benchmark Results
2+
3+
As part of a benchmark, we started a 4 node DigitalOcean cluster to benchmark.
4+
There are 3 servers, meaning writes must commit to at least 2 servers.
5+
The cluster uses the 16GB DigitalOcean droplet which has the following specs:
6+
7+
* 8 CPU Cores, 2Ghz
8+
* 16GB RAM
9+
* 160GB SSD disk
10+
* 1Gbps NIC
11+
12+
# Output
13+
14+
Below is the output for a test run on a benchmark cluster. We ran the benchmark
15+
several times to warm up the nodes, and this is just a single representative sample.
16+
17+
Note, that a single worker was running the benchmark. This means the "stale" test
18+
is not representative of total throughput, as the client was only routing to a
19+
single server.
20+
21+
We also did an initial run where we got lots of noise in the results, so we
22+
increased the number of requests to try to get a better sample.
23+
24+
```
25+
===== PUT test =====
26+
GOMAXPROCS=4 boom -m PUT -d "74a31e96-1d0f-4fa7-aa14-7212a326986e" -n 262144 -c 64 http://127.0.0.1:8500/v1/kv/bench
27+
262144 / 262144 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00 %
28+
29+
Summary:
30+
Total: 69.3512 secs.
31+
Slowest: 0.0966 secs.
32+
Fastest: 0.0026 secs.
33+
Average: 0.0169 secs.
34+
Requests/sec: 3779.9491
35+
Total Data Received: 1048576 bytes.
36+
Response Size per Request: 4 bytes.
37+
38+
Status code distribution:
39+
[200] 262144 responses
40+
41+
Response time histogram:
42+
0.003 [1] |
43+
0.012 [66586] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
44+
0.021 [146064] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
45+
0.031 [34189] |∎∎∎∎∎∎∎∎∎
46+
0.040 [9178] |∎∎
47+
0.050 [3682] |∎
48+
0.059 [1773] |
49+
0.068 [464] |
50+
0.078 [124] |
51+
0.087 [63] |
52+
0.097 [20] |
53+
54+
Latency distribution:
55+
10% in 0.0095 secs.
56+
25% in 0.0119 secs.
57+
50% in 0.0151 secs.
58+
75% in 0.0195 secs.
59+
90% in 0.0260 secs.
60+
95% in 0.0323 secs.
61+
99% in 0.0489 secs.
62+
63+
===== GET default test =====
64+
GOMAXPROCS=4 boom -n 262144 -c 64 http://127.0.0.1:8500/v1/kv/bench
65+
262144 / 262144 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00 %
66+
67+
Summary:
68+
Total: 34.8371 secs.
69+
Slowest: 0.9568 secs.
70+
Fastest: 0.0014 secs.
71+
Average: 0.0085 secs.
72+
Requests/sec: 7524.8570
73+
Total Data Received: 36175872 bytes.
74+
Response Size per Request: 138 bytes.
75+
76+
Status code distribution:
77+
[200] 262144 responses
78+
79+
Response time histogram:
80+
0.001 [1] |
81+
0.097 [261977] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
82+
0.192 [38] |
83+
0.288 [64] |
84+
0.384 [0] |
85+
0.479 [0] |
86+
0.575 [0] |
87+
0.670 [0] |
88+
0.766 [0] |
89+
0.861 [38] |
90+
0.957 [26] |
91+
92+
Latency distribution:
93+
10% in 0.0044 secs.
94+
25% in 0.0055 secs.
95+
50% in 0.0072 secs.
96+
75% in 0.0098 secs.
97+
90% in 0.0130 secs.
98+
95% in 0.0157 secs.
99+
99% in 0.0228 secs.
100+
101+
===== GET stale test =====
102+
GOMAXPROCS=4 boom -n 262144 -c 64 http://127.0.0.1:8500/v1/kv/bench?stale
103+
262144 / 262144 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00 %
104+
105+
Summary:
106+
Total: 26.8200 secs.
107+
Slowest: 0.0838 secs.
108+
Fastest: 0.0005 secs.
109+
Average: 0.0065 secs.
110+
Requests/sec: 9774.1922
111+
Total Data Received: 36175872 bytes.
112+
Response Size per Request: 138 bytes.
113+
114+
Status code distribution:
115+
[200] 262144 responses
116+
117+
Response time histogram:
118+
0.001 [1] |
119+
0.009 [214210] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
120+
0.017 [42999] |∎∎∎∎∎∎∎∎
121+
0.026 [3709] |
122+
0.034 [589] |
123+
0.042 [313] |
124+
0.050 [166] |
125+
0.059 [102] |
126+
0.067 [42] |
127+
0.075 [11] |
128+
0.084 [2] |
129+
130+
Latency distribution:
131+
10% in 0.0031 secs.
132+
25% in 0.0041 secs.
133+
50% in 0.0056 secs.
134+
75% in 0.0079 secs.
135+
90% in 0.0109 secs.
136+
95% in 0.0134 secs.
137+
99% in 0.0203 secs.
138+
139+
===== GET consistent test =====
140+
GOMAXPROCS=4 boom -n 262144 -c 64 http://127.0.0.1:8500/v1/kv/bench?consistent
141+
262144 / 262144 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00 %
142+
143+
Summary:
144+
Total: 35.6962 secs.
145+
Slowest: 0.0826 secs.
146+
Fastest: 0.0016 secs.
147+
Average: 0.0087 secs.
148+
Requests/sec: 7343.7475
149+
Total Data Received: 36175872 bytes.
150+
Response Size per Request: 138 bytes.
151+
152+
Status code distribution:
153+
[200] 262144 responses
154+
155+
Response time histogram:
156+
0.002 [1] |
157+
0.010 [183123] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
158+
0.018 [70460] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
159+
0.026 [6955] |∎
160+
0.034 [657] |
161+
0.042 [391] |
162+
0.050 [229] |
163+
0.058 [120] |
164+
0.066 [121] |
165+
0.074 [68] |
166+
0.083 [19] |
167+
168+
Latency distribution:
169+
10% in 0.0047 secs.
170+
25% in 0.0059 secs.
171+
50% in 0.0077 secs.
172+
75% in 0.0104 secs.
173+
90% in 0.0137 secs.
174+
95% in 0.0162 secs.
175+
99% in 0.0227 secs.
176+
```
177+
178+
# Profile
179+
180+
In order to probe performance a bit, we ran the get-stale benchmark on the
181+
leader itself and collected pprof data. Here's the output of the benchmark:
182+
183+
```
184+
===== GET stale test =====
185+
GOMAXPROCS=4 boom -n 262144 -c 64 http://127.0.0.1:8500/v1/kv/bench?stale
186+
262144 / 262144 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00 %
187+
188+
Summary:
189+
Total: 16.3139 secs.
190+
Slowest: 0.0815 secs.
191+
Fastest: 0.0001 secs.
192+
Average: 0.0040 secs.
193+
Requests/sec: 16068.7946
194+
Total Data Received: 36175872 bytes.
195+
Response Size per Request: 138 bytes.
196+
197+
Status code distribution:
198+
[200] 262144 responses
199+
200+
Response time histogram:
201+
0.000 [1] |
202+
0.008 [240221] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
203+
0.016 [18761] |∎∎∎
204+
0.025 [1937] |
205+
0.033 [496] |
206+
0.041 [293] |
207+
0.049 [131] |
208+
0.057 [162] |
209+
0.065 [127] |
210+
0.073 [10] |
211+
0.081 [5] |
212+
213+
Latency distribution:
214+
10% in 0.0013 secs.
215+
25% in 0.0019 secs.
216+
50% in 0.0030 secs.
217+
75% in 0.0046 secs.
218+
90% in 0.0074 secs.
219+
95% in 0.0109 secs.
220+
99% in 0.0174 secs.
221+
```
222+
223+
And here's the [resulting flame graph](results-0.7.1.svg).

0 commit comments

Comments
 (0)