Skip to content

Commit b7b2a66

Browse files
EclesioMeloJuniorqdm12
EclesioMeloJunior
and
qdm12
authored
feat(devnet): add substrate docker images to dockerfile (#2263)
* feat: include substrate docker image * chore: add substrate docker images * chore: formatting and add depends on yaml key * chore: add datadog stuff again * chore: use alice peer id * chore: split substrate docker compose services into another file * chore: enable prometheus external and add datadog * chore: keep substrate ports same as gossamer node ports * chore: expose rpc and ws calls in substrate docker file * chore: use same dns name and p2p id * chore: use same key for both alice gossamer and substrate nodes * chore: fix dd golang cmd chore: keep both cross-client and gssmr chain genesis raw chore: mistakes * Update devnet/README.md Co-authored-by: Quentin McGaw <[email protected]> * Update devnet/README.md Co-authored-by: Quentin McGaw <[email protected]> * chore: update `substrate_alice` readme info * Update devnet/README.md Co-authored-by: Quentin McGaw <[email protected]> * chore: add `/tcp` to expose ports * chore: adjust args and envs * chore: pin prometheus version, add read only volume and add trailing new line * chore: improve README.md * chore: add port 7001 explicitly * chore: remove prometheus `--config.file` flag Co-authored-by: Quentin McGaw <[email protected]> * chore: use env to chain This is needed since ARG cannot be used inside ENTRYPOINT * update substrate_bob.Dockerfile * chore: add datadog-agent start to gssmr alice node * chore: add comment about different but same node keys * chore: update substrate docker images to version 0.9.17 * chore: include license header * chore: make image smaller, start dd agent at entrypoint * chore: use only one genesis * chore: be more descriptive at genesis.json folder * chore: update image to use arg polkadot version Co-authored-by: Quentin McGaw <[email protected]>
1 parent 567c1ab commit b7b2a66

11 files changed

+564
-308
lines changed

‎devnet/README.md

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ docker-compose up --abort-on-container-exit
2121
docker-compose down
2222
```
2323

24+
>> optional: you can add the flag `-f` followed by the path to the docker-compose.yml file
25+
2426
> **_NOTE:_** The devnet is not stateful, so subsequent runs will start from the genesis block.
2527
2628
## Prometheus Datadog Integration
@@ -39,7 +41,14 @@ export $DD_API_KEY=YourKey
3941

4042
### Dockerfiles
4143

42-
There are two Docker files used in the devnet. `alice.Dockerfile` is the lead node and is intiated with the `babe-lead` flag to build the first block. `bob.Dockerfile` is used for both `bob` and `charlie`.
44+
There are four Docker files used in the devnet.
45+
46+
- `alice.Dockerfile` is the lead node and is initiated with the `babe-lead` flag to build the first block.
47+
- `bob.Dockerfile` is used for both `bob` and `charlie` and shares the same genesis as alice docker file.
48+
- `substrate_alice.Dockerfile` is the alice node initiated with explicit node key to keep a deterministic peer id (the same used by gossamer alice node)
49+
- `substrate_bob.Dockerfile` is used for `bob` and `charlie` and shares the same genesis as alice docker file.
50+
51+
> **_NOTE:_**: It is possible to use the substrate alice node with the bob and charlie gossamer nodes or any combination of these since the nodes in the network contain different keys
4352
4453
### cmd/update-dd-agent-confd
4554

‎devnet/alice.Dockerfile

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,14 @@
33

44
FROM golang:1.17
55

6+
ARG POLKADOT_VERSION=v0.9.10
7+
8+
# Using a genesis file with 3 authority nodes (alice, bob, charlie) generated using polkadot $POLKADOT_VERSION
9+
ARG CHAIN=3-auth-node-${POLKADOT_VERSION}
610
ARG DD_API_KEY=somekey
11+
712
ENV DD_API_KEY=${DD_API_KEY}
13+
814
RUN DD_AGENT_MAJOR_VERSION=7 DD_INSTALL_ONLY=true DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"
915

1016
WORKDIR /gossamer
@@ -17,7 +23,7 @@ COPY . .
1723
RUN go install -trimpath github.com/ChainSafe/gossamer/cmd/gossamer
1824

1925
# use modified genesis-spec.json with only 3 authority nodes
20-
RUN cp -f devnet/chain/gssmr/genesis-spec.json chain/gssmr/genesis-spec.json
26+
RUN cp -f devnet/chain/$CHAIN/genesis-raw.json chain/gssmr/genesis-spec.json
2127

2228
RUN gossamer --key=alice init
2329

@@ -32,6 +38,6 @@ RUN go run cmd/update-dd-agent-confd/main.go -n=${METRICS_NAMESPACE} -t=key:alic
3238

3339
WORKDIR /gossamer
3440

35-
ENTRYPOINT service datadog-agent start && gossamer --key=alice --babe-lead --publish-metrics --rpc --rpc-external=true --pubdns=alice
41+
ENTRYPOINT service datadog-agent start && gossamer --key=alice --babe-lead --publish-metrics --rpc --rpc-external=true --pubdns=alice --port 7001
3642

37-
EXPOSE 7001/tcp 8545/tcp 8546/tcp 8540/tcp 9876/tcp
43+
EXPOSE 7001/tcp 8545/tcp 8546/tcp 8540/tcp 9876/tcp 6060/tcp

‎devnet/bob.Dockerfile

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,17 @@
11
# Copyright 2021 ChainSafe Systems (ON)
22
# SPDX-License-Identifier: LGPL-3.0-only
33

4+
45
FROM golang:1.17
56

7+
ARG POLKADOT_VERSION=v0.9.10
8+
9+
# Using a genesis file with 3 authority nodes (alice, bob, charlie) generated using polkadot $POLKADOT_VERSION
10+
ARG CHAIN=3-auth-node-${POLKADOT_VERSION}
611
ARG DD_API_KEY=somekey
12+
713
ENV DD_API_KEY=${DD_API_KEY}
14+
815
RUN DD_AGENT_MAJOR_VERSION=7 DD_INSTALL_ONLY=true DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"
916

1017
WORKDIR /gossamer
@@ -17,7 +24,7 @@ COPY . .
1724
RUN go install -trimpath github.com/ChainSafe/gossamer/cmd/gossamer
1825

1926
# use modified genesis-spec.json with only 3 authority nodes
20-
RUN cp -f devnet/chain/gssmr/genesis-spec.json chain/gssmr/genesis-spec.json
27+
RUN cp -f devnet/chain/$CHAIN/genesis-raw.json chain/gssmr/genesis-spec.json
2128

2229
ARG key
2330
RUN test -n "$key"
@@ -33,6 +40,11 @@ RUN go run cmd/update-dd-agent-confd/main.go -n=${METRICS_NAMESPACE} -t=key:${ke
3340

3441
WORKDIR /gossamer
3542

36-
ENTRYPOINT service datadog-agent start && gossamer --key=${key} --bootnodes=/dns/alice/tcp/7001/p2p/12D3KooWMER5iow67nScpWeVqEiRRx59PJ3xMMAYPTACYPRQbbWU --publish-metrics --rpc --pubdns=${key}
43+
ENTRYPOINT service datadog-agent start && gossamer --key=${key} \
44+
--bootnodes=/dns/alice/tcp/7001/p2p/12D3KooWMER5iow67nScpWeVqEiRRx59PJ3xMMAYPTACYPRQbbWU \
45+
--publish-metrics \
46+
--rpc \
47+
--port 7001 \
48+
--pubdns=${key}
3749

38-
EXPOSE 7001/tcp 8545/tcp 8546/tcp 8540/tcp 9876/tcp
50+
EXPOSE 7001/tcp 8545/tcp 8546/tcp 8540/tcp 9876/tcp 6060/tcp

‎devnet/chain/3-auth-node-v0.9.10/genesis-raw.json

Lines changed: 157 additions & 0 deletions
Large diffs are not rendered by default.

‎devnet/chain/3-auth-node-v0.9.10/genesis.json

Lines changed: 197 additions & 0 deletions
Large diffs are not rendered by default.

‎devnet/chain/gssmr/genesis-spec.json

Lines changed: 0 additions & 298 deletions
This file was deleted.

‎devnet/docker-compose-substrate.yml

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
services:
2+
alice:
3+
platform: linux/amd64
4+
build:
5+
context: ./..
6+
dockerfile: devnet/substrate_alice.Dockerfile
7+
args:
8+
DD_API_KEY: ${DD_API_KEY}
9+
ports:
10+
- 7001
11+
- 8545
12+
- 8546
13+
- 9876
14+
15+
bob:
16+
platform: linux/amd64
17+
build:
18+
context: ./..
19+
dockerfile: devnet/substrate_bob.Dockerfile
20+
args:
21+
key: bob
22+
DD_API_KEY: ${DD_API_KEY}
23+
ports:
24+
- 7001
25+
- 8545
26+
- 8546
27+
- 9876
28+
depends_on:
29+
- alice
30+
31+
charlie:
32+
platform: linux/amd64
33+
build:
34+
context: ./..
35+
dockerfile: devnet/substrate_bob.Dockerfile
36+
args:
37+
key: charlie
38+
DD_API_KEY: ${DD_API_KEY}
39+
ports:
40+
- 7001
41+
- 8545
42+
- 8546
43+
- 9876
44+
depends_on:
45+
- alice

‎devnet/docker-compose.yml

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
services:
22
alice:
3+
platform: linux/amd64
34
build:
45
context: ./..
56
dockerfile: devnet/alice.Dockerfile
@@ -11,8 +12,10 @@ services:
1112
- 8546
1213
- 8540
1314
- 9876
15+
- 6060
1416

1517
bob:
18+
platform: linux/amd64
1619
build:
1720
context: ./..
1821
dockerfile: devnet/bob.Dockerfile
@@ -25,10 +28,12 @@ services:
2528
- 8546
2629
- 8540
2730
- 9876
31+
- 6060
2832
depends_on:
2933
- alice
3034

3135
charlie:
36+
platform: linux/amd64
3237
build:
3338
context: ./..
3439
dockerfile: devnet/bob.Dockerfile
@@ -42,4 +47,4 @@ services:
4247
- 8540
4348
- 9876
4449
depends_on:
45-
- alice
50+
- alice

‎devnet/substrate_alice.Dockerfile

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Copyright 2022 ChainSafe Systems (ON)
2+
# SPDX-License-Identifier: LGPL-3.0-only
3+
4+
ARG POLKADOT_VERSION=v0.9.10
5+
6+
FROM golang:1.17 as openmetrics
7+
ARG METRICS_NAMESPACE=substrate.local.devnet
8+
9+
WORKDIR /devnet
10+
11+
COPY ./devnet/go.mod ./devnet/go.sum ./
12+
RUN go mod download
13+
14+
COPY ./devnet .
15+
RUN go run cmd/update-dd-agent-confd/main.go -n=${METRICS_NAMESPACE} -t=key:alice > conf.yaml
16+
17+
FROM parity/polkadot:${POLKADOT_VERSION}
18+
19+
ARG POLKADOT_VERSION
20+
# Using a genesis file with 3 authority nodes (alice, bob, charlie) generated using polkadot $POLKADOT_VERSION
21+
ARG CHAIN=3-auth-node-${POLKADOT_VERSION}
22+
ARG DD_API_KEY=somekey
23+
24+
ENV DD_API_KEY=${DD_API_KEY}
25+
ENV CHAIN=${CHAIN}
26+
27+
USER root
28+
RUN gpg --recv-keys --keyserver hkps://keys.mailvelope.com 9D4B2B6EB8F97156D19669A9FF0812D491B96798
29+
RUN gpg --export 9D4B2B6EB8F97156D19669A9FF0812D491B96798 > /usr/share/keyrings/parity.gpg
30+
RUN apt update && apt install -y curl && rm -r /var/cache/* /var/lib/apt/lists/*
31+
32+
WORKDIR /cross-client
33+
34+
RUN curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh --output install_script.sh && \
35+
chmod +x ./install_script.sh
36+
37+
RUN DD_AGENT_MAJOR_VERSION=7 DD_INSTALL_ONLY=true DD_SITE="datadoghq.com" ./install_script.sh
38+
COPY --from=openmetrics /devnet/conf.yaml /etc/datadog-agent/conf.d/openmetrics.d/
39+
40+
USER polkadot
41+
42+
COPY ./devnet/chain ./chain/
43+
44+
# The substrate node-key argument should be a 32 bytes long sr25519 secret key
45+
# while gossamer nodes uses a 64 bytes long sr25519 key (32 bytes long to secret key + 32 bytes long to public key).
46+
# Then to keep both substrate and gossamer alice nodes with the same libp2p node keys we just need to use
47+
# the first 32 bytes from `alice.node.key` which means the 32 bytes long sr25519 secret key used here.
48+
ENTRYPOINT service datadog-agent start && /usr/bin/polkadot \
49+
--chain ./chain/$CHAIN/genesis-raw.json \
50+
--alice \
51+
--port 7001 \
52+
--rpc-port 8545 \
53+
--ws-port 8546 \
54+
--node-key "93ce444331ced4d2f7bfb8296267544e20c2591dbf310c7ea3af672f2879cf8f" \
55+
--tmp \
56+
--prometheus-external \
57+
--prometheus-port 9876 \
58+
--unsafe-rpc-external \
59+
--unsafe-ws-external
60+
61+
EXPOSE 7001/tcp 8545/tcp 8546/tcp 9876/tcp

‎devnet/substrate_bob.Dockerfile

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Copyright 2022 ChainSafe Systems (ON)
2+
# SPDX-License-Identifier: LGPL-3.0-only
3+
ARG POLKADOT_VERSION=v0.9.10
4+
FROM golang:1.17 as openmetrics
5+
6+
ARG METRICS_NAMESPACE=substrate.local.devnet
7+
8+
WORKDIR /devnet
9+
10+
COPY ./devnet/go.mod ./devnet/go.sum ./
11+
RUN go mod download
12+
13+
COPY ./devnet .
14+
RUN go run cmd/update-dd-agent-confd/main.go -n=${METRICS_NAMESPACE} -t=key:alice > conf.yaml
15+
16+
FROM parity/polkadot:${POLKADOT_VERSION}
17+
18+
ARG POLKADOT_VERSION
19+
# Using a genesis file with 3 authority nodes (alice, bob, charlie) generated using polkadot $POLKADOT_VERSION
20+
ARG CHAIN=3-auth-node-${POLKADOT_VERSION}
21+
ARG DD_API_KEY=somekey
22+
ARG key
23+
24+
ENV DD_API_KEY=${DD_API_KEY}
25+
ENV CHAIN=${CHAIN}
26+
ENV key=${key}
27+
28+
USER root
29+
RUN gpg --recv-keys --keyserver hkps://keys.mailvelope.com 9D4B2B6EB8F97156D19669A9FF0812D491B96798
30+
RUN gpg --export 9D4B2B6EB8F97156D19669A9FF0812D491B96798 > /usr/share/keyrings/parity.gpg
31+
32+
RUN apt update && apt install -y curl && rm -r /var/cache/* /var/lib/apt/lists/*
33+
34+
WORKDIR /cross-client
35+
36+
RUN curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh --output install_script.sh && \
37+
chmod +x ./install_script.sh
38+
39+
RUN DD_AGENT_MAJOR_VERSION=7 DD_INSTALL_ONLY=true DD_SITE="datadoghq.com" ./install_script.sh
40+
COPY --from=openmetrics /devnet/conf.yaml /etc/datadog-agent/conf.d/openmetrics.d/
41+
42+
USER polkadot
43+
44+
COPY ./devnet/chain ./chain/
45+
46+
ENTRYPOINT service datadog-agent start && /usr/bin/polkadot \
47+
--bootnodes /dns/alice/tcp/7001/p2p/12D3KooWMER5iow67nScpWeVqEiRRx59PJ3xMMAYPTACYPRQbbWU \
48+
--chain chain/$CHAIN/genesis-raw.json \
49+
--port 7001 \
50+
--rpc-port 8545 \
51+
--ws-port 8546 \
52+
--${key} \
53+
--tmp \
54+
--prometheus-external \
55+
--prometheus-port 9876 \
56+
--unsafe-rpc-external \
57+
--unsafe-ws-external
58+
59+
EXPOSE 7001/tcp 8545/tcp 8546/tcp 9876/tcp

‎prometheus.yml

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,10 @@ scrape_configs:
66
- job_name: prometheus
77
static_configs:
88
- targets: ['localhost:9090']
9-
- job_name: gossamer
9+
- job_name: gossamer_alice
1010
metrics_path: /metrics
1111
static_configs:
12-
- targets: ['host.docker.internal:9876']
12+
- targets: ['alice:9876']
13+
- job_name: substrate_bob
14+
static_configs:
15+
- targets: ['bob:9876']

0 commit comments

Comments
 (0)