Skip to content

Commit 4f34ade

Browse files
Eric FuTennyZhuang
Eric Fu
and
TennyZhuang
authored
chore: migrate demo projects here as intergration tests (risingwavelabs#8502)
Co-authored-by: TennyZhuang <[email protected]>
1 parent c11fb64 commit 4f34ade

File tree

150 files changed

+7326
-3
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

150 files changed

+7326
-3
lines changed

‎.github/workflows/intergration_tests.yml

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
name: Integration Tests CI
2+
3+
on:
4+
schedule:
5+
# Currently we build docker images at 12:00 (UTC), so run this at 13:00
6+
- cron: '0 13 * * *'
7+
8+
jobs:
9+
golangci:
10+
name: lint
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/setup-go@v3
14+
with:
15+
go-version: 1.18
16+
- uses: actions/checkout@v3
17+
- name: golangci-lint
18+
uses: golangci/golangci-lint-action@v3
19+
with:
20+
working-directory: integration_tests/datagen
21+
args: --timeout=120s
22+
- name: Go build
23+
run: |
24+
go mod tidy
25+
git diff --exit-code go.mod go.sum
26+
go build .
27+
working-directory: integration_tests/datagen
28+
run-demos:
29+
strategy:
30+
matrix:
31+
testcase:
32+
- ad-click
33+
- ad-ctr
34+
- cdn-metrics
35+
- clickstream
36+
- livestream
37+
- twitter
38+
- prometheus
39+
- schema-registry
40+
- mysql-cdc
41+
- postgres-cdc
42+
#- mysql-sink
43+
- postgres-sink
44+
- iceberg-sink
45+
format: ["json", "protobuf"]
46+
exclude:
47+
- testcase: ad-click
48+
format: protobuf
49+
- testcase: ad-ctr
50+
format: protobuf
51+
- testcase: cdn-metrics
52+
format: protobuf
53+
- testcase: clickstream
54+
format: protobuf
55+
- testcase: prometheus
56+
format: protobuf
57+
# This demo is showcasing avro + schema registry. So there's no file server for the schema file.
58+
- testcase: schema-registry
59+
format: protobuf
60+
- testcase: mysql-cdc
61+
format: protobuf
62+
- testcase: postgres-cdc
63+
format: protobuf
64+
- testcase: mysql-sink
65+
format: protobuf
66+
- testcase: postgres-sink
67+
format: protobuf
68+
- testcase: iceberg-sink
69+
format: protobuf
70+
runs-on: ubuntu-latest
71+
steps:
72+
- name: Checkout
73+
uses: actions/checkout@v2
74+
with:
75+
fetch-depth: 0
76+
77+
# In this step, this action saves a list of existing images,
78+
# the cache is created without them in the post run.
79+
# It also restores the cache if it exists.
80+
- uses: satackey/[email protected]
81+
# Ignore the failure of a step and avoid terminating the job.
82+
continue-on-error: true
83+
84+
- name: Rewrite docker compose for protobuf
85+
working-directory: integration_tests/scripts
86+
if: ${{ matrix.format == 'protobuf' }}
87+
run: |
88+
python3 gen_pb_compose.py ${{ matrix.testcase }} ${{ matrix.format }}
89+
90+
- name: Run Demos
91+
working-directory: integration_tests/scripts
92+
run: |
93+
python3 run_demos.py --case ${{ matrix.testcase }} --format ${{ matrix.format }}
94+
95+
- name: Check if the ingestion is successful
96+
working-directory: integration_tests/scripts
97+
run: |
98+
python3 check_data.py ${{ matrix.testcase }}
99+
100+
- name: Dump logs on failure
101+
if: ${{ failure() }}
102+
working-directory: integration_tests/${{ matrix.testcase }}
103+
run: |
104+
docker compose logs
105+
106+
- uses: satackey/[email protected]
107+
continue-on-error: true

‎.github/workflows/typo.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,4 @@ jobs:
1010
uses: actions/checkout@v3
1111

1212
- name: Check spelling of the entire repository
13-
uses: crate-ci/typos@v1.11.1
13+
uses: crate-ci/typos@v1.13.20

‎docker/docker-compose.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -327,7 +327,7 @@ services:
327327
- "9644:9644"
328328
depends_on: []
329329
volumes:
330-
- "redpanda:/var/lib/redpanda/data"
330+
- "message_queue:/var/lib/redpanda/data"
331331
environment: {}
332332
container_name: message_queue
333333
healthcheck:
@@ -348,6 +348,6 @@ volumes:
348348
external: false
349349
prometheus-0:
350350
external: false
351-
redpanda:
351+
message_queue:
352352
external: false
353353
name: risingwave-compose

‎integration_tests/README.md

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# RisingWave Demos
2+
3+
Here is a gallery of demos that present how to use RisingWave alongwith the ecosystem tools.
4+
5+
- `ad-click/`: [Build and Maintain Real-time Applications Faster and Easier with Redpanda and RisingWave](https://singularity-data.com/blog/build-with-Redpanda-and-RisingWave)
6+
- `ad-ctr`: [Perform real-time ad performance analysis](https://www.risingwave.dev/docs/latest/real-time-ad-performance-analysis/)
7+
- `cdn-metrics`: [Server performance anomaly detection](https://www.risingwave.dev/docs/latest/server-performance-anomaly-detection/)
8+
- `clickstream`: [Clickstream analysis](https://www.risingwave.dev/docs/latest/clickstream-analysis/)
9+
- `twitter`: [Fast Twitter events processing](https://www.risingwave.dev/docs/latest/fast-twitter-events-processing/)
10+
- `twitter-pulsar`: [Tutorial: Pulsar + RisingWave for Fast Twitter Event Processing](https://www.risingwave.com/blog/tutorial-pulsar-risingwave-for-fast-twitter-events-processing/)
11+
- `live-stream`: [Live stream metrics analysis](https://www.risingwave.dev/docs/latest/live-stream-metrics-analysis/)
12+
13+
## Demo Runnability Testing
14+
15+
The demos listed above will all run through a series of tests when each PR is merged, including:
16+
17+
- Run the queries mentioned in the demos.
18+
- Ingest the data in various formats, including Protobuf, Avro, and JSON. Each format will be tested individually.
19+
- For each demo test, we check if the sources and MVs have successfully ingested data, meaning that they should have >0 records.
20+
21+
## Workload Generator
22+
23+
The workloads presented in the demos are produced by a golang program in `/datagen`. You can get this tool in multiple ways:
24+
25+
- Download pre-built binaries from [Releases](https://github.com/risingwavelabs/risingwave-demo/releases)
26+
- Pull the latest docker image via `docker pull ghcr.io/risingwavelabs/demo-datagen:v1.0.9`.
27+
- Build the binary from source:
28+
```sh
29+
cd datagen && go build
30+
```
31+
32+
To use this tool, you can run the following command:
33+
34+
```sh
35+
./datagen --mode clickstream --qps 10 kafka --brokers 127.0.0.1:57801
36+
```
37+
38+
or
39+
40+
```sh
41+
./datagen --mode ecommerce --qps 10000000 postgres --port 6875 --user materialize --db materialize
42+
```
43+
44+
- `--mode clickstream` indicates that it will produce random clickstream data.
45+
- `--qps 10` sets a QPS limit to 10.
46+
- `kafka | postgres` chooses the destination. For kafka, you will need to specify the brokers.

‎integration_tests/ad-click/create_mv.sql

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
-- The number of clicks on the ad within one minute after the ad was shown.
2+
create materialized view m_click_statistic as
3+
select
4+
count(user_id) as clicks_count,
5+
ad_id
6+
from
7+
ad_source
8+
where
9+
click_timestamp is not null
10+
and impression_timestamp < click_timestamp
11+
and impression_timestamp + interval '1' minute >= click_timestamp
12+
group by
13+
ad_id;

‎integration_tests/ad-click/create_source.sql

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
-- impression_timestamp: The time when the ad was shown.
2+
-- click_timestamp: The time when the ad was clicked.
3+
create source ad_source (
4+
user_id bigint,
5+
ad_id bigint,
6+
click_timestamp timestamptz,
7+
impression_timestamp timestamptz
8+
) with (
9+
connector = 'kafka',
10+
topic = 'ad_clicks',
11+
properties.bootstrap.server = 'message_queue:29092',
12+
scan.startup.mode = 'earliest'
13+
) row format json;

‎integration_tests/ad-click/data_check

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
ad_source,m_click_statistic

‎integration_tests/ad-click/docker-compose.yml

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
---
2+
version: "3"
3+
services:
4+
compactor-0:
5+
extends:
6+
file: ../../docker/docker-compose.yml
7+
service: compactor-0
8+
compute-node-0:
9+
extends:
10+
file: ../../docker/docker-compose.yml
11+
service: compute-node-0
12+
etcd-0:
13+
extends:
14+
file: ../../docker/docker-compose.yml
15+
service: etcd-0
16+
frontend-node-0:
17+
extends:
18+
file: ../../docker/docker-compose.yml
19+
service: frontend-node-0
20+
grafana-0:
21+
extends:
22+
file: ../../docker/docker-compose.yml
23+
service: grafana-0
24+
meta-node-0:
25+
extends:
26+
file: ../../docker/docker-compose.yml
27+
service: meta-node-0
28+
minio-0:
29+
extends:
30+
file: ../../docker/docker-compose.yml
31+
service: minio-0
32+
prometheus-0:
33+
extends:
34+
file: ../../docker/docker-compose.yml
35+
service: prometheus-0
36+
message_queue:
37+
extends:
38+
file: ../../docker/docker-compose.yml
39+
service: message_queue
40+
datagen:
41+
build: ../datagen
42+
depends_on: [message_queue]
43+
command:
44+
- /bin/sh
45+
- -c
46+
- /datagen --mode ad-click --qps 2 kafka --brokers message_queue:29092
47+
restart: always
48+
container_name: datagen
49+
volumes:
50+
compute-node-0:
51+
external: false
52+
etcd-0:
53+
external: false
54+
grafana-0:
55+
external: false
56+
minio-0:
57+
external: false
58+
prometheus-0:
59+
external: false
60+
message_queue:
61+
external: false
62+
name: risingwave-compose

‎integration_tests/ad-click/query.sql

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
select
2+
*
3+
from
4+
m_click_statistic
5+
limit
6+
10;

‎integration_tests/ad-ctr/create_mv.sql

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
CREATE MATERIALIZED VIEW ad_ctr AS
2+
SELECT
3+
ad_clicks.ad_id AS ad_id,
4+
ad_clicks.clicks_count :: NUMERIC / ad_impressions.impressions_count AS ctr
5+
FROM
6+
(
7+
SELECT
8+
ad_impression.ad_id AS ad_id,
9+
COUNT(*) AS impressions_count
10+
FROM
11+
ad_impression
12+
GROUP BY
13+
ad_id
14+
) AS ad_impressions
15+
JOIN (
16+
SELECT
17+
ai.ad_id,
18+
COUNT(*) AS clicks_count
19+
FROM
20+
ad_click AS ac
21+
LEFT JOIN ad_impression AS ai ON ac.bid_id = ai.bid_id
22+
GROUP BY
23+
ai.ad_id
24+
) AS ad_clicks ON ad_impressions.ad_id = ad_clicks.ad_id;
25+
26+
CREATE MATERIALIZED VIEW ad_ctr_5min AS
27+
SELECT
28+
ac.ad_id AS ad_id,
29+
ac.clicks_count :: NUMERIC / ai.impressions_count AS ctr,
30+
ai.window_end AS window_end
31+
FROM
32+
(
33+
SELECT
34+
ad_id,
35+
COUNT(*) AS impressions_count,
36+
window_end
37+
FROM
38+
TUMBLE(
39+
ad_impression,
40+
impression_timestamp,
41+
INTERVAL '5' MINUTE
42+
)
43+
GROUP BY
44+
ad_id,
45+
window_end
46+
) AS ai
47+
JOIN (
48+
SELECT
49+
ai.ad_id,
50+
COUNT(*) AS clicks_count,
51+
ai.window_end AS window_end
52+
FROM
53+
TUMBLE(ad_click, click_timestamp, INTERVAL '5' MINUTE) AS ac
54+
INNER JOIN TUMBLE(
55+
ad_impression,
56+
impression_timestamp,
57+
INTERVAL '5' MINUTE
58+
) AS ai ON ai.bid_id = ac.bid_id
59+
AND ai.window_end = ac.window_end
60+
GROUP BY
61+
ai.ad_id,
62+
ai.window_end
63+
) AS ac ON ai.ad_id = ac.ad_id
64+
AND ai.window_end = ac.window_end;

‎integration_tests/ad-ctr/create_source.sql

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
CREATE SOURCE ad_impression (
2+
bid_id BIGINT,
3+
ad_id BIGINT,
4+
impression_timestamp TIMESTAMPTZ
5+
) WITH (
6+
connector = 'kafka',
7+
topic = 'ad_impression',
8+
properties.bootstrap.server = 'message_queue:29092',
9+
scan.startup.mode = 'earliest'
10+
) ROW FORMAT JSON;
11+
12+
CREATE SOURCE ad_click (
13+
bid_id BIGINT,
14+
click_timestamp TIMESTAMPTZ
15+
) WITH (
16+
connector = 'kafka',
17+
topic = 'ad_click',
18+
properties.bootstrap.server = 'message_queue:29092',
19+
scan.startup.mode = 'earliest'
20+
) ROW FORMAT JSON;

‎integration_tests/ad-ctr/data_check

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
ad_impression,ad_click,ad_ctr,ad_ctr_5min

0 commit comments

Comments
 (0)