test: Add nexmark test, kafka test for backwards compatibility testing #10413

kwannoel · 2023-06-19T11:53:52Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Closes #9973

Changes to test framework

Major refactoring of backwards compatibility test. It now uses sqllogictest for the testing part, rather than handrolled test framework.
Nexmark test (without source, only RW tables)
Kafka test (with kafka source)
We provide a script to make it easy to run backwards compat tests locally, see the developer guide.
Tpch test.

Other improvements

We fix ./risedev k on full cluster with kafka. Previously kafka did not exit cleanly. The draw back is if kafka is enabled, we need to wait 5s. We could also use jps but I don't want to include an external dependency on risedev k.
Always fetch pre-built binaries for the backwards compat test. For old version, we just download the risingwave binary from github. For current version, just from the buildkite build step.
When we restart the cluster, we restart ALL components, not just RW.
This is because we use risedev to manage components, and it is easier to just kill everything, rather than restart a subset of components.

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

Documentation

My PR contains user-facing changes.

Click here for Documentation

Types of user-facing changes

Please keep the types that apply to your changes, and remove the others.

Installation and deployment
Connector (sources & sinks)
SQL commands, functions, and operators
RisingWave cluster configuration changes
Other (please specify in the release note below)

Release note

codecov · 2023-06-19T12:18:49Z

Codecov Report

Merging #10413 (8f821fc) into main (52192e6) will increase coverage by 0.00%.
Report is 1 commits behind head on main.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main   #10413   +/-   ##
=======================================
  Coverage   69.76%   69.76%           
=======================================
  Files        1404     1404           
  Lines      235002   235002           
=======================================
+ Hits       163953   163955    +2     
+ Misses      71049    71047    -2

Flag	Coverage Δ
rust	`69.76% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 4 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

ci/scripts/backwards-compat-test.sh

github-actions · 2023-08-20T01:48:18Z

This PR has been open for 60 days with no activity. Could you please update the status? Feel free to ping a reviewer if you are waiting for review.

kwannoel · 2023-09-05T16:13:43Z

Just testing it on CI.

lmatz

LGTM, thanks!

ci/scripts/backwards-compat-test.sh

fuyufjh

The rest LGTM

fuyufjh · 2023-09-06T07:52:12Z

backwards-compat-tests/README.md

Is there any reason to place is as a separated folder (backwards-compat-tests/)?

The previous convention seems to be placing

slt tests in e2e_test/

run_local.sh in scripts/

I prefer all backwards-compat-tests components:

utils.sh

run_local.sh

slt files

To be all placed together, rather than split it into different directories for maintainability reasons.

For run_local.sh I will add a risedev backwards-compat-test for it to make it discoverable.

For run_local.sh I will add a risedev backwards-compat-test for it to make it discoverable.

Done.

backwards-compat-tests/scripts/run_local.sh

commit e097adc Author: StrikeW <[email protected]> Date: Fri Sep 8 13:16:15 2023 +0800 fix: mysql-sink integration test (risingwavelabs#12161) commit d972225 Author: congyi wang <[email protected]> Date: Fri Sep 8 12:16:06 2023 +0800 chore(object store): change streaming write buffer size to 16MB for gcs (risingwavelabs#12150) commit 8bd524b Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Sep 8 11:36:49 2023 +0800 chore(deps): Bump xxhash-rust from 0.8.6 to 0.8.7 (risingwavelabs#12159) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit c924c37 Author: Yufan Song <[email protected]> Date: Thu Sep 7 10:55:57 2023 -0700 feat(stream): add multiple server url feature into nats, delete useless field (risingwavelabs#12138) commit 471aa2b Author: William Wen <[email protected]> Date: Thu Sep 7 22:31:44 2023 +0800 refactor(source): specify cdc generic type parameter for different cdc source (risingwavelabs#12109) commit 7d940cf Author: Xinhao Xu <[email protected]> Date: Thu Sep 7 21:22:06 2023 +0800 feat(sink): Support cassandra sink (risingwavelabs#11878) Co-authored-by: Patrick Huang <[email protected]> commit cc3faf6 Author: Yuhao Su <[email protected]> Date: Thu Sep 7 18:58:44 2023 +0800 feat(watermark): handle watermark in project_set (risingwavelabs#12128) commit 4bb7c66 Author: Li0k <[email protected]> Date: Thu Sep 7 18:44:17 2023 +0800 feat(storage): reduce group deltas for l0 task (risingwavelabs#12085) commit b2dc4eb Author: Eric Fu <[email protected]> Date: Thu Sep 7 18:33:01 2023 +0800 refactor(metrics): unify storage_metric_level & streaming_metric_level to 'metrics_level' (risingwavelabs#12099) commit 5f75f9b Author: Li0k <[email protected]> Date: Thu Sep 7 17:52:33 2023 +0800 feat(storage): sepearte intra picker (risingwavelabs#12147) commit f1672f7 Author: Richard Chien <[email protected]> Date: Thu Sep 7 16:26:22 2023 +0800 fix(eowc): fix SortBuffer when there are multiple records having same timestamp (risingwavelabs#12146) Signed-off-by: Richard Chien <[email protected]> commit 183a98b Author: Xu <[email protected]> Date: Thu Sep 7 04:23:15 2023 -0400 feat(expr): add `array_max` (risingwavelabs#12100) Co-authored-by: Runji Wang <[email protected]> commit 893f636 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu Sep 7 08:06:35 2023 +0000 chore(deps): Bump chrono from 0.4.26 to 0.4.29 (risingwavelabs#12136) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Runji Wang <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Runji Wang <[email protected]> commit 3383291 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu Sep 7 15:49:07 2023 +0800 chore(deps): Bump regex-automata from 0.3.7 to 0.3.8 (risingwavelabs#12134) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 5a8866d Author: Li0k <[email protected]> Date: Thu Sep 7 15:28:56 2023 +0800 refactor(storage): seperate validator logic from picker (risingwavelabs#11984) commit b578db6 Author: Bugen Zhao <[email protected]> Date: Thu Sep 7 14:35:14 2023 +0800 fix(parser): more error reporting with locations (risingwavelabs#12129) Signed-off-by: Bugen Zhao <[email protected]> commit 3ee8a87 Author: Bugen Zhao <[email protected]> Date: Thu Sep 7 14:09:08 2023 +0800 refactor(meta): refine `pause`/`resume` response (risingwavelabs#12079) Signed-off-by: Bugen Zhao <[email protected]> commit f16809d Author: xiangjinwu <[email protected]> Date: Thu Sep 7 13:58:38 2023 +0800 fix(expr): parse `timestamptz` without seconds but with offset (risingwavelabs#12084) Co-authored-by: StrikeW <[email protected]> commit 01ce1bb Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu Sep 7 11:56:53 2023 +0800 chore(deps): Bump num-bigint from 0.4.3 to 0.4.4 (risingwavelabs#12137) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit d79113e Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu Sep 7 10:42:13 2023 +0800 chore(deps): Bump sysinfo from 0.29.9 to 0.29.10 (risingwavelabs#12135) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit ff4ef3a Author: Noel Kwan <[email protected]> Date: Thu Sep 7 10:03:57 2023 +0800 fix(ci): fix backwards compat with `ci-release` profile (risingwavelabs#12139) commit 9c98d26 Author: Xu <[email protected]> Date: Wed Sep 6 14:31:10 2023 -0400 feat(expr): add `array_min` (risingwavelabs#12071) Signed-off-by: Runji Wang <[email protected]> Co-authored-by: Runji Wang <[email protected]> commit 084daa8 Author: Noel Kwan <[email protected]> Date: Wed Sep 6 23:07:24 2023 +0800 test: Add nexmark test, kafka test for backwards compatibility testing (risingwavelabs#10413) commit 3c19064 Author: Yufan Song <[email protected]> Date: Wed Sep 6 07:07:40 2023 -0700 feat(stream): Add recovery feature for nats source (risingwavelabs#12073) Signed-off-by: tabVersion <[email protected]> Co-authored-by: tabVersion <[email protected]> commit 52192e6 Author: Croxx <[email protected]> Date: Wed Sep 6 19:18:32 2023 +0800 refactor: refactor file cache metrics build, bump foyer (risingwavelabs#12121) Signed-off-by: MrCroxx <[email protected]> commit da89875 Author: William Wen <[email protected]> Date: Wed Sep 6 18:27:52 2023 +0800 feat(log-store): refine log store trait and add update vnode bitmap in reader (risingwavelabs#11959) commit e64a075 Author: William Wen <[email protected]> Date: Wed Sep 6 18:21:39 2023 +0800 refactor(meta): introduce meta store ref and avoid pass generic parameter (risingwavelabs#12114) commit b3a4195 Author: Yufan Song <[email protected]> Date: Tue Sep 5 23:57:14 2023 -0700 fix(stream): remove useless stream in nats sink (risingwavelabs#12107) commit 68087f3 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed Sep 6 06:24:42 2023 +0000 chore(deps): bump flate2 from 1.0.26 to 1.0.27 (risingwavelabs#12103) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Bugen Zhao <[email protected]> commit e3f3d47 Author: Shanicky Chen <[email protected]> Date: Wed Sep 6 14:07:14 2023 +0800 fix: temporarily remove the assert checking for split migration (risingwavelabs#12117) commit 61ef35b Author: Li0k <[email protected]> Date: Wed Sep 6 14:04:14 2023 +0800 chore(storage): remove compactor workload (risingwavelabs#12005) commit a76e519 Author: Dylan <[email protected]> Date: Wed Sep 6 13:17:22 2023 +0800 refactor(java_binding): extract java_binding to jni_core (risingwavelabs#12108) Co-authored-by: Eric Fu <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> commit 99ecd5d Author: Noel Kwan <[email protected]> Date: Wed Sep 6 12:48:47 2023 +0800 fix(streaming): handle missing `cache_policy` field in `OverWindow` (risingwavelabs#12110) commit 50792cc Author: congyi wang <[email protected]> Date: Wed Sep 6 12:15:17 2023 +0800 refactor(compaction): remove meta client from compactor context (risingwavelabs#12020) commit b5dd428 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed Sep 6 12:02:52 2023 +0800 chore(deps): bump openssl-sys from 0.9.92 to 0.9.93 (risingwavelabs#12106) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 1c87421 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed Sep 6 12:02:49 2023 +0800 chore(deps): bump walkdir from 2.3.3 to 2.4.0 (risingwavelabs#12101) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 8c0da22 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed Sep 6 12:02:46 2023 +0800 chore(deps): bump deranged from 0.3.7 to 0.3.8 (risingwavelabs#12102) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 9ccdbe3 Author: zwang28 <[email protected]> Date: Tue Sep 5 21:26:31 2023 +0800 chore(rpc): set message size to max for stream client (risingwavelabs#12082) commit 4f50903 Author: Eric Fu <[email protected]> Date: Tue Sep 5 20:42:25 2023 +0800 refactor(metrics): support streaming metrics level (risingwavelabs#11994) commit 74c96fc Author: Richard Chien <[email protected]> Date: Tue Sep 5 19:47:23 2023 +0800 perf(over window): fix over window performance degradation after risingwavelabs#11576 (risingwavelabs#12093) Signed-off-by: Richard Chien <[email protected]> commit fdaf30e Author: Richard Chien <[email protected]> Date: Tue Sep 5 18:59:16 2023 +0800 chore: add over window executor cache metrics (risingwavelabs#12092) Signed-off-by: Richard Chien <[email protected]> commit ec0027e Author: William Wen <[email protected]> Date: Tue Sep 5 18:22:38 2023 +0800 refactor(source): implement the common split reader as a generic function (risingwavelabs#12059) commit 5cf0351 Author: Shanicky Chen <[email protected]> Date: Tue Sep 5 18:09:27 2023 +0800 feat: modify the scale command to support both horizontal and vertical scaling. (risingwavelabs#12087) commit 69bde53 Author: Bohan Zhang <[email protected]> Date: Tue Sep 5 18:03:18 2023 +0800 feat: HA for schema registry (risingwavelabs#11982) Signed-off-by: tabVersion <[email protected]> commit 477bfd4 Author: StrikeW <[email protected]> Date: Tue Sep 5 16:57:55 2023 +0800 fix(jdbc-sink): quote identifier to handle case-sensitive case (risingwavelabs#12078) commit aea50f5 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue Sep 5 07:29:58 2023 +0000 chore(deps): bump async-recursion from 1.0.4 to 1.0.5 (risingwavelabs#12069) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 579405e Author: Bugen Zhao <[email protected]> Date: Tue Sep 5 14:43:46 2023 +0800 feat(grafana): report local barrier manager progress (risingwavelabs#12001) Signed-off-by: Bugen Zhao <[email protected]> commit e4b9a08 Author: Bugen Zhao <[email protected]> Date: Tue Sep 5 14:39:58 2023 +0800 refactor(streaming): better traces for development (risingwavelabs#12024) Signed-off-by: Bugen Zhao <[email protected]> commit 65fc959 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue Sep 5 06:37:26 2023 +0000 chore(deps): bump tracing-opentelemetry from 0.20.0 to 0.21.0 (risingwavelabs#12010) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Runji Wang <[email protected]> commit aadc605 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue Sep 5 14:17:42 2023 +0800 chore(deps): bump google-cloud-pubsub from 0.19.0 to 0.20.0 (risingwavelabs#12066) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit a5ef008 Author: Noel Kwan <[email protected]> Date: Tue Sep 5 14:05:59 2023 +0800 feat(storage): refactor `LocalStateStore::init` and `StateTable::init` (risingwavelabs#12050) commit ff623ee Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue Sep 5 03:49:12 2023 +0000 chore(deps): bump syn from 2.0.29 to 2.0.31 (risingwavelabs#12067) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit f95d972 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue Sep 5 03:49:01 2023 +0000 chore(deps): bump memchr from 2.6.2 to 2.6.3 (risingwavelabs#12068) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 3f5a6c8 Author: Tao Wu <[email protected]> Date: Tue Sep 5 11:23:49 2023 +0800 feat: support information_schema.views (risingwavelabs#12045) commit 999b9d2 Author: wu <[email protected]> Date: Mon Sep 4 18:10:08 2023 +0800 feat: auto rebuild source reader in executor (risingwavelabs#11993) Co-authored-by: Bohan Zhang <[email protected]> commit fa8b5e3 Author: Shanicky Chen <[email protected]> Date: Mon Sep 4 17:10:38 2023 +0800 feat: add `target_parallelism_per_worker` field for scaling (risingwavelabs#11945) commit deac61e Author: Croxx <[email protected]> Date: Mon Sep 4 17:07:26 2023 +0800 feat(hummock): refill cache before apply version delta (risingwavelabs#11950) Signed-off-by: MrCroxx <[email protected]> commit 2ea66b8 Author: Bohan Zhang <[email protected]> Date: Mon Sep 4 16:15:30 2023 +0800 chore: refine error message on unhandled error in kinesis (risingwavelabs#12055) Signed-off-by: tabVersion <[email protected]> commit dcfbfb5 Author: August <[email protected]> Date: Mon Sep 4 14:50:29 2023 +0800 feat: support for migration by new parallel units (risingwavelabs#11892) commit 0d4da3b Author: Bohan Zhang <[email protected]> Date: Mon Sep 4 14:10:49 2023 +0800 fix: reject kinesis source if the start up mode is not sequence number but sequence number is provided (risingwavelabs#12048) Signed-off-by: Little-Wallace <[email protected]>

#10413)

kwannoel added the ci/run-backwards-compat-tests Run backwards compatibility tests in your PR. label Jun 19, 2023

github-actions bot added the type/chore label Jun 19, 2023

kwannoel mentioned this pull request Jun 19, 2023

fix(risedevtool): try download kafka from archive.apache if downloads.apache does not work #10411

Merged

7 tasks

kwannoel changed the title ~~chore(ci): enable kafka in backwards compat test.~~ chore(ci): enable kafka in backwards compat test Jun 20, 2023

kwannoel commented Jun 20, 2023

View reviewed changes

ci/scripts/backwards-compat-test.sh Outdated Show resolved Hide resolved

kwannoel marked this pull request as draft June 20, 2023 03:44

kwannoel mentioned this pull request Jul 3, 2023

Tracking: Automating Kernel backwards compatibility tests #9973

Closed

9 tasks

github-actions bot added the no-pr-activity label Aug 20, 2023

kwannoel force-pushed the kwannoel/enable-kafka branch from 018114f to 36064ff Compare August 21, 2023 08:45

github-actions bot removed the no-pr-activity label Aug 22, 2023

kwannoel force-pushed the kwannoel/enable-kafka branch from 36064ff to 9b1631b Compare September 5, 2023 02:07

kwannoel marked this pull request as ready for review September 5, 2023 16:13

kwannoel requested a review from a team as a code owner September 5, 2023 16:43

kwannoel force-pushed the kwannoel/enable-kafka branch from 622fa77 to f0d7c30 Compare September 5, 2023 16:45

kwannoel changed the title ~~chore(ci): enable kafka in backwards compat test~~ test: Add nexmark test, kafka test Sep 6, 2023

github-actions bot added A-test Area: Testing framework & misc tests not belonging to any specific component. and removed type/chore labels Sep 6, 2023

kwannoel changed the title ~~test: Add nexmark test, kafka test~~ test: Add nexmark test, kafka test for backwards compatibility testing Sep 6, 2023

kwannoel force-pushed the kwannoel/enable-kafka branch 2 times, most recently from 45880c6 to 073f9ee Compare September 6, 2023 06:52

kwannoel requested review from lmatz, fuyufjh and BugenZhao September 6, 2023 07:02

lmatz approved these changes Sep 6, 2023

View reviewed changes

ci/scripts/backwards-compat-test.sh Outdated Show resolved Hide resolved

fuyufjh approved these changes Sep 6, 2023

View reviewed changes

kwannoel added the ci/run-main-cron label Sep 6, 2023

enable kafka

95c2686

kwannoel added 7 commits September 6, 2023 20:42

fix

46733ad

fix

13e7975

fix run_local

f514660

fix

3e45915

fix

3cfbc28

use get_rw_versions in ci

61e1892

cleanup

faa5f23

kwannoel force-pushed the kwannoel/enable-kafka branch from f38a446 to faa5f23 Compare September 6, 2023 12:42

doc

c180eb8

kwannoel removed the ci/run-main-cron label Sep 6, 2023

kwannoel added 10 commits September 6, 2023 21:01

try fetch

577799b

debug

1b130bd

only run backwards-compat-test

be44212

try git fetch all?

5026a59

make sure to match on the exact prefix

d2d028e

dont block stdin

100bb80

use git branch -r to list remote branches

182656d

grep only origin

4e4e418

fix

377057f

rerun full pr

327902e

kwannoel enabled auto-merge September 6, 2023 13:42

no need github token

8f821fc

kwannoel added this pull request to the merge queue Sep 6, 2023

Merged via the queue into main with commit 084daa8 Sep 6, 2023

kwannoel deleted the kwannoel/enable-kafka branch September 6, 2023 15:42

This was referenced Sep 11, 2023

risedev: provide kafka persistence property over cluster teardown #9437

Closed

chore(ci): bump backwards compat tests to 1.0 #11634

Closed

Li0k pushed a commit that referenced this pull request Sep 15, 2023

test: Add nexmark test, kafka test for backwards compatibility testing (

8e3117c

#10413)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test: Add nexmark test, kafka test for backwards compatibility testing #10413

test: Add nexmark test, kafka test for backwards compatibility testing #10413

Uh oh!

kwannoel commented Jun 19, 2023 •

edited

Loading

Uh oh!

codecov bot commented Jun 19, 2023 •

edited

Loading

Uh oh!

Uh oh!

github-actions bot commented Aug 20, 2023

Uh oh!

kwannoel commented Sep 5, 2023

Uh oh!

lmatz left a comment

Uh oh!

Uh oh!

fuyufjh left a comment

Uh oh!

fuyufjh Sep 6, 2023

Uh oh!

kwannoel Sep 6, 2023 •

edited

Loading

Uh oh!

kwannoel Sep 6, 2023

Uh oh!

Uh oh!

Uh oh!

test: Add nexmark test, kafka test for backwards compatibility testing #10413

test: Add nexmark test, kafka test for backwards compatibility testing #10413

Uh oh!

Conversation

kwannoel commented Jun 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's changed and what's your intention?

Changes to test framework

Other improvements

Checklist

Documentation

Types of user-facing changes

Release note

Uh oh!

codecov bot commented Jun 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

github-actions bot commented Aug 20, 2023

Uh oh!

kwannoel commented Sep 5, 2023

Uh oh!

lmatz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fuyufjh left a comment

Choose a reason for hiding this comment

Uh oh!

fuyufjh Sep 6, 2023

Choose a reason for hiding this comment

Uh oh!

kwannoel Sep 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kwannoel Sep 6, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kwannoel commented Jun 19, 2023 •

edited

Loading

codecov bot commented Jun 19, 2023 •

edited

Loading

kwannoel Sep 6, 2023 •

edited

Loading