Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: prepare v0.13.0 #2549

Merged
merged 1 commit into from
Apr 7, 2025
Merged

chore: prepare v0.13.0 #2549

merged 1 commit into from
Apr 7, 2025

Conversation

mxinden
Copy link
Collaborator

@mxinden mxinden commented Apr 1, 2025

Bugfixes I would like to wait for:

Any objections? Anything else to include?

@larseggert
Copy link
Collaborator

Maybe #2470 and/or #2505?

Copy link

github-actions bot commented Apr 1, 2025

Failed Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

Copy link

github-actions bot commented Apr 1, 2025

Benchmark results

Performance differences relative to d947c94.

1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client: 💚 Performance has improved.
       time:   [700.55 ms 705.48 ms 710.33 ms]
       thrpt:  [140.78 MiB/s 141.75 MiB/s 142.75 MiB/s]
change:
       time:   [-3.6796% -2.7335% -1.7463%] (p = 0.00 < 0.05)
       thrpt:  [+1.7773% +2.8103% +3.8202%]
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client: No change in performance detected.
       time:   [348.48 ms 350.23 ms 351.97 ms]
       thrpt:  [28.412 Kelem/s 28.552 Kelem/s 28.696 Kelem/s]
change:
       time:   [-0.8573% -0.1160% +0.6344%] (p = 0.76 > 0.05)
       thrpt:  [-0.6304% +0.1161% +0.8647%]
1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client: No change in performance detected.
       time:   [25.686 ms 25.843 ms 25.998 ms]
       thrpt:  [38.465  elem/s 38.696  elem/s 38.932  elem/s]
change:
       time:   [-1.7789% -0.8794% +0.0043%] (p = 0.05 > 0.05)
       thrpt:  [-0.0043% +0.8872% +1.8111%]
1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client: 💚 Performance has improved.
       time:   [2.2949 s 2.3162 s 2.3376 s]
       thrpt:  [42.780 MiB/s 43.175 MiB/s 43.575 MiB/s]
change:
       time:   [-5.2546% -4.0278% -2.8255%] (p = 0.00 < 0.05)
       thrpt:  [+2.9076% +4.1968% +5.5461%]
decode 4096 bytes, mask ff: Change within noise threshold.
       time:   [12.069 µs 12.099 µs 12.137 µs]
       change: [+0.0380% +0.5587% +1.0865%] (p = 0.03 < 0.05)

Found 18 outliers among 100 measurements (18.00%)
6 (6.00%) low severe
4 (4.00%) low mild
1 (1.00%) high mild
7 (7.00%) high severe

decode 1048576 bytes, mask ff: 💔 Performance has regressed.
       time:   [3.0747 ms 3.0836 ms 3.0935 ms]
       change: [+3.8959% +4.3469% +4.8098%] (p = 0.00 < 0.05)

Found 8 outliers among 100 measurements (8.00%)
1 (1.00%) high mild
7 (7.00%) high severe

decode 4096 bytes, mask 7f: Change within noise threshold.
       time:   [20.207 µs 20.270 µs 20.335 µs]
       change: [+0.6421% +1.0114% +1.3731%] (p = 0.00 < 0.05)

Found 25 outliers among 100 measurements (25.00%)
3 (3.00%) low severe
2 (2.00%) low mild
1 (1.00%) high mild
19 (19.00%) high severe

decode 1048576 bytes, mask 7f: 💔 Performance has regressed.
       time:   [5.2538 ms 5.2669 ms 5.2805 ms]
       change: [+9.1750% +9.6070% +10.049%] (p = 0.00 < 0.05)

Found 15 outliers among 100 measurements (15.00%)
15 (15.00%) high severe

decode 4096 bytes, mask 3f: 💔 Performance has regressed.
       time:   [7.0161 µs 7.0455 µs 7.0823 µs]
       change: [+10.386% +11.250% +12.392%] (p = 0.00 < 0.05)

Found 13 outliers among 100 measurements (13.00%)
2 (2.00%) low severe
1 (1.00%) low mild
3 (3.00%) high mild
7 (7.00%) high severe

decode 1048576 bytes, mask 3f: 💚 Performance has improved.
       time:   [1.7908 ms 1.7964 ms 1.8034 ms]
       change: [-17.179% -16.767% -16.352%] (p = 0.00 < 0.05)

Found 6 outliers among 100 measurements (6.00%)
1 (1.00%) high mild
5 (5.00%) high severe

1 streams of 1 bytes/multistream: No change in performance detected.
       time:   [71.474 µs 71.678 µs 71.886 µs]
       change: [-2.3706% -0.3158% +1.1986%] (p = 0.78 > 0.05)

Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

1000 streams of 1 bytes/multistream: Change within noise threshold.
       time:   [25.359 ms 25.403 ms 25.448 ms]
       change: [-0.4806% -0.2471% -0.0080%] (p = 0.04 < 0.05)
10000 streams of 1 bytes/multistream: No change in performance detected.
       time:   [1.7010 s 1.7030 s 1.7050 s]
       change: [-0.3082% -0.1366% +0.0223%] (p = 0.11 > 0.05)

Found 8 outliers among 100 measurements (8.00%)
2 (2.00%) low mild
6 (6.00%) high mild

1 streams of 1000 bytes/multistream: No change in performance detected.
       time:   [73.075 µs 74.125 µs 75.617 µs]
       change: [-2.6924% +0.1566% +3.0273%] (p = 0.92 > 0.05)

Found 6 outliers among 100 measurements (6.00%)
4 (4.00%) high mild
2 (2.00%) high severe

100 streams of 1000 bytes/multistream: Change within noise threshold.
       time:   [3.4123 ms 3.4195 ms 3.4276 ms]
       change: [+0.5495% +0.8292% +1.1091%] (p = 0.00 < 0.05)

Found 21 outliers among 100 measurements (21.00%)
21 (21.00%) high severe

1000 streams of 1000 bytes/multistream: Change within noise threshold.
       time:   [145.92 ms 146.00 ms 146.09 ms]
       change: [+0.0069% +0.0861% +0.1648%] (p = 0.03 < 0.05)

Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild

coalesce_acked_from_zero 1+1 entries: No change in performance detected.
       time:   [94.731 ns 95.019 ns 95.310 ns]
       change: [-0.7920% -0.1285% +0.5555%] (p = 0.73 > 0.05)

Found 13 outliers among 100 measurements (13.00%)
11 (11.00%) high mild
2 (2.00%) high severe

coalesce_acked_from_zero 3+1 entries: No change in performance detected.
       time:   [112.52 ns 112.86 ns 113.22 ns]
       change: [-0.6928% -0.2842% +0.1482%] (p = 0.21 > 0.05)

Found 15 outliers among 100 measurements (15.00%)
2 (2.00%) low mild
1 (1.00%) high mild
12 (12.00%) high severe

coalesce_acked_from_zero 10+1 entries: No change in performance detected.
       time:   [111.74 ns 112.00 ns 112.34 ns]
       change: [-0.8697% -0.2636% +0.4776%] (p = 0.44 > 0.05)

Found 21 outliers among 100 measurements (21.00%)
3 (3.00%) low severe
5 (5.00%) low mild
3 (3.00%) high mild
10 (10.00%) high severe

coalesce_acked_from_zero 1000+1 entries: No change in performance detected.
       time:   [93.970 ns 94.380 ns 94.822 ns]
       change: [-2.0716% -0.8335% +0.3671%] (p = 0.19 > 0.05)

Found 5 outliers among 100 measurements (5.00%)
3 (3.00%) high mild
2 (2.00%) high severe

RxStreamOrderer::inbound_frame(): 💚 Performance has improved.
       time:   [114.61 ms 114.67 ms 114.73 ms]
       change: [-2.2983% -2.2201% -2.1507%] (p = 0.00 < 0.05)

Found 20 outliers among 100 measurements (20.00%)
4 (4.00%) low severe
5 (5.00%) low mild
11 (11.00%) high mild

SentPackets::take_ranges: Change within noise threshold.
       time:   [8.3622 µs 8.5685 µs 8.7607 µs]
       change: [+0.9506% +4.3454% +7.9817%] (p = 0.01 < 0.05)

Found 16 outliers among 100 measurements (16.00%)
3 (3.00%) low severe
11 (11.00%) low mild
2 (2.00%) high severe

transfer/pacing-false/varying-seeds: Change within noise threshold.
       time:   [34.509 ms 34.570 ms 34.630 ms]
       change: [-2.8093% -2.5508% -2.3016%] (p = 0.00 < 0.05)

Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) low mild

transfer/pacing-true/varying-seeds: Change within noise threshold.
       time:   [34.612 ms 34.667 ms 34.722 ms]
       change: [-2.1063% -1.8915% -1.6701%] (p = 0.00 < 0.05)
transfer/pacing-false/same-seed: Change within noise threshold.
       time:   [34.330 ms 34.386 ms 34.443 ms]
       change: [-2.9999% -2.7929% -2.5835%] (p = 0.00 < 0.05)

Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) low mild

transfer/pacing-true/same-seed: Change within noise threshold.
       time:   [35.039 ms 35.086 ms 35.132 ms]
       change: [-3.1684% -2.9814% -2.7807%] (p = 0.00 < 0.05)

Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) low mild

Client/server transfer results

Performance differences relative to d947c94.

Transfer of 33554432 bytes over loopback, 30 runs. All unit-less numbers are in milliseconds.

Client Server CC Pacing Mean ± σ Min Max MiB/s ± σ Δ main Δ main
neqo neqo reno on 413.4 ± 37.6 374.4 588.8 77.4 ± 0.9 0.3 0.1%
neqo neqo reno 436.9 ± 137.4 366.5 1121.2 73.2 ± 0.2 -18.5 -4.1%
neqo neqo cubic on 413.9 ± 35.3 388.2 585.2 77.3 ± 0.9 -2.3 -0.5%
neqo neqo cubic 408.7 ± 19.8 375.7 484.0 78.3 ± 1.6 -1.4 -0.3%
google neqo reno on 758.0 ± 77.4 560.4 839.0 42.2 ± 0.4 -1.3 -0.2%
google neqo reno 756.3 ± 82.1 558.7 834.9 42.3 ± 0.4 -4.9 -0.6%
google neqo cubic on 754.9 ± 89.0 545.4 952.1 42.4 ± 0.4 -0.9 -0.1%
google neqo cubic 751.3 ± 82.7 567.9 857.8 42.6 ± 0.4 -4.0 -0.5%
google google 574.8 ± 38.7 545.3 756.3 55.7 ± 0.8 8.9 1.6%
neqo msquic reno on 265.9 ± 20.6 246.1 347.0 120.3 ± 1.6 -7.3 -2.7%
neqo msquic reno 264.0 ± 13.0 245.9 292.4 121.2 ± 2.5 -4.1 -1.5%
neqo msquic cubic on 265.6 ± 15.3 246.2 316.6 120.5 ± 2.1 -0.3 -0.1%
neqo msquic cubic 264.3 ± 12.8 244.0 293.8 121.1 ± 2.5 -6.6 -2.4%
msquic msquic 171.7 ± 21.5 149.7 225.6 186.4 ± 1.5 -9.2 -5.1%

⬇️ Download logs

@mxinden
Copy link
Collaborator Author

mxinden commented Apr 7, 2025

Maybe #2470 and/or #2505?

Both merged. @larseggert fine if I proceed here?

@larseggert
Copy link
Collaborator

Go for it.

@mxinden mxinden marked this pull request as ready for review April 7, 2025 14:43
@larseggert larseggert added this pull request to the merge queue Apr 7, 2025
Merged via the queue into mozilla:main with commit 00646f0 Apr 7, 2025
119 of 121 checks passed
@larseggert
Copy link
Collaborator

larseggert commented Apr 7, 2025

@mxinden we should also include #2568!

@mxinden
Copy link
Collaborator Author

mxinden commented Apr 8, 2025

For the record, downstream tracking issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1959128

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants