Skip to content

adding ORSet benchmarks #4990

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
May 21, 2021
Merged

adding ORSet benchmarks #4990

merged 7 commits into from
May 21, 2021

Conversation

Aaronontheweb
Copy link
Member

Working on #4956

@Aaronontheweb Aaronontheweb mentioned this pull request Apr 29, 2021
Closed
@Aaronontheweb
Copy link
Member Author

Baseline Performance Numbers

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.928 (2004/?/20H1)
AMD Ryzen 7 1700, 1 CPU, 16 logical and 8 physical cores
.NET Core SDK=5.0.201
  [Host]     : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  DefaultJob : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
Method NumElements NumNodes Mean Error StdDev Gen 0 Gen 1 Gen 2 Allocated
Should_add_node_to_ORSet 10 3 12.40 μs 0.150 μs 0.117 μs 1.0681 - - 4.4 KB
Should_add_elements_for_Same_node 10 3 28.58 μs 0.257 μs 0.228 μs 2.6245 - - 10.77 KB
Should_merge_in_new_Elements_from_other_nodes 10 3 11.38 μs 0.134 μs 0.118 μs 0.8087 - - 3.35 KB
Should_merge_in_removed_Elements_from_other_nodes 10 3 10.24 μs 0.167 μs 0.139 μs 0.8087 - - 3.36 KB
Should_merge_in_add_and_removed_Elements_from_other_nodes 10 3 29.11 μs 0.355 μs 0.315 μs 2.1362 - - 8.77 KB
Should_add_node_to_ORSet 10 10 52.77 μs 0.388 μs 0.363 μs 4.6387 - - 18.96 KB
Should_add_elements_for_Same_node 10 10 29.19 μs 0.384 μs 0.320 μs 2.6245 - - 10.77 KB
Should_merge_in_new_Elements_from_other_nodes 10 10 21.31 μs 0.212 μs 0.188 μs 1.3428 - - 5.52 KB
Should_merge_in_removed_Elements_from_other_nodes 10 10 21.42 μs 0.107 μs 0.095 μs 1.3428 - - 5.52 KB
Should_merge_in_add_and_removed_Elements_from_other_nodes 10 10 58.89 μs 0.699 μs 0.654 μs 3.7842 - - 15.58 KB
Should_add_node_to_ORSet 25 3 12.13 μs 0.142 μs 0.126 μs 1.0681 - - 4.4 KB
Should_add_elements_for_Same_node 25 3 83.29 μs 1.425 μs 1.750 μs 7.5684 - - 31.31 KB
Should_merge_in_new_Elements_from_other_nodes 25 3 11.55 μs 0.222 μs 0.304 μs 0.8087 - - 3.35 KB
Should_merge_in_removed_Elements_from_other_nodes 25 3 10.25 μs 0.191 μs 0.169 μs 0.8087 - - 3.36 KB
Should_merge_in_add_and_removed_Elements_from_other_nodes 25 3 29.30 μs 0.449 μs 0.420 μs 2.1362 - - 8.77 KB
Should_add_node_to_ORSet 25 10 53.99 μs 1.033 μs 1.269 μs 4.6387 - - 18.96 KB
Should_add_elements_for_Same_node 25 10 84.07 μs 1.468 μs 1.226 μs 7.6904 - - 31.69 KB
Should_merge_in_new_Elements_from_other_nodes 25 10 21.50 μs 0.157 μs 0.139 μs 1.3428 - - 5.52 KB
Should_merge_in_removed_Elements_from_other_nodes 25 10 21.60 μs 0.309 μs 0.258 μs 1.3428 - - 5.52 KB
Should_merge_in_add_and_removed_Elements_from_other_nodes 25 10 58.46 μs 0.372 μs 0.348 μs 3.7842 - - 15.58 KB

@Aaronontheweb
added VersionVectorBenchmark
c94a2b6 
Using this to help gauge ORSet / other CRDT merge performance
@Aaronontheweb
Copy link
Member Author

VersionVector Baseline Numbers

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.928 (2004/?/20H1)
AMD Ryzen 7 1700, 1 CPU, 16 logical and 8 physical cores
.NET Core SDK=5.0.201
  [Host]     : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  DefaultJob : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
Method ClockSize Iterations Mean Error StdDev Gen 0 Gen 1 Gen 2 Allocated
VectorClock_comparisons_should_compare_same 100 1000 125.352 ms 1.0077 ms 0.8414 ms 7750.0000 - - 31.3 MB
VectorClock_comparisons_should_compare_Before_last 100 1000 123.932 ms 0.7283 ms 0.6081 ms 7800.0000 - - 31.3 MB
VectorClock_comparisons_should_compare_After_last 100 1000 121.413 ms 1.0413 ms 0.8130 ms 7800.0000 - - 31.3 MB
VectorClock_comparisons_should_compare_Concurrent_last 100 1000 119.488 ms 0.3350 ms 0.2797 ms 7600.0000 - - 31.01 MB
VectorClock_comparisons_should_compare_Before_middle 100 1000 121.540 ms 0.6755 ms 0.5988 ms 7750.0000 - - 31.3 MB
VectorClock_comparisons_should_compare_After_middle 100 1000 121.492 ms 0.4260 ms 0.3326 ms 7800.0000 - - 31.3 MB
VectorClock_comparisons_should_compare_Concurrent_middle 100 1000 61.325 ms 0.2811 ms 0.2347 ms 4111.1111 - - 16.43 MB
VectorClock_comparisons_should_compare_notEquals_Before_Middle 100 1000 62.342 ms 0.8807 ms 0.8238 ms 4000.0000 - - 16.39 MB
VectorClock_comparisons_should_compare_notEquals_After_Middle 100 1000 2.894 ms 0.0119 ms 0.0106 ms 378.9063 - - 1.52 MB
VectorClock_comparisons_should_compare_notEquals_Concurrent_Middle 100 1000 2.871 ms 0.0518 ms 0.0485 ms 378.9063 - - 1.52 MB

@Aaronontheweb
Copy link
Member Author

Moved all comparisons inside VersionVector to use ValueTuple instead of KeyValuePair - same move we made to VectorClock

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.928 (2004/?/20H1)
AMD Ryzen 7 1700, 1 CPU, 16 logical and 8 physical cores
.NET Core SDK=5.0.201
  [Host]     : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  DefaultJob : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
Method ClockSize Iterations Mean Error StdDev Gen 0 Gen 1 Gen 2 Allocated
VectorClock_comparisons_should_compare_same 100 1000 66.345 ms 1.2651 ms 1.4569 ms 4875.0000 - - 19.53 MB
VectorClock_comparisons_should_compare_Before_last 100 1000 68.630 ms 0.5018 ms 0.4191 ms 4875.0000 - - 19.53 MB
VectorClock_comparisons_should_compare_After_last 100 1000 66.418 ms 0.6771 ms 0.5654 ms 4875.0000 - - 19.53 MB
VectorClock_comparisons_should_compare_Concurrent_last 100 1000 68.557 ms 1.3139 ms 1.4059 ms 4875.0000 - - 19.68 MB
VectorClock_comparisons_should_compare_Before_middle 100 1000 72.974 ms 1.2673 ms 0.9894 ms 4857.1429 - - 19.53 MB
VectorClock_comparisons_should_compare_After_middle 100 1000 68.830 ms 1.3581 ms 1.6167 ms 4875.0000 - - 19.53 MB
VectorClock_comparisons_should_compare_Concurrent_middle 100 1000 36.151 ms 0.7020 ms 0.9128 ms 2666.6667 - - 10.71 MB
VectorClock_comparisons_should_compare_notEquals_Before_Middle 100 1000 36.690 ms 0.7016 ms 0.8616 ms 2642.8571 - - 10.68 MB
VectorClock_comparisons_should_compare_notEquals_After_Middle 100 1000 2.654 ms 0.0508 ms 0.0586 ms 378.9063 - - 1.52 MB
VectorClock_comparisons_should_compare_notEquals_Concurrent_Middle 100 1000 2.637 ms 0.0408 ms 0.0735 ms 378.9063 - - 1.52 MB
  • Reduced memory on the most expensive compare operation by 37%
  • Improved throughput on comparison by 47%

@Aaronontheweb
Copy link
Member Author

Checking to see if VersionVector read-side performance improvements had any impact on ORSet

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.928 (2004/?/20H1)
AMD Ryzen 7 1700, 1 CPU, 16 logical and 8 physical cores
.NET Core SDK=5.0.201
  [Host]     : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
  DefaultJob : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
Method NumElements NumNodes Mean Error StdDev Gen 0 Gen 1 Gen 2 Allocated
Should_add_node_to_ORSet 25 10 41.90 μs 0.645 μs 0.603 μs 4.1504 - - 17.13 KB
Should_add_elements_for_Same_node 25 10 80.75 μs 0.917 μs 0.813 μs 7.5684 - - 31.06 KB
Should_merge_in_new_Elements_from_other_nodes 25 10 15.31 μs 0.302 μs 0.348 μs 1.0529 - - 4.37 KB
Should_merge_in_removed_Elements_from_other_nodes 25 10 14.44 μs 0.107 μs 0.089 μs 0.9918 - - 4.09 KB
Should_merge_in_add_and_removed_Elements_from_other_nodes 25 10 39.42 μs 0.311 μs 0.259 μs 2.8687 - - 11.84 KB

@Aaronontheweb Aaronontheweb mentioned this pull request May 21, 2021
Merged
@Aaronontheweb Aaronontheweb marked this pull request as ready for review May 21, 2021 18:28
Arkatufus
Arkatufus approved these changes May 21, 2021
View reviewed changes
Copy link
Contributor

@Arkatufus Arkatufus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks good to me.

@Aaronontheweb Aaronontheweb enabled auto-merge (squash) May 21, 2021 18:48
@Aaronontheweb Aaronontheweb disabled auto-merge May 21, 2021 18:51
@Aaronontheweb Aaronontheweb merged commit aa8a3de into akkadotnet:dev May 21, 2021
@Aaronontheweb Aaronontheweb deleted the benchmark/ddata-orset branch May 21, 2021 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Arkatufus Arkatufus Arkatufus approved these changes

Assignees
No one assigned
Labels
akka-ddata perf
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Aaronontheweb @Arkatufus