Skip to content

Commit b849956

Browse files
taglialatagliala
committed
Test difference among benchmark executions
According to the order of run, we have 14% difference among executions for `(a:, **kwargs)` ### GitHub main tested before local ``` $ bundle exec ruby benchmarks.rb Will BENCHMARK_GEMS: memo_wise-github-main (1.10.0) memo_wise-local (1.10.0.dev) ruby 3.3.5 (2024-09-03 revision ef084cc8f4) [x86_64-darwin23] Warming up -------------------------------------- memo_wise-github-main (1.10.0): (a:, **kwargs) 125.193k i/100ms memo_wise-local (1.10.0.dev): (a:, **kwargs) 143.649k i/100ms Calculating ------------------------------------- memo_wise-github-main (1.10.0): (a:, **kwargs) 1.146M (±23.0%) i/s (872.35 ns/i) - 5.508M in 5.067507s memo_wise-local (1.10.0.dev): (a:, **kwargs) 1.309M (±18.0%) i/s (764.21 ns/i) - 6.464M in 5.169169s Comparison: memo_wise-local (1.10.0.dev): (a:, **kwargs): 1308539.2 i/s memo_wise-github-main (1.10.0): (a:, **kwargs): 1146322.9 i/s - same-ish: difference falls within error ``` |Method arguments|`memo_wise-github-main` (1.10.0)| |--|--| |`(a:, **kwargs)`|1.14x| ### local tested before GitHub main ``` $ bundle exec ruby benchmarks.rb Will BENCHMARK_GEMS: memo_wise-local (1.10.0.dev) memo_wise-github-main (1.10.0) ruby 3.3.5 (2024-09-03 revision ef084cc8f4) [x86_64-darwin23] Warming up -------------------------------------- memo_wise-local (1.10.0.dev): (a:, **kwargs) 130.824k i/100ms memo_wise-github-main (1.10.0): (a:, **kwargs) 150.842k i/100ms Calculating ------------------------------------- memo_wise-local (1.10.0.dev): (a:, **kwargs) 1.184M (±23.6%) i/s (844.93 ns/i) - 5.756M in 5.139460s memo_wise-github-main (1.10.0): (a:, **kwargs) 1.381M (±17.8%) i/s (723.94 ns/i) - 6.637M in 5.026159s Comparison: memo_wise-github-main (1.10.0): (a:, **kwargs): 1381332.2 i/s memo_wise-local (1.10.0.dev): (a:, **kwargs): 1183526.0 i/s - same-ish: difference falls within error ``` |Method arguments|`memo_wise-github-main` (1.10.0)| |--|--| |`(a:, **kwargs)`|0.86x| Ref: panorama-ed#349
1 parent 40cb797 commit b849956

File tree

3 files changed

+3
-94
lines changed

3 files changed

+3
-94
lines changed

‎benchmarks/Gemfile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ git_source(:github) { |repo| "https://github.com/#{repo}.git" }
77
ruby ">= 2.7.7"
88

99
gem "benchmark-ips", "2.14.0"
10-
gem "gem_bench", "2.0.3"
10+
gem "gem_bench", "2.0.5"
1111

1212
# NOTE: Regarding `require: false` below
1313
# 1. GitHub version of MemoWise and the local source of MemoWise share a namespace
@@ -28,4 +28,4 @@ else
2828
gem "memoizer", "1.0.3"
2929
end
3030

31-
gem "memo_wise", github: "panorama-ed/memo_wise", branch: "main", require: false
31+
gem "memo_wise", github: "tagliala/memo_wise", branch: "chore/prefer-require-relative", require: false

‎benchmarks/benchmarks.rb

Lines changed: 0 additions & 91 deletions
Original file line numberDiff line numberDiff line change
@@ -26,36 +26,6 @@
2626
memoization_method: :memo_wise,
2727
},
2828
),
29-
GemBench::Jersey.new(
30-
gem_name: "alt_memery",
31-
trades: {
32-
"Memery" => "AltMemery"
33-
},
34-
metadata: {
35-
activation_code: "include AltMemery",
36-
memoization_method: :memoize,
37-
},
38-
),
39-
GemBench::Jersey.new(
40-
gem_name: "memoist3",
41-
trades: {
42-
"Memoist" => "MemoistThree"
43-
},
44-
metadata: {
45-
activation_code: "extend MemoistThree",
46-
memoization_method: :memoize,
47-
},
48-
),
49-
GemBench::Jersey.new(
50-
gem_name: "memoist",
51-
trades: {
52-
"Memoist" => "MemoistOne"
53-
},
54-
metadata: {
55-
activation_code: "extend MemoistOne",
56-
memoization_method: :memoize,
57-
},
58-
),
5929
].each(&:doff_and_don) # Copies, re-namespaces, and requires each gem.
6030

6131
# We've already installed the `memo_wise` version on the `main` branch from GitHub in the
@@ -119,11 +89,6 @@ def benchmark_name
11989
end
12090
benchmarked_gems.push(
12191
BenchmarkGem.new(MemoWise, "prepend MemoWise", :memo_wise, LOCAL_BENCHMARK_NAME),
122-
(BenchmarkGem.new(DDMemoize, "DDMemoize.activate(self)", :memoize, "ddmemoize") if defined?(DDMemoize)),
123-
(BenchmarkGem.new(Dry::Core, "include Dry::Core::Memoizable", :memoize, "dry-core") if defined?(Dry::Core)),
124-
(BenchmarkGem.new(Memery, "include Memery", :memoize, "memery") if defined?(Memery)),
125-
(BenchmarkGem.new(Memoized, "include Memoized", :memoize, "memoized") if defined?(Memoized)),
126-
(BenchmarkGem.new(Memoizer, "include Memoizer", :memoize, "memoizer") if defined?(Memoizer))
12792
)
12893
BENCHMARK_GEMS = benchmarked_gems.compact.shuffle
12994

@@ -189,69 +154,13 @@ def positional_splat_keyword_and_double_splat_args(a, *args, b:, **kwargs)
189154
# Each method within these benchmarks is initially run once to memoize the
190155
# result value, so our benchmark only tests memoized retrieval time.
191156
benchmark_lambdas = [
192-
lambda do |x, instance, benchmark_gem|
193-
instance.no_args
194-
195-
x.report("#{benchmark_gem.benchmark_name}: ()") do
196-
instance.no_args
197-
end
198-
end,
199-
lambda do |x, instance, benchmark_gem|
200-
instance.one_positional_arg(1)
201-
202-
x.report("#{benchmark_gem.benchmark_name}: (a)") do
203-
instance.one_positional_arg(1)
204-
end
205-
end,
206-
lambda do |x, instance, benchmark_gem|
207-
instance.positional_args(1, 2)
208-
209-
x.report("#{benchmark_gem.benchmark_name}: (a, b)") do
210-
instance.positional_args(1, 2)
211-
end
212-
end,
213-
lambda do |x, instance, benchmark_gem|
214-
instance.one_keyword_arg(a: 1)
215-
216-
x.report("#{benchmark_gem.benchmark_name}: (a:)") do
217-
instance.one_keyword_arg(a: 1)
218-
end
219-
end,
220-
lambda do |x, instance, benchmark_gem|
221-
instance.keyword_args(a: 1, b: 2)
222-
223-
x.report("#{benchmark_gem.benchmark_name}: (a:, b:)") do
224-
instance.keyword_args(a: 1, b: 2)
225-
end
226-
end,
227-
lambda do |x, instance, benchmark_gem|
228-
instance.positional_and_keyword_args(1, b: 2)
229-
230-
x.report("#{benchmark_gem.benchmark_name}: (a, b:)") do
231-
instance.positional_and_keyword_args(1, b: 2)
232-
end
233-
end,
234-
lambda do |x, instance, benchmark_gem|
235-
instance.positional_and_splat_args(1, 2)
236-
237-
x.report("#{benchmark_gem.benchmark_name}: (a, *args)") do
238-
instance.positional_and_splat_args(1, 2)
239-
end
240-
end,
241157
lambda do |x, instance, benchmark_gem|
242158
instance.keyword_and_double_splat_args(a: 1, b: 2)
243159

244160
x.report("#{benchmark_gem.benchmark_name}: (a:, **kwargs)") do
245161
instance.keyword_and_double_splat_args(a: 1, b: 2)
246162
end
247163
end,
248-
lambda do |x, instance, benchmark_gem|
249-
instance.positional_splat_keyword_and_double_splat_args(1, 2, b: 3, a: 4)
250-
251-
x.report("#{benchmark_gem.benchmark_name}: (a, *args, b:, **kwargs)") do
252-
instance.positional_splat_keyword_and_double_splat_args(1, 2, b: 3, a: 4)
253-
end
254-
end
255164
]
256165

257166
# We benchmark different cases separately, to ensure that slow performance in

‎lib/memo_wise/version.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# frozen_string_literal: true
22

33
module MemoWise
4-
VERSION = "1.10.0"
4+
VERSION = "1.10.0.dev"
55
end

0 commit comments

Comments
 (0)