Skip to content

Benchmark harness with some initial reference cases #750

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 12, 2023

Conversation

TristonianJones
Copy link
Collaborator

@TristonianJones TristonianJones commented Jun 22, 2023

Benchmarking harness and input specification for CEL benchmark tests for #729

Example output:

BenchmarkReferenceCases/string_value_==_'value'/optimized-10         	18056289	        59.17 ns/op	      16 B/op	       1 allocs/op
BenchmarkReferenceCases/string_value_==_'value'/unoptimized-10       	20531859	        58.62 ns/op	      16 B/op	       1 allocs/op
BenchmarkReferenceCases/string_value_==_'value'/trace-10             	 1489496	       824.6 ns/op	    1064 B/op	      23 allocs/op
BenchmarkReferenceCases/string_value_!=_'value'/optimized-10         	19404576	        60.92 ns/op	      16 B/op	       1 allocs/op
BenchmarkReferenceCases/string_value_!=_'value'/unoptimized-10       	19588212	        61.08 ns/op	      16 B/op	       1 allocs/op
BenchmarkReferenceCases/string_value_!=_'value'/trace-10             	 1487623	       815.6 ns/op	    1064 B/op	      23 allocs/op
BenchmarkReferenceCases/'value'_in_list_value/optimized-10           	 3781479	       318.8 ns/op	     232 B/op	      11 allocs/op
BenchmarkReferenceCases/'value'_in_list_value/unoptimized-10         	 3786957	       316.6 ns/op	     232 B/op	      11 allocs/op
BenchmarkReferenceCases/'value'_in_list_value/trace-10               	 1000000	      1138 ns/op	    1328 B/op	      33 allocs/op
BenchmarkReferenceCases/!('value'_in_list_value)/optimized-10        	 3770986	       321.1 ns/op	     232 B/op	      11 allocs/op
BenchmarkReferenceCases/!('value'_in_list_value)/unoptimized-10      	 3750060	       319.1 ns/op	     232 B/op	      11 allocs/op
BenchmarkReferenceCases/!('value'_in_list_value)/trace-10            	  830226	      1293 ns/op	    1448 B/op	      36 allocs/op
BenchmarkReferenceCases/x_in_['a',_'b',_'c',_'d']/optimized-10       	17989554	        65.58 ns/op	      16 B/op	       1 allocs/op
BenchmarkReferenceCases/x_in_['a',_'b',_'c',_'d']/unoptimized-10     	 4879408	       247.6 ns/op	     208 B/op	       6 allocs/op
BenchmarkReferenceCases/x_in_['a',_'b',_'c',_'d']/trace-10           	  700068	      1705 ns/op	    1716 B/op	      44 allocs/op
BenchmarkReferenceCases/!(x_in_['a',_'b',_'c',_'d'])/optimized-10    	18499521	        64.24 ns/op	      16 B/op	       1 allocs/op
BenchmarkReferenceCases/!(x_in_['a',_'b',_'c',_'d'])/unoptimized-10  	 4456545	       269.8 ns/op	     208 B/op	       6 allocs/op
BenchmarkReferenceCases/!(x_in_['a',_'b',_'c',_'d'])/trace-10        	  665133	      1842 ns/op	    1836 B/op	      47 allocs/op
BenchmarkReferenceCases/x_in_list_value/optimized-10                 	 3905000	       307.5 ns/op	     216 B/op	      10 allocs/op
BenchmarkReferenceCases/x_in_list_value/unoptimized-10               	 3904173	       306.7 ns/op	     216 B/op	      10 allocs/op
BenchmarkReferenceCases/x_in_list_value/trace-10                     	 1000000	      1138 ns/op	    1432 B/op	      32 allocs/op
BenchmarkReferenceCases/!(x_in_list_value)/optimized-10              	 3395769	       354.1 ns/op	     248 B/op	      12 allocs/op
BenchmarkReferenceCases/!(x_in_list_value)/unoptimized-10            	 3367008	       356.4 ns/op	     248 B/op	      12 allocs/op
BenchmarkReferenceCases/!(x_in_list_value)/trace-10                  	  848238	      1398 ns/op	    1584 B/op	      37 allocs/op
BenchmarkReferenceCases/list_value.exists(e,_e.contains('cd'))/optimized-10         	 2065491	       579.0 ns/op	     248 B/op	      10 allocs/op
BenchmarkReferenceCases/list_value.exists(e,_e.contains('cd'))/unoptimized-10       	 2064439	       579.6 ns/op	     248 B/op	      10 allocs/op
BenchmarkReferenceCases/list_value.exists(e,_e.contains('cd'))/trace-10             	  350517	      3420 ns/op	    3136 B/op	      62 allocs/op
BenchmarkReferenceCases/list_value.exists(e,_e.startsWith('cd'))/optimized-10       	 1686837	       712.8 ns/op	     280 B/op	      12 allocs/op
BenchmarkReferenceCases/list_value.exists(e,_e.startsWith('cd'))/unoptimized-10     	 1654932	       711.4 ns/op	     280 B/op	      12 allocs/op
BenchmarkReferenceCases/list_value.exists(e,_e.startsWith('cd'))/trace-10           	  322887	      3701 ns/op	    3168 B/op	      64 allocs/op
BenchmarkReferenceCases/list_value.exists(e,_e.matches('cd*'))/optimized-10         	 2269831	       529.4 ns/op	     269 B/op	      10 allocs/op
BenchmarkReferenceCases/list_value.exists(e,_e.matches('cd*'))/unoptimized-10       	  814364	      1450 ns/op	    1717 B/op	      25 allocs/op
BenchmarkReferenceCases/list_value.exists(e,_e.matches('cd*'))/trace-10             	  282592	      4261 ns/op	    4673 B/op	      77 allocs/op
BenchmarkReferenceCases/list_value.filter(e,_e.matches('^cd+'))_==_['cde']/optimized-10         	  847449	      1389 ns/op	     849 B/op	      32 allocs/op
BenchmarkReferenceCases/list_value.filter(e,_e.matches('^cd+'))_==_['cde']/unoptimized-10       	  210804	      5616 ns/op	    7583 B/op	     105 allocs/op
BenchmarkReferenceCases/list_value.filter(e,_e.matches('^cd+'))_==_['cde']/trace-10             	  117058	     10430 ns/op	   12348 B/op	     178 allocs/op
BenchmarkReferenceCases/'formatted_list:_%s,_size:_%d'.format([['abc',_'cde'],_2])/optimized-10 	 1256002	       947.9 ns/op	     464 B/op	      21 allocs/op
BenchmarkReferenceCases/'formatted_list:_%s,_size:_%d'.format([['abc',_'cde'],_2])/unoptimized-10         	  961419	      1237 ns/op	     784 B/op	      31 allocs/op
BenchmarkReferenceCases/'formatted_list:_%s,_size:_%d'.format([['abc',_'cde'],_2])/trace-10               	  442335	      2821 ns/op	    2162 B/op	      67 allocs/op

@TristonianJones
Copy link
Collaborator Author

@jpbetz could you suggest some reference test cases? My guess is that you'd have to connect the K8s environment in a similar way to how I connect the CEL standard environment.

@jpbetz
Copy link
Collaborator

jpbetz commented Jul 10, 2023

https://docs.google.com/spreadsheets/d/123G4mLDTWjpCCbfop7HL3NkDvIBLaPu6YqV8ro3Isnw/edit#gid=0 contains a bunch of the cases we benchmarked when working on estimated cost (join [email protected] for comment permissions). Maybe check those lists of any cases you'd like? The list of benchmark cases is about the same as long as this one, with some minor differences. This is a great baseline!

@TristonianJones
Copy link
Collaborator Author

@jpbetz I'll update the baselines to cover common cases. For object creation, I think I'll need a bit more thought about how to structure the test since ideally you could swap out different type providers to characterize the cost of object instantiation for different type representations. Some of the features in the Kubernetes list aren't part of core CEL (or a canonical extension), so those will likely be excluded unless you'd like to consider upstreaming some of the features into CEL.

@jpbetz
Copy link
Collaborator

jpbetz commented Jul 12, 2023

LGTM

This looks great. I'm in favor of merging and iterating from here.

@TristonianJones TristonianJones merged commit 152a561 into google:master Jul 12, 2023
@TristonianJones TristonianJones deleted the bench-framework branch July 12, 2023 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants