Create vectorized versions of ScalarQuantizer.quantize and recalculateCorrectiveOffset

[thecoop]

This resolves #13922. It takes the existing methods in ScalarQuantizer, and creates vectorized versions of that same algorithm.

JMH shows a ~13x speedup:

Benchmark              Mode  Cnt     Score    Error   Units
Quant.quantize        thrpt    5   235.029 ±  3.204  ops/ms
Quant.quantizeVector  thrpt    5  3153.388 ± 192.635  ops/ms

[jpountz]

Have you been able to run luceneutil to get a sense of the indexing and search speedups?

[thecoop]

Unfortunately not, I've been unable to get the quantized vector datasets working on my machine

[benwtrent]

I compared this branch with main. There are measurable improvements, but the quantization step isn't the main bottle neck. Vector comparisons still dominate the costs. But, its a nice bump I would say.

candidate:

recall  latency (ms)    nDoc  topK  fanout  maxConn  beamWidth  quantized  index s  index docs/s  force merge s  num segments  index size (MB)  vec disk (MB)  vec RAM (MB)
 0.826         2.340  500000   100      50       32        100     7 bits    86.54       5777.61         337.47             1          1859.34       1831.055       366.211

baseline:

recall  latency (ms)    nDoc  topK  fanout  maxConn  beamWidth  quantized  index s  index docs/s  force merge s  num segments  index size (MB)  vec disk (MB)  vec RAM (MB)
 0.828         2.680  500000   100      50       32        100     7 bits    88.48       5650.74         357.45             1          1859.57       1831.055       366.211

[jpountz]

12.5% faster search overall if I read correctly? This is pretty cool! We've been excited about smaller speedups many times in Lucene's history. :)

[jpountz]

Hmm maybe I got confused, as quantization only needs to be applied to the query vector at query time, so the search speedup is noise and I should rather be looking at the indexing speedup (+2%) and merging speedup (+5%)?

[benwtrent]

as quantization only needs to be applied to the query vector at query time, so the search speedup is noise and I should rather be looking at the indexing speedup (+2%) and merging speedup (+5%)?

That is what I think. We can run a bunch more times, but I do think this provides a marginal improvement at indexing time, where we may actually re-quantize all the vectors.

I bet for "flat" indices, that only utilize the quantization, the speed up is significant. Though I haven't had time to benchmark that yet.

[benwtrent]

Could you collect the corrections in a float array? This way we keep all lanes parallized and then sum the floats later?

[benwtrent]

I think if you could keep the lanes separate for as long as possible, we get a bigger perf boost. Reducing lanes is a serious bottleneck.

[thecoop]

Indeed it is - this doubles the performance

Benchmark              Mode  Cnt     Score    Error   Units
Quant.quantize        thrpt    5   235.029 ±  3.204  ops/ms
Quant.quantizeVector  thrpt    5  2831.313 ± 46.475  ops/ms

[thecoop]

And even more with FMA operations

[benwtrent]

Oh yes ;). Thats the numbers I am expecting.

[benwtrent]

Let's name this something better, we can call it "minMaxScalarQuantization" or something?

[thecoop]

Done - and the recalculate method too

[benwtrent]

Ugh, my benchmark was on my laptop, which I think counts as "not having nice byte vectors". I will attempt to benchmark correctly on a cloud machine soon-ish.

Sorry @jpountz @thecoop for the incorrect benchmark numbers :)

[benwtrent]

On GCP, there isn't much difference. I wouldn't expect there to be a huge amount of difference as the dominate cost is the vector comparisons not the quantization.

I haven't tested with "flat" yet.

BASELINE

recall  latency (ms)    nDoc  topK  fanout  maxConn  beamWidth  quantized  visited  index s  index docs/s  force merge s  num segments  index size (MB)  vec disk (MB)  vec RAM (MB)
 0.961         2.910  200000   100      50       64        250     7 bits     6677   111.44       1794.70          79.03             1           997.58        976.563       195.313

CANDIDATE

recall  latency (ms)    nDoc  topK  fanout  maxConn  beamWidth  quantized  visited  index s  index docs/s  force merge s  num segments  index size (MB)  vec disk (MB)  vec RAM (MB)
 0.960         2.460  200000   100      50       64        250     7 bits     6527   110.99       1801.98          76.68             1           997.55        976.563       195.313

[benwtrent]

OK, sorry for the random and sparse feedback. I think this is almost there. Then I can take over merging and backporting.

Please add a CHANGES entry for 10.2 optimizations indicating a minor speed improvement for scalar quantized query & indexing speed.