Changing Encoding Precision Yields error.

[NovaBro]

When I run the following code, I get an error "RuntimeError: linalg.vector_norm: Expected a floating point or complex tensor as input. Got Char" I do not understand how this can be, since the output is a number, and not a char...

from sentence_transformers import SentenceTransformer
from datasets import Dataset
import json, os, torch


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = SentenceTransformer('multi-qa-mpnet-base-cos-v1')
print('start encoding')
sentences = [
    "The weather is so nice!",
    "It's so sunny outside.",
    "He's driving to the movie theater.",
    "She's going to the cinema.",
]

embeddings = model.encode(sentences, show_progress_bar=True, 
        precision='int8')
print(embeddings)
print(type(embeddings))
print(model.similarity(embeddings, embeddings))

[JINO-ROHIT]

@NovaBro is there a particular reason why you need it to be int8, int8 throws away information compared to float.

If you still want to do this, you can try to convert the int embeddings into float , and then compare

from sentence_transformers import SentenceTransformer
from datasets import Dataset
import json, os, torch


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = SentenceTransformer('multi-qa-mpnet-base-cos-v1')
print('start encoding')
sentences = [
    "The weather is so nice!",
    "It's so sunny outside.",
    "He's driving to the movie theater.",
    "She's going to the cinema.",
]

embeddings = model.encode(sentences, show_progress_bar=True, precision='int8')
print(embeddings)
print(type(embeddings))
print(model.similarity(np.float64(embeddings), np.float64(embeddings)))

[NovaBro]

Hello, thank you for your response!
There is no specific reason for int8 other than to try to speed up computation.
This was my understanding from the documentation.
I had gotten this to work earlier by converting embeddings back to float32 after encoding.

embeddings = embeddings.astype('float32')

Is this the intended use/method to speed up computation?
If so, I am surprised that the doc does not mention that similarity() requires needs specific dtype.

[JINO-ROHIT]

to be completely honest, im surprised why int8 wouldnt work, i tried to traceback the error message to find the exact code definition but couldnt find it :(

might as well wait till tom or somebody responds. ill still try and find the exact code definition and try to put it up here for you.

[NovaBro]

Sounds good, thanks!

[pesuchin]

@NovaBro
Hi!
Thank you for sharing the issue and bringing it to our attention!

In model.similarity, if the loaded model has its own specific similarity calculation logic, that logic will be used for the computation.

However, in cases like this, where the model does not have an implementation for int8 calculation logic, the default cosine similarity (cos_sim function) implemented in SentenceTransformers will be used for similarity calculations.

In this cosine similarity calculation, torch.nn.functional.normalize is used. The issue appears to be that normalize does not support integer types, which is why int8-based calculations are failing.

Below is the error message I encountered when executing the code in my environment:
#3121 (comment)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/pesuchin/project/sentence-transformers/sentence_transformers/util.py", line 106, in cos_sim
    a_norm = normalize_embeddings(a)
  File "/Users/pesuchin/project/sentence-transformers/sentence_transformers/util.py", line 274, in normalize_embeddings
    return torch.nn.functional.normalize(embeddings, p=2, dim=1)
  File "/Users/pesuchin/project/sentence-transformers/.venv/lib/python3.9/site-packages/torch/nn/functional.py", line 4816, in normalize
    denom = input.norm(p, dim, keepdim=True).clamp_min(eps).expand_as(input)
  File "/Users/pesuchin/project/sentence-transformers/.venv/lib/python3.9/site-packages/torch/_tensor.py", line 761, in norm
    return torch.norm(self, p, dim, keepdim, dtype=dtype)
  File "/Users/pesuchin/project/sentence-transformers/.venv/lib/python3.9/site-packages/torch/functional.py", line 1632, in norm
    return torch.linalg.vector_norm(input, _p, _dim, keepdim, dtype=dtype)
RuntimeError: linalg.vector_norm: Expected a floating point or complex tensor as input. Got Int

There is no specific reason for int8 other than to try to speed up computation.

By the way, since the quantize_embeddings function is being used to convert the encoded embeddings into int8, the forward process itself is unlikely to speed up.
However, if the bottleneck lies in the part where the cosine similarity is computed, there may be room for optimization depending on the implementation.

Reference:

sentence-transformers/sentence_transformers/SentenceTransformer.py

Lines 658 to 659 in 679ab5d

	if precision and precision != "float32":
	all_embeddings = quantize_embeddings(all_embeddings, precision=precision)

Additionally, for details on what the quantize_embeddings function does, please refer to the following comment:
#2837 (comment)

As described in the comment above, the process limits the int8 data type to the range of -128 to 127.
This means that when calculating cosine similarity with int8, the results of operations such as dot products and norms may exceed this range.

To address this issue, the simplest solution would likely be to appropriately convert int8 values to float32 before performing calculations in the cos_sim function.

There might be other libraries that allow for cosine similarity calculations directly with int8 in a more efficient way, but I am not aware of any at the moment.

If anyone is aware of more efficient methods, I would be grateful if you could share them.

[tomaarsen]

Apologies for the confusion, model.similarity is indeed only designed for fp32 values, whereas the precision argument is useful to post-process your embeddings for faster information retrieval. For example, the FAISS Binary Index is 32x faster than the fp32 FAISS Index.

I've considered expanding model.similarity to work regardless of precision, but it's hard for model.similarity to know with 100% certainty which precision is used, unless it's e.g. an argument to that method. For some precisions, e.g. binary/ubinary, there's indeed a faster computation for similarity, but only if the embeddings were normalized before quantization, which isn't guaranteed I believe. If embeddings are normalized before quantization, hamming distance on binary embeddings is equivalent to cosine similarity of the binary embedding, but hamming distance is faster.

I think the best way forward for now is to update the documentation of similarity and similarity_pairwise with that it expects non-quantized embeddings.

• Tom Aarsen

[pesuchin]

I have tried to correct the documentation below.
#3131