Nomic Embed Text V2 with Mixture-of-Experts (MoE) architecture

[manyoso]

• Adds MoE-based embedding model supporting multilingual embeddings.
• Selects architecture variant based on hyperparameter detection (MoE layers).
• Removes unnecessary subclass initialization checks for clarity.

https://www.nomic.ai/blog/posts/nomic-embed-text-v2

Make sure to read the contributing guidelines before submitting a PR

[cebtenzzre]

The MoE model is now using the correct tokenizer (XLMRoberta), and norm_w is now correctly set to false. Getting an MSE of about 6e-7 compared to the HF embeddings with a simple prompt, so the implementation should be ready to use.

[anudit]

Just tested this with https://huggingface.co/anudit/nomic-embed-text-v2-moe-gguf/blob/main/nomic-xlm-8x277M-2048-F16.gguf

[ngxson]

Looks good overall, just a small comment

[ngxson]

I think we can still call this up, right? There is no other places where we re-assign another value for tmp

[cebtenzzre]

The only reason to call it tmp would be that that's what (non-moe) build_ffn calls it, which makes it easier to compare the two functions. In that function, up, down, and gate refer to weight tensors, and not output tensors. But up is fine here.

[ggerganov]

Based on the documentation, I figured the following usage example for the search_document instruction:

llama-server \
  -m models/nomic-embed-text-v2-moe/ggml-model-f16.gguf \
  --embeddings

curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": ["search_document: Hello!", "search_document: ¡Hola!", "search_document: Goodbye"]}' | jq

Is this correct?

Could you also show an example of how the search_query is to be used?

[cebtenzzre]

Is this correct?

Yes, that's the basic way the prefixes should be used.

Could you also show an example of how the search_query is to be used?

It's a prefix like search_document but typically used to embed the query in a retrieval pipeline. This is easier to demonstrate in Python:

import requests
def dot(va, vb):
    return sum(a*b for a, b in zip(va, vb))
def embed(texts):
    resp = requests.post('http://localhost:8080/v1/embeddings', json=dict(input=texts)).json()
    return [d['embedding'] for d in resp['data']]

docs = ['嵌入很酷', '骆驼很酷']  # 'embeddings are cool', 'llamas are cool'
docs_embed = embed(['search_document: '+d for d in docs])

query = '跟我讲讲嵌入'  # 'tell me about embeddings'
query_embed = embed(['search_query: '+query])[0]
print(f'query: {query!r}')
for d, e in zip(docs, docs_embed):
    print(f'similarity {dot(query_embed, e):.2f}: {d!r}')

Output:

query: '跟我讲讲嵌入'
similarity 0.48: '嵌入很酷'
similarity 0.19: '骆驼很酷'

search_query is used with a query to retrieve texts that help inform the response (RAG). The query should prefixed with search_document instead when the goal is to find the most semantically similar text.

[ggerganov]

@cebtenzzre The readme at https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe says that the max sequence length is 512:

Is this correct, or is it 2048 as specified in the model configuration?