.Net: Azure AI Search connector lacks key features

[markwallace-microsoft]

Missing functionality will make it hard to work with existing indexes (created by other apps/services). Missing scalability options.
Suggestion: extend the connector to cover these features:

• Allow indexes without vector search (already supported where the DB supports this)
• Allow hidden vectors (write and retrieval)

Need more clarity

• Support IsStored = false
• Support vector encoding format
• Support​ sticky sessions

[roji]

@dluc as some point, I don't think it makes sense to ask the abstraction to support each and every feature of a specific store (Azure AI Search) in this case - the point of the abstraction is to cover commonalities, and if users want to use very store-specific features, they can use the native SDK.

Having said that, we had discussions around this with @westey-m, and specifically features which are "creation-only" could make more sense via a provider-specific extensibility model in VectorStoreRecordDefinition; we'd basically e.g. add a Dictionary<string, object> to VectorStoreRecordProperty, and then specific connectors could provide extension methods over it to allow configure provider-specific aspects, which would get picked up.

Where I think we'd stop, is when a provider-specific feature actually requires an API modification, e.g. when searching or inserting.

[markwallace-microsoft]

@dluc Can you provide more details on the following:

• Support IsStored = false
• Support vector encoding format
• Support​ sticky sessions

[dluc]

Context:

• Azure AI Search indexes can be created by third party tools, so the goal was to allow RAG on indexes created outside, as long as one requirement is met: search, either by text or by vector.
• Vectors are optional, because they are mostly part of an indexing strategy, so the service allows to hide that complexity to some degree.
• Azure AI Search allows to search in multiple ways, embeddings are not mandatory, particularly with semantic search.
• Index fields can be marked as "hidden", which is similar to the previous point, ie "don't assume there's a vector field".
• Embedding can be autogenerated by connecting an Azure OpenAI service, so a request to search or to write should be allowed without providing a vector, if the service is configured accordingly.
• Similarly, embeddings can be stored in multiple formats - IIRC it's a matter of cost and max space available.
• The service scales earch search horizontally using multiple nodes, so there's a risk that pagination hits nodes with a different index, which can lead to missing results or incorrect ordering. There's a request parameter asking the service for sticky sessions (best effort, not guaranteed) to mitigate the problem when that's the case, at the cost of some extra latency.

The suggestion at the start, is to allow custom parameters, similarly to how SK supports custom LLM parameters, for example TopK, MinP, MiroStat params for Ollama, params that don't exist for OpenAI. The stricter is the interface and the fewer integrations we will enable, and these 3 seem to be a good compromise:

1. For Azure AI Search, remove the assumption that a vector is mandatory, similarly to all other Azure storage services.
   • On write, allow to write any record definition, only the key is mandatory. If the index has been created by a third party for semantic search or automatic embeddings, no error. If the index has been created by the user, they know what they are doing.
   • On read, a vector might be missing or hidden, it's up to the user to make the right choice.
2. In the collection definition, allow to select the embedding format, either float[] or base64, which are the most common AFAIK. If the definition is incompatible let the engine throw an error.
3. In the search reques, allow to pass custom params that engine implementations handle with bespoke logic. The Azure AI Search VectorStore connector should support something like a UseStickySession bool, and set Azure.Search.Documents.SearchOptions.SessionId to a unique value during the search results retrieval. This model can be leveraged by other connectors if they want to support custom search options (there are many, so we should leave the door open)

[roji]

There's several different things above, I'll try to tease them apart and answer each one separately.

So first, we are currently working on introducing a filter-only search, which doesn't accept a vector and doesn't do any similarity. This is tracked by #10295, and makes sense as virtually all vector databases have some sort of support for this. While this covers criteria-only filtering, it doesn't cover pure full-text searching that isn't coupled to a vector search (full-text that's coupled to a vector search is already covered by hybrid search).

In the search reques, allow to pass custom params that engine implementations handle with bespoke logic.

If we do go in the direction of supporting full-text-only search, I don't think some "custom params" bag is a good answer for this, since the search API method would still require you to pass in a vector property. Wouldn't it make more sense for the Azure AI Search connector to simply expose an additional API that does specifically full text search? This would not be part of the abstraction - at least not yet (since no other database supports this AFAIK) - but you'd be able to simply cast your IVectorStoreRecordCollection down to AzureAISearchVectorStoreRecordCollection, and then call that connector-specific API; it would have the exact signature that makes sense for that operation, rather than an untyped "extra parameters" bag.

BTW that's another example of why you can't expect to be able to write the same code to interface with all connectors via the abstraction - actual capabilities simply are very different (in this case, a search type that's only supported on one database).

For Azure AI Search, remove the assumption that a vector is mandatory [...]

This would mainly mean removing validation that at least one vector property is defined on the record, and making sure that e.g. the Azure AI Search Upsert APIs work when one is absent. I could see us doing this if it's necessary.

In the collection definition, allow to select the embedding format, either float[] or base64, which are the most common AFAIK. If the definition is incompatible let the engine throw an error.

I don't think that's related to the rest of the conversation (best to have separate issues/discussions for separate problems). But in any case, base64 is simply a way to encode binary data, and should be an internal implementation detail of the connector as it sends the embedding to the server. In other words, the user shouldn't be doing base64-encoding of embeddings to hand them off to MEVD - they should be passing the .NET type which actually represents the embedding (e.g. a byte[] or a float[], wrapped by Embedding), and the connector should do whatever encoding/processing is necessary in order to send it to the server. That's also how we allow the user to write code that works against multiple databases.

Beyond that, MEVD already supports different types of embeddings (in the same of float[], Single[], byte[]). We may have not implemented support for them in the Azure AI Search connector yet, but at least at the API leve we've taken care not to restrict e.g. only to float[].