Create API.md #8
Conversation
owenhalpert
force-pushed
the
owenhalpert-api
branch
from
9e1a745 to
010ec0e
Compare
owenhalpert
requested review from
Rajrahane,
navneet1v,
yigithub,
vamshin,
jed326,
rchitale7 and
neetikasinghal
as code owners
owenhalpert
force-pushed
the
owenhalpert-api
branch
from
010ec0e to
80d8c62
Compare
owenhalpert
force-pushed
the
owenhalpert-api
branch
2 times, most recently
from
936c4e0 to
c40de7f
Compare
owenhalpert
force-pushed
the
owenhalpert-api
branch
from
c7ccfb0 to
974ae1d
Compare
owenhalpert
force-pushed
the
owenhalpert-api
branch
from
974ae1d to
796b785
Compare
| "doc_count": 1000000, | ||
| "data_type": "fp32", // {fp16, fp32, byte, binary} | ||
| "engine": "faiss", // {faiss, nmslib, lucene} | ||
| "index_parameters": { |
There was a problem hiding this comment.
Will this mimic the mapping? I guess one question would be do we want to also potentially allow the index build service the freedom to choose the parameters based on high level signals, like "BEST_RECALL", or "compression_level=32x".
There was a problem hiding this comment.
Will this mimic the mapping?
Ideally it should not mimic the mappings. This interface should evolve of its own.
There was a problem hiding this comment.
I guess one question would be do we want to also potentially allow the index build service the freedom to choose the parameters based on high level signals, like "BEST_RECALL", or "compression_level=32x".
This is a good suggestion. the problem is from k-NN how we can get this info on what customer is looking for.
There was a problem hiding this comment.
From knn, we can take this information in via mapping parameters: mode, compression_level. For this, we can expand/generalize these parameters.
For this API, I would add a separate field like:
- index_parameter_resolution
- workload_hints
- user_hints
There was a problem hiding this comment.
owenhalpert
force-pushed
the
owenhalpert-api
branch
from
796b785 to
2f215eb
Compare
API.md
Outdated
| "data_type": "fp32", // {fp16, fp32, byte, binary} | ||
| "engine": "faiss", // {faiss, nmslib, lucene} | ||
| "index_parameters": { | ||
| "space_type": // {l2, cosinesimil, l1, linf, innerproduct, hamming} |
There was a problem hiding this comment.
Can we bring this outside of index-parameters? Its more of a data set set defining property as opposed to an index parameter.
There was a problem hiding this comment.
The idea on the split was to have workload defining variables (doc count, data type, etc) immediately available at the first JSON level and keep the index creation parameters inside to be used once we start building the index. So space type was placed inside because it's used in index creation whereas data type for example is more for initially identifying the size of the file. How does that sound?
There was a problem hiding this comment.
I see - then does engine need to be top level?
There was a problem hiding this comment.
yeah the reason on keeping engine on top level is engine can be used by CP to determine what worker to hit. Since in a worker engine will be 1 only. Right now, if you see we only have faiss. Engine is not even required to be really honest.
|
There are other successful HTTP codes outside of 200 (other 2xx codes). Can we establish expected responses? Or at least establish that client can expect status code in range 200 - 300 to be successful? |
Signed-off-by: owenhalpert <[email protected]>
Add expected params for engine, space_type, algo. Change `error` to `error_message`. Split object path into doc ID and vector paths. Signed-off-by: owenhalpert <[email protected]>
Signed-off-by: owenhalpert <[email protected]>
owenhalpert
force-pushed
the
owenhalpert-api
branch
from
f049aa9 to
b8a98de
Compare
|
|
||
| Request Parameters | ||
| { | ||
| "repository_type": "s3", // Derived from repository settings |
There was a problem hiding this comment.
Not going to block this PR, but I think the "comments" should be moved to a table instead, similar to https://opensearch.org/docs/latest/search-plugins/knn/knn-index/#method-definitions for example.
Description
Add a readable API document to the root directory describing the contract between the server and client.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.