Pgvector hybrid

[jlonge4]

I was hoping to implement hybrid search within the pgvector integration and use RRF for merging. This has one piece missing, and that is the user query itself (commented out in line 571. I didn't want to open an issue/enhancement and not at least provide a starting point. Let me know what you think @anakin87 @vblagoje @masci

Inspiration -> https://github.com/pgvector/pgvector-python/blob/master/examples/hybrid_search_rrf.py

[CLAassistant]

All committers have signed the CLA.

[anakin87]

Hey, @jlonge4!

Thanks for your idea/initial implementation.

I know this is a missing feature of this Document Store.
I'll investigate and get back to you in a while...

[jlonge4]

@anakin87 thanks a lot let me know if I can do anything further!

[anakin87]

Hey @jlonge4, sorry for the long wait... Tomorrow I will take a proper look!

[jlonge4]

@anakin87 no worries sir! Thank you 🙏🏼

[anakin87]

Before talking about hybrid retrieval, we should introduce keyword retrieval. Then we can combine vector+keyword -> hybrid retrieval.

Keyword Retrieval

• https://github.com/pgvector/pgvector-python/blob/1b81759673df531879b5d9081025c0b356acff50/examples/hybrid_search_rrf.py#L31
• https://www.postgresql.org/docs/current/textsearch-controls.html#TEXTSEARCH-RANKING

TO DO (brainstorming mode 🙂)

• add some configurations to the Document Store (language)
• create another index on the DB as done here
• introduce a method _keyword_retrieval in the Document Store
• create a KeywordRetriever that accepts a query and calls the previous method

(unsure about the name)

WDYT?

[jlonge4]

@anakin87 sounds like a great plan, bite sized pieces are better! Made a few updates based on your thoughts.

[anakin87]

Hey!

• can you fix the linting error?
• can you add some tests for this addition?

[jlonge4]

@anakin87 Definitely, I am gonna do some local testing (might take a couple days) but will get there!

[anakin87]

Take your time...
I will be off for a few days 🙂

[anakin87]

Hey, I see that tests are no longer running for some reason.

I would suggest to focus on Keyword Retrieval and then think about Hybrid Retrieval (maybe in another PR).

Let me know if you need any help or suggestions...

[jlonge4]

@anakin87 you are right, I believe I'll kill this PR and do as you suggested 👍🏼

[kanenorman]

Hi @anakin87, sorry to resurface an old issue.

Are there any plans to introduce a PgvectorHybridRetriever, or is the expectation that hybrid retrieval should be implemented using a pipeline, similar to #738? I’ve noticed that some integrations, like QdrantHybridRetriever, already have hybrid retrievers. Does the core team prefer users implement hybrid retrieval through a pipeline instead?

[anakin87]

Hello @kanenorman!

There is no set rule.

Our main goal is to provide users with hybrid retrieval capabilities, if available (using a Pipeline or not).

I would say that implementing a Hybrid Retriever makes sense especially when:

• there is a significant difference in query times compared to using a hybrid retrieval Pipeline (due to a single optimized query in the DB + Pipeline overhead)
• the community requests this feature