Streaming with VectorDBQAChain

[mamakancha]

I'm trying to write a code which finds the topK matching vectors from Pinecone and return a summary of the asked question based on the topK results. For the summary, I'm using VectorDBQAChain. Here's the code :

import { PineconeClient } from "@pinecone-database/pinecone";
import { VectorDBQAChain } from "langchain/chains";
import { PineconeStore } from "langchain/vectorstores/pinecone";
import { OpenAIEmbeddings } from "langchain/embeddings/openai";
import { OpenAI } from "langchain/llms/openai";



const embeddings = new OpenAIEmbeddings({
    openAIApiKey: process.env.OPENAI_API_KEY,
});


const pinecone = new PineconeClient();

const model = new OpenAI({
    openAIApiKey: process.env.OPENAI_API_KEY,
});



async function main(namespaceID: string) {


    await pinecone.init({
        apiKey: process.env.PINECONE_API_KEY,
        environment: "us-west4-gcp-free",
    });

    const pineconeIndex = pinecone.Index("index-test");


    const vectorStore = await PineconeStore.fromExistingIndex(
        embeddings,
        { pineconeIndex }
    );

    vectorStore.namespace = namespaceID

    const chain = VectorDBQAChain.fromLLM(model, vectorStore, {
        k: 5,
        returnSourceDocuments: true,
        verbose: true,
    });
    const response = await chain.call({ query: "What is the status of Project X" });
    console.log(response);



}


main("TEST_NAMESPACE")

This is working fine but I need to stream this response. Is there any way I can stream the response to my console? Any sort of workaround is also appreciated.

[ianmcfall]

@mamakancha I think will help:

Add streaming and callbacks to your model init:

const model = new OpenAI({
    openAIApiKey: process.env.OPENAI_API_KEY,
    streaming: true,
    callbacks: [
      {
        // callback method
        // called with every new token
        handleLLMNewToken(token: string) {
          // write output to console
          process.stdout.write(token);
        }
      }
    ]
});

Also, thanks for posting your question - I inadvertently found the solution to my vectorStore namespace problem 🙏