@langchain/aws ChatBedrockConverse throws error when used in streaming mode with tool calling

[tusharm]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import { ChatBedrockConverse } from '@langchain/aws';
import { DynamicStructuredTool } from '@langchain/core/tools';
import { z } from 'zod';

async function modelWithMultiplier(prompt: string) {
  const multiplier = new DynamicStructuredTool({
    name: 'multiply',
    description: 'Multiply two numbers',
    schema: z.object({
      a: z.number(),
      b: z.number(),
    }),
    func: async (input: { a: number; b: number }) => {
      return input.a * input.b;
    },
  });

  const model = new ChatBedrockConverse({
    model: 'anthropic.claude-3-haiku-20240307-v1:0',
    region: 'us-east-1',
    temperature: 0.7,
    maxTokens: 256,
    streaming: true,
  }).bindTools([multiplier]);

  // replacing this with model.invoke() works fine
  let response = '';
  for await (const msg of await model.stream(prompt)) {
    response += msg;
  }
  return response;
}

describe('Tool usage', () => {
  it('should stream response after tool call', async () => {
    expect(async () => {
      return await modelWithMultiplier('what is the product of 12352 and 025.50?');
    }).not.toThrow();
  });
});

Description

• I am using a model in streaming mode in my application. I’d like to add a tool call to save the chat summary in a store
• When I bind a tool with the model, it throws error as below. Note that it works, if I turn off the streaming mode

Attempted to log "field[contentBlockIndex] already exists in this message chunk and value has unsupported type.".

  at console.warn (node_modules/@jest/console/build/BufferedConsole.js:191:10)
  at _mergeDicts (node_modules/@langchain/core/dist/messages/base.cjs:258:21)
  at _mergeDicts (node_modules/@langchain/core/dist/messages/base.cjs:249:27)
  at AIMessageChunk.concat (node_modules/@langchain/core/dist/messages/ai.cjs:251:58)
  at ChatGenerationChunk.concat (node_modules/@langchain/core/dist/outputs.cjs:55:35)
  at ChatBedrockConverse._streamIterator (node_modules/@langchain/core/dist/language_models/chat_models.cjs:115:59)
      at _streamIterator.next (<anonymous>)
  at node_modules/@langchain/core/dist/utils/stream.cjs:246:39
  at MockAsyncLocalStorage.run (node_modules/@langchain/core/dist/singletons/async_local_storage/index.cjs:13:16)
  at AsyncLocalStorageProvider.runWithConfig (node_modules/@langchain/core/dist/singletons/async_local_storage/index.cjs:60:24)
  at AsyncGeneratorWithSetup.next (node_modules/@langchain/core/dist/utils/strea

System Info

"@langchain/aws": "^0.1.7”,
"@langchain/core": "^0.3.20",
"@langchain/langgraph": "^0.2.63”

Node: v23.4.0
MacOS Ventura 13.5

[dosubot[bot]]

Hello @tusharm! I'm Dosu, here to help you solve bugs, answer questions, and support your journey as a contributor while we wait for a human maintainer. Let's tackle this together! 🚀

I found a similar unsolved discussion regarding streaming issues with tool binding: ChatAnthropic streamEvents not working with bindTools ^[1]. This might be relevant to your situation, as it also involves streaming and tool binding.

Additionally, there is a documented workaround for handling tool calls in streaming mode with the ChatBedrockConverse model. The solution involves modifying the stream handling code to check for tool calls in the stream chunks, invoke the tool when a tool call is detected, and process the tool response. Here is the relevant code snippet:

for (const response of responses) {
  console.log(`User input: ${response}\n`);
  const responseStream = await chainWithHistory.stream(
    {
      input: response,
      name: "Stacy",
      company: "Tech Unicorn",
      questions: questions.join("\n"),
      companyListings: companyListings,
    },
    {
      configurable: {
        sessionId: "123",
      },
    }
  );
  process.stdout.write("Ai Response: ");
  for await (const chunk of responseStream) {
    if (chunk?.tool_calls && chunk?.tool_calls?.length > 0) {
      // Handle tool calls
      for (const toolCall of chunk.tool_calls) {
        const toolResult = await analysisTool.invoke(toolCall);
        process.stdout.write(String(toolResult));
      }
    } else if (chunk.content) {
      process.stdout.write(String(chunk.content));
    }
  }
  console.log("\n");
}

This code ensures that tool calls are properly handled and responded to in the streaming process ^[2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[tusharm]

Thanks @dosu! let me give this a try.
Wondering if there is an issue I can track for updates on this bug?

[dosubot[bot]]

The approach of handling tool calls directly from the chunk in streaming mode is indeed part of the intended design in LangChain. The framework differentiates between tool calls and regular message chunks using the .tool_call_chunks attribute. This allows for the detection and handling of tool calls during streaming, ensuring that each tool call is followed by a corresponding tool message response ^[1][2].

However, there are known limitations, such as not all providers supporting streaming for tool calls, and specific issues like the ChatBedrockConverse model not including an ID in the response of a streaming call ^[3]. These limitations might make the process seem like a workaround, but they are part of the current design to handle tool calls effectively in streaming mode.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}