Skip to content

fix: AmazonBedrockChatGenerator does not pass Tool Call streaming chunk to streaming_callback #1680

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sjrl opened this issue Apr 30, 2025 · 1 comment
Labels
bug Something isn't working P1

Comments

@sjrl
Copy link
Contributor

sjrl commented Apr 30, 2025

The ``

For example, in the OpenAIChatGenerator we handle the stream response in this function which is shown below:

    def _handle_stream_response(self, chat_completion: Stream, callback: SyncStreamingCallbackT) -> List[ChatMessage]:
        chunks: List[StreamingChunk] = []
        chunk = None
        chunk_delta: StreamingChunk

        for chunk in chat_completion:  # pylint: disable=not-an-iterable
            assert len(chunk.choices) <= 1, "Streaming responses should have at most one choice."
            # NOTE: We convert both Tool Call and Text chunks into a chunk_delta here
            chunk_delta = self._convert_chat_completion_chunk_to_streaming_chunk(chunk)
            chunks.append(chunk_delta)
            # NOTE: Here we always pass that chunk_delta to the callback
            callback(chunk_delta)
        return [self._convert_streaming_chunks_to_chat_message(chunk, chunks)]

as you can see we pass all chunk deltas whether they are text or tool calls to the callback allowing streaming of the tool call.

We should update the AmazonBedrockChatGenerator to allow for streaming of tool calls as well by passing the tool call chunks to the streaming callback.

@sjrl sjrl added the bug Something isn't working label Apr 30, 2025
@sjrl
Copy link
Contributor Author

sjrl commented Apr 30, 2025

cc @tstadel

@julian-risch julian-risch added the P1 label May 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working P1
Projects
None yet
Development

No branches or pull requests

2 participants