feat: Add AsyncExecutor in MCPTool, ensure MCPTool works in hayhooks

[vblagoje]

Why:

To simplify and standardize the use of MCPTool across different deployment contexts—whether in standalone scripts, sync/async environments, or Hayhooks-based REST deployments. This is achieved by introducing AsyncExecutor and using it for MCPTool invocation.

• fixes Issue with MCPTool demo haystack#9137

What:

• Introduced AsyncExecutor: a utility for executing async code in sync contexts, enabling cleaner and safer integration of MCPTool.
• Added PipelineWrapper example: a concrete examples/hayhooks/pipeline_wrapper.py of deploying MCPTool in a Haystack pipeline using Hayhooks, exposing it as a REST API.
• Enhanced client-server communication and tool invocation with improved logging for easier debugging and observability.

How it can be used:

Example hayhooks pipeline wrapper of a time tool in Hayhooks:

class PipelineWrapper(BasePipelineWrapper):
    def setup(self) -> None:
        time_tool = MCPTool(
            name="get_current_time",
            server_info=StdioServerInfo(command="uvx", args=["mcp-server-time", "--local-timezone=Europe/Berlin"]),
        )
        self.pipeline.add_component("llm", OpenAIChatGenerator(model="gpt-4o-mini", tools=[time_tool]))

Example invocation:

curl -X POST 'http://localhost:1416/time_pipeline/run' \
     -H 'accept: application/json' \
     -H 'Content-Type: application/json' \
     -d '{"query":"What is the time in San Francisco? Be brief"}'

How it was tested:

• Updated unit tests
• Validated that the REST deployment works end-to-end with the time query example, see examples/hayhooks/pipeline_wrapper.py
• Validated all existing examples continue to work as well

Notes for the reviewer:

• Focus on AsyncExecutor and how it simplifies coroutine management across different environments.
• Review the logging enhancements which improve transparency during tool execution and aid in debugging issues across async boundaries.

[mpangrazzi]

I'll approve as discussed!

I'll leave here some notes for next iterations:

• The main need for this in Hayhooks context is to have a reliable way to run async code (MCP related) in sync environments (PipelineWrapper), which in turn runs in async environment (FastAPI). The main problem is that MCP SDK seems to be not designed to be run on another event loop context (FastAPI), especially if wrapped in sync code.
• We'll need to implement async versions of setup / run_api in Hayhooks environment, which will ease the dealing with existing event loops.
• The singleton pattern used in AsyncExecutor to ensure only one executor exists in the application can be a bottleneck: when multiple concurrent requests call run(), they queue up waiting for the single executor thread. A better approach should be to call directly async ainvoke() from an async run_api().

[vblagoje]

I'll approve as discussed!

I'll leave here some notes for next iterations:

• The main need for this in Hayhooks context is to have a reliable way to run async code (MCP related) in sync environments (PipelineWrapper), which in turn runs in async environment (FastAPI). The main problem is that MCP SDK seems to be not designed to be run on another event loop context (FastAPI), especially if wrapped in sync code.
• We'll need to implement async versions of setup / run_api in Hayhooks environment, which will ease the dealing with existing event loops.
• The singleton pattern used in AsyncExecutor to ensure only one executor exists in the application can be a bottleneck: when multiple concurrent requests call run(), they queue up waiting for the single executor thread. A better approach should be to call directly async ainvoke() from an async run_api().

In full agreement - couldn't have said it better than this - thanks @mpangrazzi