How to have chunked answers from a long running tool call (like a streaming callback)?

[nettnikl]

I have a pipeline which is long running, and can partially answer during its run time (as it includes a classical chat generator). It is invoked as a tool, as there are other tools and other pipelines wrapped as tools that the LLm needs to select from.

This works, but the answers are not streamed if they are coming from the pipeline called by the tool, as the chat generator has no access to the callback (makes sense - from the top level pipeline, there is no llm, but only a tool invoker which accepts messages, but no callback).

How can i make this work?
Maybe i can work with the new super component somehow? Or i can create a llm node instance that includes the streaming callback instead of reusing the same instance? Or can we add a streaming_callback to the tool_invoker?

[julian-risch]

Hello @nettnikl and thank you for your feedback. We have an open issue for streaming output from Tool calls in Agents and plan to start working on it this week: https://github.com/deepset-ai/haystack-experimental/issues/219

[nettnikl]

Hey @julian-risch thank you very much for theanswer. I've worked with the Agent from the experimental package - it got rid of huge chunks of code, so that's great!
Also, the streaming callback support seems to be what i'm looking for. Also i can utilize it for RAG as it outputs the tool calls including the source documents the tool outputs as well.

Will check out the progress in the upcoming time.

Thanks!