Making Hybrid search Better

[EggzYy]

I am always getting the same issue with the similar platforms that when making planning none of the local documents are taking into account so the whole planing becomes unrelated from the contents of local documents.

Can it be a solution that user can specify if local documents can be checked before planing the sub-queries, tasks?

[ElishaKay]

Green light for the PR for whoever would like to take it

[EggzYy]

I implemented a RAG pre-workaround (first building up a milvus standalone vector store with bm25 search tool and do search) with langchain for now creating a detailed and a summary report and injecting it to the first query before sending it to conduct_research. However a deep research and maybe injecting it in a better way could be more beneficial since RAG has its own pros and cons also milvus.

Even I done this I realized detailed_report in hybrid mode is not checking files it's directly focusing on web_search.

[ElishaKay]

Sup @EggzYy

Interesting!

Happy to see a PR with the relevant changes (even as a PR draft) to better understand the architecture you have in mind - that's actually a solid idea of injecting a bunch of context into the query itself.

In case it helps, we're moving in the direction of persistence on the backend for the embeddings and report data, which hopefully will be a good opportunity to re-meditate on some of these challenges

[EggzYy]

Memory agent can be a good idea with a user preference if you can add in new instance and connect it to a RAG or something similar so a preliminary data can be fetched from documents before creating the first sub-queries. In that case the research direction will be much more precise in case there are local documents. I will follow the document you suggested also.