Feature/accurate retrieval benchmark #97

gaganahluwalia · 2024-10-16T13:00:25Z

Description

This includes changes for the benchmarking tests.

Changelog

Changes to Web Agent, File Agent and adding history and context.

…ieval-benchmark

gaganahluwalia · 2024-10-17T21:02:09Z

backend/src/agents/chart_generator_agent.py

-from src.websockets.user_confirmer import UserConfirmer
-from src.websockets.confirmations_manager import confirmations_manager
+# from src.websockets.user_confirmer import UserConfirmer
+# from src.websockets.confirmations_manager import confirmations_manager



These were used for the Human COnfirmation loop which is broken at the moment, so I commented it out and when next time someone picks up InferGPT they can fix it.

gaganahluwalia · 2024-10-17T21:02:31Z

backend/src/agents/chart_generator_agent.py

-        is_confirmed = await confirmer.confirm("Would you like to generate a graph?")
+        # confirmer = UserConfirmer(confirmations_manager)
+        is_confirmed = True
+        # await confirmer.confirm("Would you like to generate a graph?")


saem like above, these were used for the Human COnfirmation loop which is broken at the moment, so I commented it out and when next time someone picks up InferGPT they can fix it.

hsauve-scottlogic

Looks great Gagan! I've just put a few comments in

hsauve-scottlogic · 2024-10-17T12:48:31Z

backend/src/agents/chart_generator_agent.py

-        confirmer = UserConfirmer(confirmations_manager)
-        is_confirmed = await confirmer.confirm("Would you like to generate a graph?")
+        # confirmer = UserConfirmer(confirmations_manager)
+        is_confirmed = True


are we saying that it is always confirmed? At the moment I don't get the message asking for confirmation

Yes, I didn't want to totally remove the Human loop, so I am always setting it to True. THis true means the user is always returning true and there won't be any pop up for the user.

hsauve-scottlogic · 2024-10-17T12:53:04Z

backend/src/agents/intent_agent.py

    async def invoke(self, utterance: str) -> str:
-        user_prompt = engine.load_prompt("intent", question=utterance)
+        chat_history = await self.read_file_core("conversation-history.txt")


if the file doesn't exist, what happens?

THen it won't return the file and history will be blank and it will be created at the end..

hsauve-scottlogic · 2024-10-17T12:59:23Z

backend/src/agents/web_agent.py

+        answer_to_user = await answer_user_ques(search_query, llm, model)
+        answer_result = json.loads(answer_to_user)
+        if answer_result["status"] == "error":
+            return ""


it returns an empty string, should it return an error? Just wondering about what actually happens in that case

hsauve-scottlogic · 2024-10-17T13:03:47Z

backend/src/agents/web_agent.py

@@ -103,6 +147,76 @@ async def web_general_search(search_query, llm, model) -> str:
 async def web_pdf_download(pdf_url, llm, model) -> str:
    return await web_pdf_download_core(pdf_url, llm, model)

+async def web_scrape_core(url: str) -> str:
+    try:
+        logger.info(f"Scraping the price of the book from URL: {url}")


should the message be a bit more generic?

hsauve-scottlogic · 2024-10-17T13:06:37Z

backend/src/prompts/templates/answer-user-ques.j2

+}
+
+
+Important: If the question is realted to real time data, the LLM should provide is_valid is false.


can you add an extra line please

hsauve-scottlogic · 2024-10-17T13:10:51Z

backend/src/prompts/templates/intent.j2

+}
+
+Important:
+Please always create the last intent to append the retrieved info in a 'conversation-history.txt' file and make sure this history file is always 'conversation-history.txt'


please add an additional line

hsauve-scottlogic · 2024-10-17T13:11:21Z

backend/src/prompts/templates/intent.j2

+}
+
+Important:
+Please always create the last intent to append the retrieved info in a 'conversation-history.txt' file and make sure this history file is always 'conversation-history.txt'


Suggested change

Please always create the last intent to append the retrieved info in a 'conversation-history.txt' file and make sure this history file is always 'conversation-history.txt'

Please always create the last intent to append the retrieved info in a 'conversation-history.txt' file and make sure this history file is always named 'conversation-history.txt'

hsauve-scottlogic · 2024-10-17T13:17:20Z

backend/src/prompts/templates/math-solver.j2

+    "result": "81,462 million",
+    "steps": "1. Convert 81.462 billion to million by multiplying by 1000. Round the result to the nearest million.",
+    "reasoning": "Rounding to the nearest million ensures that the result is represented in a more practical figure, without exceeding or falling short of the actual value."
+ }


please add an extra line

hsauve-scottlogic · 2024-10-17T13:17:45Z

backend/src/prompts/templates/tool-selection-format.j2

@@ -1,4 +1,5 @@
-Reply only in json with the following format:
+Reply only in json with the following format, in the tool_paramters please include the curreny and measuring scale used in the content provided.:


Suggested change

Reply only in json with the following format, in the tool_paramters please include the curreny and measuring scale used in the content provided.:

Reply only in json with the following format, in the tool_parameters please include the currency and measuring scale used in the content provided.:

hsauve-scottlogic · 2024-10-17T13:18:15Z

backend/src/prompts/templates/validator.j2

+Task: Please find tesla's revenue every year since its creation.
+Answer: Tesla's annual revenue history from FY 2008 to FY 2023 is available, with figures for 2008 through 2020 taken from previous annual reports.
+Response: False
+Reasoning: The answer is not prvoding any actual figures but just talk about the figures.


Suggested change

Reasoning: The answer is not prvoding any actual figures but just talk about the figures.

Reasoning: The answer is not providing any actual figures but just talk about the figures.

mnyamunda-scottlogic · 2024-10-18T08:47:20Z

backend/src/agents/chart_generator_agent.py

-        is_confirmed = await confirmer.confirm("Would you like to generate a graph?")
+        # confirmer = UserConfirmer(confirmations_manager)
+        is_confirmed = True
+        # await confirmer.confirm("Would you like to generate a graph?")
        if not is_confirmed:
            raise Exception("The user did not confirm to creating a graph.")


Hi Gagan, this may be out of scope here. Is there a known timeout for the confirmation message? e.g. It pops up and if if the user doesn't interact with it in 20 seconds then it disappears and the exception is raised.

Hi Max, I have commented out the code, so there shouldn't be any pop up before the Graph generation at all now.

mnyamunda-scottlogic

🥇 Nice!

Gagan Singh added 6 commits September 25, 2024 09:03

FIrst commit with intent changes

99b9a99

Second commit with web scrape agent

ff4fb6d

Merge branch 'release/improve-reliability' into feature/accurate-retr…

6d2c3f9

…ieval-benchmark

Push the changes for the first 4 tests

925bd92

Pushing the current changes

01c8f70

Removed loggers

3ca19c1

gaganahluwalia requested review from hsauve-scottlogic and mnyamunda-scottlogic October 16, 2024 13:00

Gagan Singh added 6 commits October 17, 2024 09:17

Fixed the broken unit tests.

84dcf74

fixed linting error

9dc4373

Changes to make COnversation-history file to be exact.

46e26db

Webpack changes

4a72ccd

Pushing the Maths changes

713fc17

resolving linting error

ae05181

gaganahluwalia commented Oct 17, 2024

View reviewed changes

hsauve-scottlogic reviewed Oct 18, 2024

View reviewed changes

mnyamunda-scottlogic reviewed Oct 18, 2024

View reviewed changes

Changes suggested by Helene

52125d7

hsauve-scottlogic previously approved these changes Oct 18, 2024

View reviewed changes

Fixed failed unit ntests

8aeccdc

gaganahluwalia dismissed hsauve-scottlogic’s stale review via 8aeccdc October 18, 2024 09:07

hsauve-scottlogic approved these changes Oct 18, 2024

View reviewed changes

gaganahluwalia requested a review from mnyamunda-scottlogic October 18, 2024 09:09

mnyamunda-scottlogic approved these changes Oct 18, 2024

View reviewed changes

gaganahluwalia merged commit 70524a8 into release/improve-reliability Oct 18, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/accurate retrieval benchmark #97

Feature/accurate retrieval benchmark #97

gaganahluwalia commented Oct 16, 2024 •

edited

Loading

gaganahluwalia Oct 17, 2024

gaganahluwalia Oct 17, 2024

hsauve-scottlogic left a comment

hsauve-scottlogic Oct 17, 2024

gaganahluwalia Oct 18, 2024

hsauve-scottlogic Oct 17, 2024

gaganahluwalia Oct 18, 2024

hsauve-scottlogic Oct 17, 2024

hsauve-scottlogic Oct 17, 2024

hsauve-scottlogic Oct 17, 2024

hsauve-scottlogic Oct 17, 2024

hsauve-scottlogic Oct 17, 2024

hsauve-scottlogic Oct 17, 2024

hsauve-scottlogic Oct 17, 2024

hsauve-scottlogic Oct 17, 2024

mnyamunda-scottlogic Oct 18, 2024

gaganahluwalia Oct 18, 2024

mnyamunda-scottlogic Oct 18, 2024

mnyamunda-scottlogic left a comment

		}


		Important: If the question is realted to real time data, the LLM should provide is_valid is false.

	Please always create the last intent to append the retrieved info in a 'conversation-history.txt' file and make sure this history file is always 'conversation-history.txt'
	Please always create the last intent to append the retrieved info in a 'conversation-history.txt' file and make sure this history file is always named 'conversation-history.txt'

		@@ -1,4 +1,5 @@
		Reply only in json with the following format:
		Reply only in json with the following format, in the tool_paramters please include the curreny and measuring scale used in the content provided.:

	Reasoning: The answer is not prvoding any actual figures but just talk about the figures.
	Reasoning: The answer is not providing any actual figures but just talk about the figures.

Feature/accurate retrieval benchmark #97

Feature/accurate retrieval benchmark #97

Conversation

gaganahluwalia commented Oct 16, 2024 • edited Loading

Description

Changelog

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hsauve-scottlogic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mnyamunda-scottlogic left a comment

Choose a reason for hiding this comment

gaganahluwalia commented Oct 16, 2024 •

edited

Loading