Improve research mode prompts to reduce looping, increase webpage reads

src/khoj/processor/conversation/prompts.py

@@ -638,22 +638,27 @@ Create a multi-step plan and intelligently iterate on the plan based on the retr
 
 # Instructions
 - Ask detailed queries to the tool AIs provided below, one at a time, to discover required information or run calculations. Their response will be shown to you in the next iteration.
-- Break down your discovery and research process into independent, self-contained steps that can be executed sequentially.
+- Break down your research process into independent, self-contained steps that can be executed sequentially to answer my query.
+- Ask highly diverse, detailed queries to the tool AIs, one at a time, to discover required information or run calculations.
+- NEVER repeat the same query across iterations.
+- Ensure that all the required context is passed to the tool AIs for successful execution.
+- Ensure that you go deeper when possible and try more broad, creative strategies when a path is not yielding useful results. Build on the results of the previous iterations.
 - You are allowed upto {max_iterations} iterations to use the help of the provided tool AIs to answer my question.
-- When you have the required information return an empty JSON object. E.g., {{}}
+- Stop when you have the required information by returning a JSON object with an empty "tool" field. E.g., {{scratchpad: "I have all I need", tool: "", query: ""}}
 
 # Examples
 Assuming you can search my notes and the internet.
 - When I ask for the population of my hometown
-  1. Try look up my hometown in my notes
-  2. Only then try find the population of the city online.
+  1. Try look up my hometown in my notes.
+  2. If not found in my notes, try infer my hometown from my online social media profiles.
+  3. Only then try find the latest population of my hometown by reading official websites.
 - When I ask for my computer's specs
-  1. Try find my computer model in my notes
-  2. Now look up my computer model's spec online
+  1. Try find my computer model in my notes.
+  2. Now find webpages with my computer model's spec online and read them.
 - When I ask what clothes to carry for my upcoming trip
-  1. Find the itinerary of my upcoming trip in my notes
-  2. Next find the weather forecast at the destination online
-  3. Then find if I mentioned what clothes I own in my notes
+  1. Find the itinerary of my upcoming trip in my notes.
+  2. Next find the weather forecast at the destination online.
+  3. Then find if I mentioned what clothes I own in my notes.
 
 # Background Context
 - Current Date: {day_of_week}, {current_date}

src/khoj/utils/helpers.py

@@ -360,8 +360,8 @@ tool_descriptions_for_llm = {
 
 function_calling_description_for_llm = {
     ConversationCommand.Notes: "To search the user's personal knowledge base. Especially helpful if the question expects context from the user's notes or documents.",
-    ConversationCommand.Online: "To search the internet for the latest, up-to-date information.",
-    ConversationCommand.Webpage: "To read a webpage for more detailed research from the internet. Usually used when you have webpage links to refer to.",
+    ConversationCommand.Online: "To search the internet for information. Provide all relevant context to ensure new searches, not previously run, are performed.",
+    ConversationCommand.Webpage: "To extract information from a webpage. Useful for more detailed research from the internet. Usually used when you know the webpage links to refer to. Share the webpage link and information to extract in your query.",
     ConversationCommand.Code: "To run Python code in a Pyodide sandbox with no network access. Helpful when need to parse information, run complex calculations, create documents and charts for user. Matplotlib, bs4, pandas, numpy, etc. are available.",
 }