Sweep: Auto Summarize Working Context

[blazickjp]

Details

Add a feature to the working context that automatically summarizes the working context after every 5 turns of the conversation.

Checklist

• Modify backend/memory/memory_manager.py ✓ b11cd11 Edit
• Running GitHub Actions for backend/memory/memory_manager.py ✓ Edit
• Modify backend/tests/test_memory_manager.py ✓ 459d4cf Edit
• Running GitHub Actions for backend/tests/test_memory_manager.py ✓ Edit
• Modify backend/agent/coding_agent.py ✓ 1e0a34a Edit
• Running GitHub Actions for backend/agent/coding_agent.py ✓ Edit

[sweep-ai-deprecated]

🚀 Here's the PR! #24

See Sweep's progress at the progress dashboard!

⚡ Sweep Basic Tier: I'm using GPT-4. You have 5 GPT-4 tickets left for the month and 3 for the day. (tracking ID: 1ba266c898)

For more GPT-4 tickets, visit our payment portal. For a one week free trial, try Sweep Pro (unlimited GPT-4 tickets).

---

Tip

I can email you next time I complete a pull request if you set up your email here!

---

Actions (click)

• ↻ Restart Sweep

Sandbox Execution ✓

Here are the sandbox execution logs prior to making any changes:

Sandbox logs for 08b3592

Checking backend/memory/memory_manager.py for syntax errors... ✅ backend/memory/memory_manager.py has no syntax errors! 1/1 ✓

Checking backend/memory/memory_manager.py for syntax errors...
✅ backend/memory/memory_manager.py has no syntax errors!

Sandbox passed on the latest main, so sandbox checks will be enabled for this issue.

---

Step 1: 🔎 Searching

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I think are relevant in decreasing order of relevance (click to expand). If some file is missing from here, you can mention the path in the ticket description.

GPT-CodeApp/backend/memory/memory_manager.py

Lines 321 to 366 in 08b3592

	async def update_context(self):
	ctx = self.working_context.get_context()
	print("Working Context: ", ctx)
	prompt = f"""
	You are monitoring a conversation between an engineer and their AI Assistant.
	Your mission is to manage the working memory for the AI Assistant.
	You do this by adding information to the working context (short-term memory) based on the conversation history.
	## Guidelines
	- Your insertions should be short, concise, and relevant to the future of the conversation.
	- Keep track of facts, ideas, and concepts that are important to the conversation.
	- Monitor the personality of the person you're speaking with and adjust your responses accordingly.
	- Keep track of things that the user appeared to like or dislike.
	- In your thoughts, justify why you are adding or removing information from the working context.
	You can see the current working context below.
	Working Context:
	{ctx}
	Please make any updates accordingly. Be sure the think step by step as you work.
	"""
	messages = [
	{"role": item["role"], "content": item["content"]}
	for item in self.get_messages()
	]
	for message in messages:
	if message["role"] == "system":
	message["content"] = prompt
	print(messages)
	update = await self.working_context.client.chat.completions.create(
	model="gpt-4-1106-preview",
	response_model=ContextUpdate,
	messages=messages,
	)
	print(update)
	self.working_context = update.execute(self.working_context)
	self.prompt_handler.set_system()

GPT-CodeApp/backend/memory/memory_manager.py

Lines 15 to 54 in 08b3592

	class WorkingContext:
	def __init__(self, db_connection, project_directory) -> None:
	"""Initializes the WorkingContext class.
	Args:
	db_connection: The database connection object.
	project_directory: The path to the project directory.
	Attributes:
	context: The working context string.
	conn: The database connection.
	cur: The database cursor.
	client: The OpenAI API client.
	project_directory: The project directory path.
	"""
	self.context = "The user is named Joe"
	self.conn = db_connection
	self.cur = self.conn.cursor()
	self.client = CLIENT
	self.project_directory = project_directory
	self.create_tables()
	def create_tables(self) -> None:
	try:
	self.cur.execute(
	"""
	CREATE TABLE IF NOT EXISTS working_context
	(
	context TEXT,
	created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
	project_directory TEXT
	);
	"""
	)
	except Exception as e:
	print("Failed to create tables: ", str(e))
	return

GPT-CodeApp/backend/tests/test_memory_manager.py

Lines 85 to 118 in 08b3592

	def test_get_context(self):
	# Arrange: Prepare the context to be added
	messages = [
	{
	"role": "system",
	"content": "You are an AI Pair Programmer and a world class python developer helping the Human work on a project.",
	},
	{
	"role": "user",
	"content": "Hello! My name is John. I am a software engineer working at Google. I am working on a project to build a new search engine.",
	},
	{
	"role": "assistant",
	"content": "Hello John! I am an AI Pair Programmer and a world class python developer helping you work on your project.",
	},
	]
	self.memory_manager.get_messages = Mock(return_value=messages)
	self.memory_manager.working_context.get_context = Mock(
	return_value="test_context"
	)
	# Act: Call the get_context method
	context = self.memory_manager.working_context.get_context()
	# Assert: Verify that a database insert command was executed
	self.memory_manager.working_context.get_context.assert_called()
	self.assertIn("test_context", context)
	# def test_update_context(self):
	# messages = [
	# {
	# "role": "system",
	# "content": "You are an AI Pair Programmer and a world class python developer helping the Human work on a project.",
	# },
	# {

GPT-CodeApp/backend/agent/coding_agent.py

Lines 59 to 120 in 08b3592

	class CodingAgent:
	"""
	A class to represent a coding agent that uses OpenAI's GPT-3 model to generate code.
	Attributes:
	memory_manager (MemoryManager): Manages the memory of the agent.
	functions (Optional[List[dict]]): A list of functions that the agent can call.
	callables (Optional[List[Callable]]): A list of callable functions.
	GPT_MODEL (str): The GPT-3 model used by the agent.
	function_map (dict): A dictionary mapping function names to their callable objects.
	"""
	def __init__(
	self,
	memory_manager,
	function_map: Optional[dict] = None,
	codebase: Optional[MyCodebase] = None,
	):
	"""
	Constructs all the necessary attributes for the CodingAgent object.
	Args:
	memory_manager (MemoryManager): Manages the memory of the agent.
	functions (Optional[List[dict]]): A list of functions that the agent can call.
	callables (Optional[List[Callable]]): A list of callable functions.
	"""
	self.memory_manager = memory_manager
	self.function_map = function_map
	self.GPT_MODEL = None
	self.codebase = codebase
	self.max_tokens = 4000
	self.temperature = 0.75
	self.tool_choice = "auto"
	self.function_to_call = None
	self.ops_to_execute = []
	self.client = instructor.patch(OpenAI())
	if function_map:
	self.tools = [
	{"type": "function", "function": op.openai_schema}
	for op in self.function_map[0].values()
	]
	else:
	self.tools = None
	def query(self, input: str, command: Optional[str] = None) -> List[str]:
	"""
	Queries the GPT-3 model with the given input and command.
	Args:
	input (str): The input text to be processed by the GPT-3 model.
	command (Optional[str]): The command to be executed by the agent.
	Returns:
	List[str]: The output generated by the GPT-3 model.
	"""
	print(f"Input Text: {input}\nCommand: {command}")
	self.memory_manager.add_message("user", input)
	message_history = [
	{"role": i["role"], "content": i["content"]}
	for i in self.memory_manager.get_messages()

---

Step 2: ⌨️ Coding

• Modify backend/memory/memory_manager.py ✓ b11cd11 Edit

Modify backend/memory/memory_manager.py with contents:
• Import the summarize method from gensim at the top of the file: `from gensim.summarize import summarize`.
• Add a new attribute to the WorkingContext class to keep track of the turns. Initialize it to 0 in the __init__ method: `self.turn_counter = 0`.
• Add a new method to the WorkingContext class called summarize_context. This method will call the gensim summarize method on the context attribute and return the result: ``` def summarize_context(self): return summarize(self.context) ```
• In the update_context method, increment the turn_counter by 1 each time the method is called: `self.turn_counter += 1`.
• Still in the update_context method, check if the turn_counter is equal to 5. If it is, call the summarize_context method, assign the result to the context attribute, and reset the turn_counter to 0: ``` if self.turn_counter == 5: self.context = self.summarize_context() self.turn_counter = 0 ```

--- 
+++ 
@@ -9,6 +9,7 @@
 from instructor import OpenAISchema
 from openai import OpenAI, AsyncOpenAI
 import logging
+from gensim.summarize import summarize
 
 CLIENT = instructor.patch(AsyncOpenAI())
 
@@ -34,6 +35,7 @@
         self.cur = self.conn.cursor()
         self.client = CLIENT
         self.project_directory = project_directory
+        self.turn_counter = 0
         self.create_tables()
 
     def create_tables(self) -> None:
@@ -92,6 +94,9 @@
             (context, self.project_directory),
         )
         self.conn.commit()
+
+    def summarize_context(self):
+        return summarize(self.context)
 
     def __str__(self) -> str:
         return self.context
@@ -321,6 +326,10 @@
 
     async def update_context(self):
         ctx = self.working_context.get_context()
+        self.turn_counter += 1
+        if self.turn_counter == 5:
+            self.context = self.summarize_context()
+            self.turn_counter = 0
         print("Working Context: ", ctx)
         prompt = f"""
 You are monitoring a conversation between an engineer and their AI Assistant.

• Running GitHub Actions for backend/memory/memory_manager.py ✓ Edit

Check backend/memory/memory_manager.py with contents:

Ran GitHub Actions for b11cd11459272b993c369412c13f17cf0877806a:

• Modify backend/tests/test_memory_manager.py ✓ 459d4cf Edit

Modify backend/tests/test_memory_manager.py with contents:
• Add a new test method to test the summarize_context method. This method will create a long context, call the summarize_context method, and assert that the returned summary is shorter than the original context.
• Add a new test method to test the turn_counter attribute. This method will call the update_context method 5 times and assert that the turn_counter is reset to 0 after the 5th call.

--- 
+++ 
@@ -68,6 +68,24 @@
         self.cursor.fetchall.return_value = [("test_context",)]
         self.memory_manager = MemoryManager(db_connection=self.conn)
 
+    def test_summarize_context_shortens_long_context(self):
+        # Arrange: Create a context longer than what summarize would return
+        long_context = 'A very long repetitive context ' * 30  # assuming summarization would shorten this
+        self.memory_manager.working_context.context = long_context
+        # Act: Call summarize_context method
+        summary = self.memory_manager.working_context.summarize_context()
+        # Assert: Check that the summary is shorter than the original context
+        assert len(summary) < len(long_context), 'Summarize did not shorten the context'
+
+    def test_turn_counter_reset_after_5th_update(self):
+        # Arrange: Reset the turn counter to 0
+        self.memory_manager.working_context.turn_counter = 0
+        # Act: Call the update_context method 5 times
+        for _ in range(5):
+            asyncio.run(self.memory_manager.update_context())
+        # Assert: Check whether the turn_counter is reset to 0
+        assert self.memory_manager.working_context.turn_counter == 0, 'Turn counter was not reset after 5th update'
+
     def test_add_message(self):
         # Arrange: Prepare the message to be added
         role = "user"

• Running GitHub Actions for backend/tests/test_memory_manager.py ✓ Edit

Check backend/tests/test_memory_manager.py with contents:

Ran GitHub Actions for 459d4cf4b6896fb17bc275efaa5e919603af55ad:

• Modify backend/agent/coding_agent.py ✓ 1e0a34a Edit

Modify backend/agent/coding_agent.py with contents:
• In the query method, after the line where the add_message method is called, check if the turn_counter of the memory_manager's working_context is 0. If it is, print the summarized context: ``` if self.memory_manager.working_context.turn_counter == 0: print("Summarized Context: ", self.memory_manager.working_context.context) ```

--- 
+++ 
@@ -114,6 +114,9 @@
         """
         print(f"Input Text: {input}\nCommand: {command}")
         self.memory_manager.add_message("user", input)
+
+        if self.memory_manager.working_context.turn_counter == 0:
+            print("Summarized Context: ", self.memory_manager.working_context.context)
 
         message_history = [
             {"role": i["role"], "content": i["content"]}

• Running GitHub Actions for backend/agent/coding_agent.py ✓ Edit

Check backend/agent/coding_agent.py with contents:

Ran GitHub Actions for 1e0a34a20233a2c8b5293f950db716e9dea2a755:

---

Step 3: 🔁 Code Review

I have finished reviewing the code for completeness. I did not find errors for sweep/auto_summarize_working_context.

---

🎉 Latest improvements to Sweep:

• We just released a dashboard to track Sweep's progress on your issue in real-time, showing every stage of the process – from search to planning and coding.
• Sweep uses OpenAI's latest Assistant API to plan code changes and modify code! This is 3x faster and significantly more reliable as it allows Sweep to edit code and validate the changes in tight iterations, the same way as a human would.
• Try using the GitHub issues extension to create Sweep issues directly from your editor! GitHub Issues and Pull Requests.

---

💡 To recreate the pull request edit the issue title or description. To tweak the pull request, leave a comment on the pull request.
^{Join Our Discord}