An LLM UI built to prevent context degradation through branching
Was listening to the Pseudodoxology Podcast episode "Antiquarian Sociologies" when--around the 1hr 17min mark--I heard Ed say something along these lines:
"Any given event, situation, historical movement has potentially infinite levels of contextualization... There's no such thing as a digression--it's just a slippage between levels of context.
This rocks and is totally true; I've never had an entirely linear conversation that I would deem valuable, really I would ban them if I could. For all its value, the LLM can only have these linear conversations. Sure, you can interrupt an LLM discussion about gradient descent with a question microwave oven design and the output will look just fine, but this both complicates the UI and pollutes the context (explained below).
Objective: Design a LLM UI that supports organized conversational digression while preserving context.
Professionally, I think people prefer concision. I could take an hour explaining why I chose X approach for Y problem, but you don't have time for that. The opposite approach is preferred in academia: "yes give me as much context as possible."
Like in the professional world, conversations with LLMs can vary in this concision preference. But unlike the professional world, an LLM dialogue can continue at any time and--usually--at low cost. This affordability encourages me--and I'm hoping others--to ask more follow-ups; follow-ups that would be deemed too tangential in person!
Bear with me for a sec, because I am sure you're thinking "what is stopping me from asking follow-ups to LLMs right now?" And you'd be right! However, there's no clean way to ask such follow-ups. A LLM chat cannot cleanly digress from the main prompt to inquire about a tangential topic. An example:
- Say you begin an LLM conversation interested in carburetor vs. fuel injection engine design.
- 4 prompts into the conversation, the LLM mentions the origins of the fuel injection engine and how it's progenitor--John B. Fuel Injector--was a hobbyist, not a professional mechanic.
- You are rightfully compelled by the history of Mr. Fuel Injector and his circuitous rise to notoriety and want to ask a follow-up: what previous experiments, works, or fields inspired his work.
And here's the rub:
- The LLM answers normally. So you have five LLM responses: 4 related to carburetor vs fuel injector engine design, and 1 related to the influences of the inventor. You can return to the former or continue with the latter (or ask literally anything else), but there are two major drawbacks: context pollution and hierarchical disorganization.
Context pollution is bad, it leads to performance drops and hallucinations. Hierarchical disorganization is more of a sociological shortcoming, but if we were to map this conversation, it would be tree-like not linear (which is how LLMs natively present it).
Requirements: Python 3.10+
-
Clone the repository:
git clone https://github.com/yourusername/digresser.git cd digresser -
Install dependencies:
pip install -r requirements.txt -
Add your Anthropic API key. Create a
.envfile in the project root:ANTHROPIC_API_KEY=your-api-key-hereYou can get a key at console.anthropic.com.
Start the server:
python server.py
Digresser will open automatically in your browser at http://localhost:8000.
- Send a message using the input at the bottom.
- Branch a conversation by hovering over any assistant response and clicking the branch button. This forks the conversation from that point, keeping both lines of inquiry alive without polluting each other's context.
- Switch between branches using the canvas on the left, which shows the full conversation tree.
- Save and load sessions using the notebook panel to pick up where you left off.