Extending Chat History through Dynamic Summarization Workflow

[Iory1998]

Hi,

Thank you for the nice set of LLM Tools.

I'd like to suggest a workflow that may allow longer chat exeperience and/or could help maintain a more consistent chat conversation.
Below is the screenshot of the workflow:

The diagram illustrates a chat workflow designed to maintain a long conversation with a large language model (LLM) by summarizing and compressing the chat history. Here's how the process works:

Chat Interface:
This is the entry and exit point of the workflow, representing the user interface where the conversation takes place. It interacts with the LLM and displays responses to the user, ensuring the conversation flows smoothly.

1. Initial Interaction:

• System Prompt: This is a predefined system message that sets the context or behavior of the LLM for the chat. It might include instructions or constraints about how the LLM should respond.
• User Prompt: This represents the input from the user, which could be a question or a statement.

2. First LLM Response:

• LLM 1: This node takes the system and user prompts as input and generates a response using the LLM. The output of this node is the LLM's response to the user's input.
• LLM Output: This node captures the output from LLM 1 and feeds it into the chat history.

3. Chat History Accumulation:

• Chat History: As the conversation progresses, this node stores the accumulated user prompts and LLM responses. It represents the conversation's full memory.
  Summary Loop:
• Summary Loop: This node represents the cyclical nature of the summarization process. As the conversation grows, the token count is monitored, and when necessary, the summarization process is triggered again to ensure the chat history stays within manageable limits.

4. Token Count and Summarization Trigger:

• Token Counter Trigger: This node monitors the token count of the chat history. Once the chat history reaches a certain token limit (which is important to avoid exceeding the LLM's token processing capacity or to reach a level where the LLM stops providing coherent answers), it triggers the summarization process.

5. Summarization Process:

• System Prompt: A new system prompt is fed into the summarization process. This prompt likely instructs the LLM to summarize the current chat history.
• LLM 2 - Summary: This node represents a second LLM instance (or the same LLM with a different task) that takes the system prompt and chat history as input. It generates a summary of the chat history to compress the conversation.

6. New Compressed Chat History:

• New Chat History: The summary generated by LLM 2 replaces the older, longer chat history. This compressed version is then used for future interactions, allowing the conversation to continue without hitting the token limit.

So in short, the workflow I am suggesting allows the following:

1. The user interacts with the LLM through prompts.
2. The conversation is stored in the chat history.
3. The system monitors the chat history token count.
4. When the token count nears the limit, the system triggers a summarization process.
5. The conversation history is compressed, and the chat continues seamlessly using the summarized version.

This approach ensures that long conversations can be maintained without running into token limits, improving the efficiency of using LLMs for extended chats.

I hope this helps.
Please keep up the good work.

[heshengtao]

I remember that comfyui is not acceptable to form a closed loop, let alone form two nested closed loops. This idea is good, but comfyui should not be achievable. At least the current comfyui should not be achievable.