This explains why we don’t try to keep historical conversations for users to go back to. Each conversation should have a discrete purpose and goal because longer conversation dilute the dialogue.

Key Points:

1. Context Windows Explained

• Every AI model (GPT, Anthropic, Gemini) has a limited “context window”
• This is like short-term memory, ranging from 100,000 to 200,000 tokens
• Approximately 75 words per 100 tokens

1. Problem with Long Conversations

• As conversations get longer, they fill up the context window
• New information pushes out older information
• This can lead to the AI forgetting earlier parts of the conversation

1. Consequences

• The AI may lose important context from earlier in the conversation
• The core prompt and purpose can become diluted
• The AI’s responses may become less relevant or accurate over time

1. Context Windowing

• A technique used by some AIs to maintain recent context
• Involves a sliding window of recent text
• Has a recency bias and can still lead to forgetting

1. Better Approach

• Build intelligence into the system prompt and base RAG structure
• Start each new conversation with distilled knowledge from previous interactions
• Avoids starting from zero while maintaining context

Conclusion:

Relying on long, saved conversation threads with AI chatbots is not an optimal strategy due to context window limitations. A more effective approach is to use systems that incorporate learned information into their base structure, allowing for smart, contextually aware responses in each new conversation.