126. Session And Conversation History

💬 Aryan wants to have a multi-turn conversation with the LLM about his RAM snapshot. His Conversation class mutates the shared list on accident, loses messages during trimming, and doesn’t estimate tokens correctly.

💡 Fun fact: Every LLM API is stateless — the model has no memory of previous calls. When you have a “conversation” with ChatGPT, the frontend is sending your entire conversation history with every single message, consuming context window tokens each time. OpenAI’s GPT-4o has a 128,000-token context window; Anthropic’s Claude models support up to 200,000 tokens. A typical English word is about 1.3 tokens, so 200,000 tokens is roughly 150,000 words — equivalent to a full novel sent with each message.

⚠️ Watch out: The mutable default argument trap in dataclasses is subtle. messages: list = [] defines a single list object that is shared across ALL instances of the class. conv1 = Conversation(); conv2 = Conversation(); conv1.add('user', 'hi') — now conv2.messages is [Message('user', 'hi')] too. Always use field(default_factory=list) which calls list() fresh for each new instance. This is also true for regular class attributes and function default arguments.

🤔 Think about it: trim_to_limit uses list.pop() without an argument, which removes the LAST (newest) item — the opposite of what you want. Think about it from the user’s perspective: in a long conversation, which messages are most important to keep — the earliest context or the most recent exchanges? If you always discard the oldest, what happens to the original “system prompt” or the first user question that set the topic?

Learning objectives

Avoid the mutable default argument bug with field(default_factory=list)
Build and maintain a message history list for multi-turn LLM conversations
Convert message history to the API’s expected format
Estimate token count from content length
Trim history by removing oldest messages first (pop(0))

Key concepts

field(default_factory=list) — safe mutable default
stateless API — you own the conversation state
context window — token limit per API call
list.pop(0) — remove oldest item
to_api_format() — serialize history for the API

Try it

from dataclasses import dataclass, field
from typing import Literal
@dataclass
class Message:
role: Literal[‘user’, ‘assistant’]
content: str
@dataclass
class Conversation:
# 💡 : hint : -010 xp mutable default [] creates ONE list shared by ALL Conversation instances — use field(default_factory=list) so each instance gets its own independent list
# BUG 1: Mutable default argument — all Conversation instances share
# the same list object if you don’t use field(default_factory=list)
messages: list = []
<pre class="giallo" style="color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);"><code data-lang="plain">def add(self, role: Literal['user', 'assistant'], content: str):
 self.messages.append(Message(role=role, content=content))

def to_api_format(self) -&gt; list:
 &quot;&quot;&quot;Convert to the list-of-dicts format the API expects.&quot;&quot;&quot;
 return [{'role': m.role, 'content': m.content} for m in self.messages]

def token_estimate(self) -&gt; int:
 &quot;&quot;&quot;Rough estimate: 1 token ≈ 4 characters.&quot;&quot;&quot;
 # 💡 : hint : -010 xp token count comes from message content, not the role label — use len(m.content) only; the role string ('user'/'assistant') adds 1–2 tokens of fixed overhead that this rough estimate ignores
 # BUG 2: Counts characters of role too, not just content
 # Role is 'user'/'assistant' (4–9 chars) but tokens come from content
 total = sum(len(m.role) + len(m.content) for m in self.messages)
 return total // 4

def trim_to_limit(self, max_tokens: int = 4000):
 &quot;&quot;&quot;Remove oldest messages until under token limit.&quot;&quot;&quot;
 # 💡 : hint : -010 xp list.pop() removes the LAST (newest) element — use list.pop(0) to remove the FIRST (oldest) element so you preserve the most recent context
 # BUG 3: Removes from the end (newest) instead of the start (oldest)
 while self.token_estimate() &gt; max_tokens and self.messages:
 self.messages.pop() # should be pop(0) to remove oldest</code></pre>

Concept detail

Stateful LLM Conversations

LLM APIs are stateless — each call is independent. To have a conversation, you maintain the full history locally and send it with every request.

Message history

@dataclass
class Conversation:
    messages: list = field(default_factory=list)

    def add(self, role, content):
        self.messages.append(Message(role=role, content=content))

    def to_api_format(self):
        return [{'role': m.role, 'content': m.content} for m in self.messages]

Sending history to the API

conv = Conversation()
conv.add('user', 'What processes use the most RAM?')

response = call_llm(messages=conv.to_api_format(), api_key=key)
conv.add('assistant', response)

# Follow-up — full history sent each time
conv.add('user', 'I killed Chrome. Now what?')
response2 = call_llm(messages=conv.to_api_format(), api_key=key)

Context window trimming

Models have token limits (~100k–200k tokens). For long conversations:

def trim_to_limit(self, max_tokens=8000):
    while self.token_estimate() > max_tokens and self.messages:
        self.messages.pop(0)   # remove oldest first

The mutable default trap:

# WRONG — all instances share one list
@dataclass
class Foo:
    items: list = []

# RIGHT — each instance gets its own list
@dataclass
class Foo:
    items: list = field(default_factory=list)

Solution

from dataclasses import dataclass, field
from typing import Literal

@dataclass
class Message:
    role: Literal['user', 'assistant']
    content: str

@dataclass
class Conversation:
    messages: list = field(default_factory=list)  # separate list per instance

    def add(self, role: Literal['user', 'assistant'], content: str):
        self.messages.append(Message(role=role, content=content))

    def to_api_format(self) -> list:
        """Convert to the list-of-dicts format the API expects."""
        return [{'role': m.role, 'content': m.content} for m in self.messages]

    def token_estimate(self) -> int:
        """Rough estimate: 1 token ≈ 4 characters of content."""
        total = sum(len(m.content) for m in self.messages)
        return total // 4

    def trim_to_limit(self, max_tokens: int = 4000):
        """Remove oldest messages until under token limit."""
        while self.token_estimate() > max_tokens and self.messages:
            self.messages.pop(0)  # remove oldest (index 0)

Tests

def test_separate_instances_dont_share_list():
    conv1 = Conversation()
    conv2 = Conversation()
    conv1.add('user', 'hello')
    assert len(conv2.messages) == 0, (
        'conv2 should be empty — mutable default [] shares between instances. '
        'Use field(default_factory=list)'
    )

def test_add_and_to_api_format():
    conv = Conversation()
    conv.add('user', 'What is RAM?')
    conv.add('assistant', 'RAM stands for Random Access Memory.')
    fmt = conv.to_api_format()
    assert len(fmt) == 2
    assert fmt[0] == {'role': 'user', 'content': 'What is RAM?'}
    assert fmt[1] == {'role': 'assistant', 'content': 'RAM stands for Random Access Memory.'}

def test_token_estimate_counts_content_only():
    conv = Conversation()
    # 'abcdefgh' = 8 chars → 2 tokens
    conv.add('user', 'abcdefgh')
    est = conv.token_estimate()
    # Must be based on content length (8), not content+role (8+4=12)
    assert est == 2, (
        f'Expected 2 tokens (8 content chars // 4), got {est}. '
        'Count content characters only, not role.'
    )

def test_trim_removes_oldest_first():
    conv = Conversation()
    # Each message is 400 chars → 100 tokens
    for i in range(5):
        conv.add('user', f'msg{i}' + 'x' * 395)   # ~100 tokens each
    # 5 × 100 = 500 tokens, limit 250 → should drop 3 oldest
    conv.trim_to_limit(max_tokens=250)
    remaining = [m.content[:4] for m in conv.messages]
    # Newest messages (msg3, msg4) should survive
    assert 'msg4' in remaining, 'Newest message should be kept'
    assert 'msg0' not in remaining, 'Oldest message should be trimmed first'

def test_trim_stops_when_under_limit():
    conv = Conversation()
    conv.add('user', 'short')  # ~1 token
    conv.trim_to_limit(max_tokens=100)
    assert len(conv.messages) == 1  # nothing trimmed

def test_trim_empty_conversation():
    conv = Conversation()
    conv.trim_to_limit(max_tokens=100)  # should not raise
    assert conv.messages == []