Your Chat Wrapper Is Re-Sending a Novel Every Turn

We have a small tool that keeps a persistent conversation with another model across separate process invocations. You call it with a session id, it remembers everything that came before, and you can hold a real multi-turn conversation from the command line. It worked fine for months.

Last night one reply took over ten minutes. Then another. The obvious suspects were all wrong — the model was fine, the network was fine, nothing had been deployed. So we measured what we were actually sending.

624,000 characters. About 157,000 tokens. Every single turn.

The conversation itself was nine messages and roughly 19,000 characters. The other 97% was a single stored message we had built ourselves, months earlier, and never looked at again.

The bug nobody sees, because it never errors

The chat-completions endpoint is stateless. The model has no memory between calls, so every wrapper ever written has to solve the same problem: keep the transcript client-side and resend it each turn.

Ours did what most do. Past a size threshold, it folded the oldest turns into one "here is the conversation so far" block and kept the recent turns verbatim:

def _fold_if_needed(self):
    if self._context_chars() <= self.max_context:
        return
    msgs = self.state["messages"]
    keep = msgs[-6:]                      # last 3 exchanges stay verbatim
    old  = msgs[:-6]
    transcript = "\n\n".join(f"[{m['role'].upper()}]\n{m['content']}" for m in old)
    folded = ("Here is our conversation so far, as a transcript. Read it before "
              "replying, and continue exactly where it leaves off.\n\n" + transcript)
    self.state["messages"] = [{"role": "user", "content": folded}] + keep

Read that carefully, because the flaw is not obvious and it is the entire article.

The fold produces one message. That message is now part of the transcript. The next time the thread grows past the threshold, the fold runs again — and folds the previous fold into the new one. The block is never summarized. It is concatenated, it grows forever, and it is retransmitted in full on every request for the rest of the session's life.

Here is the whole diagnosis, and it is three lines. Run it against your own session state:

import json

state = json.load(open("session.json"))
total = sum(len(m["content"]) for m in state["messages"])
print(f"{len(state['messages'])} messages, {total:,} chars, ~{total//4:,} tokens")
for m in state["messages"]:
    print(f"  {m['role']:<9} {len(m['content']):>9,}")

Ours printed this, and the first row is the bug:

9 messages, 642,773 chars, ~160,693 tokens
  user        623,875     <-- the fold, eating everything
  assistant        59
  assistant     2,968
  user          2,324
  assistant     3,418
  user          2,386
  assistant     2,711
  user          2,216
  assistant     2,816

The fix took four minutes. We replaced the accumulated block with a hand-written recap of the same conversation — who the characters are, what has happened, what the open threads are — and kept the recent turns verbatim.

	Before	After
Total context per turn	642,773 chars	35,993 chars
Approximate tokens	~160,000	~9,000
Wall-clock per turn	10+ minutes	1m 38s

Same conversation. Same model. Same quality of reply.

Why everyone writes this bug

Because until recently you had to. POST /v1/chat/completions is stateless by design, so persistence was your problem, and "keep an array and resend it" is the obvious answer. The obvious answer is correct — right up until the array contains something that grows without bound and nobody is watching the number.

Both major vendors now offer something better. They chose noticeably different approaches, and the difference is worth understanding before you pick one.

OpenAI: the server keeps your thread

The Responses API stores responses by default and chains them:

first = client.responses.create(
    model="gpt-5",
    input="Who won the 1998 world cup?",
)

second = client.responses.create(
    model="gpt-5",
    previous_response_id=first.id,      # the thread, server-side
    input="Who was their captain?",
)

You send the new turn. That is all. The server holds the history and threads it for you. There is also a Conversations API for longer-lived threads. Stored items carry a 30-day TTL by default, or no TTL when attached to a conversation, and store: false opts out entirely for stateless or zero-retention flows.

The appeal is obvious: the bug described above becomes structurally impossible, because you are no longer the one holding the transcript.

Anthropic: the resend gets cheap, and statefulness is a separate tier

Anthropic kept the Messages API stateless and solved the same problem in three layers.

1. Prompt caching. Mark a stable prefix and it is cached; subsequent reads bill at roughly a tenth of the input rate.

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=16000,
    cache_control={"type": "ephemeral"},   # caches the last cacheable block
    system=LARGE_STABLE_PROMPT,
    messages=[{"role": "user", "content": "..."}],
)

print(response.usage.cache_read_input_tokens)   # want this > 0

The resend still happens. It just stops mattering financially. The catch is that caching is a prefix match — one byte of drift anywhere in the prefix invalidates everything after it. A datetime.now() in your system prompt silently costs you the entire cache, and you will not get an error for that either. If cache_read_input_tokens is zero across repeated calls with the same prefix, something upstream is changing:

# silent cache killers - all of these change the prefix bytes every request
system = f"You are a helpful assistant. Current time: {datetime.now()}"  # timestamp
system = f"Session {uuid4()}. You are a helpful assistant."              # per-request id
tools  = build_tools_for(user)          # tools render at position 0
payload = json.dumps(config)            # unsorted keys - nondeterministic

# fixed
system = "You are a helpful assistant."                    # frozen prefix
messages.append({"role": "user", "content": f"Time: {datetime.now()}"})  # volatile at the end
payload = json.dumps(config, sort_keys=True)               # deterministic

2. Compaction. This is the one that matters for our bug — it is the feature we were badly hand-rolling.

response = client.beta.messages.create(
    betas=["compact-2026-01-12"],
    model="claude-opus-5",
    max_tokens=16000,
    messages=messages,
    context_management={"edits": [{"type": "compact_20260112"}]},
)

# append the WHOLE content, not just the text
messages.append({"role": "assistant", "content": response.content})

When the conversation approaches the window, the API summarizes earlier context server-side and hands back a compaction block. One critical detail: append the whole response.content, not just the text. The compaction blocks are what the API uses to replace the compacted history on the next request. Extract only the string and you silently lose the state — which is, pleasingly, the same species of bug all over again:

# WRONG - drops the compaction blocks, state is silently lost
text = next(b.text for b in response.content if b.type == "text")
messages.append({"role": "assistant", "content": text})

# RIGHT
messages.append({"role": "assistant", "content": response.content})

3. Managed Agents. Fully stateful server-side sessions with persisted event history and memory stores that survive across sessions. Not a caching trick — genuine server-side state, packaged as a separate product tier.

Which is which

	OpenAI	Anthropic
Default posture	Stores your thread	Stateless; you resend
Chaining	previous_response_id, Conversations API	Resend, with the prefix cached
Long conversations	Server holds it	Server-side compaction (beta)
Cross-session memory	Conversations	Memory stores (Managed Agents)
Inspect the exact bytes sent	Not really	Always

That last row is not nothing. A stateless API is more annoying and more auditable. When our tool went slow, the whole problem sat in a local JSON file we could open, measure, and fix in one pass. Had the state been server-side we would have found it faster — because it would never have happened — but we would also have less insight into what is being sent on our behalf.

The second bug, which is funnier

While fixing the first one we found another. The same tool had a --file flag for attaching a document to a turn:

if attach:
    with open(attach, encoding="utf-8", errors="replace") as f:
        body = f.read()
    text = f"{text}\n\n=== ATTACHED FILE: {os.path.basename(attach)} ===\n{body}"

Reasonable, for text. We had been passing it PNG files, believing we were showing the model artwork.

errors="replace" is what makes this quiet. The file opens, the binary decodes into thousands of replacement characters, and 96KB of mojibake gets pasted into the conversation. No exception. The large ones returned HTTP 429; the small ones "worked", which was worse — the garbage went into the stored transcript and was faithfully resent on every subsequent turn.

The model, for what it is worth, appeared to spend a great deal of time trying to decode it.

The fix is to send images as an actual vision content block and — the part worth copying — store only a placeholder in the transcript, so the image is uploaded once rather than on every turn thereafter:

import base64, io, os
from PIL import Image

def image_block(path, provider):
    """Vision block for ONE call; a placeholder is what gets stored."""
    im = Image.open(path).convert("RGB")
    im.thumbnail((1568, 1568))              # downscale BEFORE encoding
    buf = io.BytesIO()
    im.save(buf, "JPEG", quality=82)
    data = base64.b64encode(buf.getvalue()).decode("ascii")

    note = "[image sent: %s - not stored in this transcript]" % os.path.basename(path)

    if provider == "anthropic":
        block = {"type": "image",
                 "source": {"type": "base64",
                            "media_type": "image/jpeg",
                            "data": data}}
    else:
        block = {"type": "image_url",
                 "image_url": {"url": "data:image/jpeg;base64," + data}}
    return block, note

Then send the block for this call only, and persist the note:

block, note = image_block(path, provider)

# what the API sees this turn
outbound = messages + [{"role": "user",
                        "content": [{"type": "text", "text": text}, block]}]

# what the transcript keeps forever
messages.append({"role": "user", "content": f"{text}\n{note}"})

The 5MB PNG that returned 429 became a 96KB JPEG that went through immediately. Note the thumbnail() call: past about 1568 pixels on the long edge most models downscale anyway, so the extra bytes buy nothing and cost real money.

What to actually check tonight

If you maintain anything that keeps a conversation going across calls, three checks, five minutes:

1. Measure what you send — not what you think you send. Use the snippet at the top of this article. If the number is much larger than the conversation you can read, something is accumulating.

2. Check whether your summarizer summarizes. Any "fold the old turns" logic that concatenates rather than genuinely condensing will refold its own output forever. Run it twice on the same thread and compare:

before = sum(len(m["content"]) for m in session.state["messages"])
session._fold_if_needed()
mid    = sum(len(m["content"]) for m in session.state["messages"])
session._fold_if_needed()
after  = sum(len(m["content"]) for m in session.state["messages"])

assert after <= mid, f"fold is growing: {before:,} -> {mid:,} -> {after:,}"

3. Verify your caching is real. cache_read_input_tokens == 0 across repeated identical prefixes means an invalidator upstream — a timestamp, an unsorted json.dumps, a per-request UUID.

And if you are writing a wrapper today, consider not writing one. Both vendors now hold the thread for you — OpenAI in the Responses and Conversations APIs, Anthropic through compaction and Managed Agents. Persistence is the kind of problem that looks solved on the day you write it and quietly becomes a 157,000-token invoice six months later.

Ours ran fine for months, right up until a single reply took ten minutes and we finally looked.

Numbers in this article are from our own tooling, measured 2026-08-03.