Yujie Zhou

Beyond “Summarize the Chat”: How Open Nua Compacts Long-Running Agent Context

Open Nua Engineering

The longer a desktop Agent works, the more likely it is to hit an apparently simple constraint: the model context window is finite.

The obvious response is to generate a summary when the message history gets too large and replace the old conversation with that summary. For an Agent that writes code, reads files, invokes tools, waits for approval, and resumes from checkpoints, “just summarize it” is nowhere near enough.

An unreliable compaction can turn unfinished work into a completed claim, separate a tool call from its result, lose recently read files and active plans, or replay encrypted reasoning state that was valid only inside the original provider request chain.

Open Nua Desktop therefore treats context compaction as an independent runtime pipeline. It does not delete conversation facts. It builds a smaller effective context projection for the next model call: older work becomes a structured summary, recent messages remain verbatim, compacted source segments are preserved separately, the boundary and summary are checkpointed, and tool, recovery, and observability semantics stay intact.

Start by separating three kinds of history

The most important design decision is not the summarization prompt. It is separating three different histories:

  1. Conversation facts: user messages, model responses, tool calls, tool results, HITL, and raw checkpoint state;
  2. Model projection: the message list actually sent with the next request;
  3. User view: the conversation, compaction notices, and execution state visible in Desktop.

Compaction rewrites only the second layer. Raw messages may keep growing while the runtime reconstructs the effective list from a persisted compaction event:

raw state
  = old message prefix + recent raw messages

effective context
  = compacted summary + recent raw messages

The implementation adds a dedicated compaction-state layer that reconnects the summary to the correct raw-message boundary. The summary is not a new source of truth. It is a replaceable read projection.

Conversation facts, compaction state, model context, and the user view have separate responsibilities inside one thread

Compacted source messages are appended to a conversation-history file in the current workspace. The checkpoint stores the summary, boundary, strategy, and before/after token estimates. Reopening the thread does not require summarizing again, and the already-compacted prefix is not sent to the model in full.

Compaction is a cascade, not a single action

Open Nua does not immediately call an LLM as the context approaches its limit. Before every model request, the runtime works through four levels:

  1. Collapse consecutive read and search traces;
  2. Truncate oversized tool-call arguments in older messages;
  3. Microcompact stale tool results;
  4. LLM compaction only if the first three steps still leave the context above the threshold.

The first three levels require no extra model call. They remove structural load that no longer needs to remain verbatim.

The context decision path from lightweight cleanup through threshold evaluation, summarization, restoration, and post-compaction verification

Collapse recognizes consecutive read_file, grep, glob, and web_search calls. It replaces completed exploration traces with a compact record that preserves paths, line spans, and result status while keeping the last call/result pair intact.

Truncate targets a different source of growth: tool arguments can themselves contain large write payloads. Shortening an old argument after the tool has already run is cheaper than rewriting the whole conversation and does not change the execution fact.

Microcompact supports threads resumed after a long pause. The current Desktop configuration uses a six-hour threshold. When it fires, old compactable tool results can be replaced by an explicit cleared marker while the five most recent results remain intact. This removes repeated payload from the next model request, not from the underlying transcript.

Only if the context is still too large does the runtime ask an LLM to summarize. When a model profile declares max_input_tokens, Desktop currently triggers at 75% of the window and retains roughly 10% as recent context. Without a reliable profile, it falls back to a 100,000-token trigger and keeps the six most recent messages.

Why partial compaction comes first

Full compaction converts everything before a boundary into one summary. It is simple, but it asks that summary to carry too much semantic weight.

The default strategy therefore prefers partial compaction: summarize the older prefix while preserving recent messages verbatim. This has three immediate benefits:

  • The user's latest request and corrections are not regenerated;
  • An active tool loop keeps precise arguments and results;
  • The next model call sees both the long-term narrative and the latest raw evidence.

The prompt distinguishes between summarizing an earlier prefix and summarizing a recent segment. Relative dates are resolved with local date and timezone context so “yesterday” or “next week” still means the same thing when the thread resumes days later. The summarizer must return text and cannot call tools during compaction.

After summarization, the runtime may re-read a small number of recently accessed text files and restore the active plan and TODO list. File restoration has its own total, per-file, and file-count budgets. Binary content is never inserted as text.

“The model remembers which file we were changing” and “the model has the file's current contents again” are deliberately separate guarantees.

The hard problem is the boundary, not the summary

In ordinary chat, summarizing everything before message 50 sounds harmless. Tool-using Agent histories have protocol structure: an assistant message emits tool calls, and the corresponding tool results must immediately follow. If the cutoff lands on a tool result, the next OpenAI-compatible request contains invalid history.

Open Nua advances the cutoff so that the tool call and its contiguous results remain on the same side. Before summarization, it also repairs cancelled or interrupted history: missing results become explicit cancelled results, while orphan results with no matching call are discarded.

This does not fabricate tool success. It restores the adjacency invariant required by the provider protocol.

A numeric index alone is not stable enough. Thread recovery, history repair, or multi-Agent prefix reconstruction can shift raw message positions. A compaction event therefore stores:

  • cutoff_index: a positional fallback for older checkpoints;
  • cutoff_message_id: a stable anchor for the first retained message;
  • summary_message: the projection used by the next call;
  • file_path: the history file that accumulates compacted source messages;
  • strategy, tokens_before, and tokens_after: strategy and diagnostic facts.

Recovery looks for the message ID first, falls back to the index only when necessary, and checks again that the boundary does not open on a tool result. This small anchor prevents a dangerous failure mode: a correct summary joined to the wrong suffix.

Token pressure cannot be estimated from characters alone

Compacting too early adds latency and summary cost. Compacting too late can cause the ordinary request to fail. The decision must approximate what the model will actually receive.

Open Nua uses a hybrid of a real usage anchor plus estimation for the new tail. It scans backward for the latest AI message with provider usage, uses real input, output, and cache usage as the anchor, and estimates only messages added after it. Without usage data, it estimates text, JSON, images, and tool blocks separately and adds a conservative pad.

There is another subtle correction. Hidden reasoning output reported in provider usage does not necessarily become visible content in the next context. Compaction pressure therefore subtracts known reasoning tokens from output tokens, or estimates them from recognizable reasoning fields when the provider omits the detail.

Real usage is not trusted blindly either. It describes one completed request and may not cover a persisted history that was later repaired or reconstructed. The runtime also estimates the full current list and never lets the usage anchor undercount that conservative total.

Provider-private state must not enter replayable history

Some providers return encrypted state inside reasoning items. That state can help the provider continue reasoning within the original request chain, but it is not portable conversation content.

After compaction, the summary and retained messages form a new request with new item identity. If old encrypted content enters the summary request, history file, or next main-model request, the provider can reject it with invalid_encrypted_content.

Automatic and manual compaction therefore apply the same rule at two points:

  1. Strip provider-only reasoning and encrypted state before summarization;
  2. Run the compacted main-model request through the shared sanitizer again.

Visible text, ordinary assistant messages, tool call/result pairs, and checkpoint semantics remain intact. Context compaction is not only information compression here; it is state sanitation across a request boundary.

Manual /compact has different product semantics

Automatic compaction happens before a model call: compact first, then continue the current task with the new context.

When the user explicitly invokes /compact, the goal is only to update thread state. The Agent should not continue working or trigger side effects after the summary finishes. Desktop treats manual compaction as its own turn operation: load the checkpoint, generate the summary, update the compaction event, save a visible compact command, and end the execution.

The Agent continues only after the user's next message.

This prevents a context-maintenance action from accidentally invoking tools, and keeps “compaction completed” distinct from “the original task completed.”

Failure must remain visible

The summary model can fail. History offload can fail. The summary plus recent messages can still be too large. A reliable runtime cannot print a warning and quietly send the original oversized request again.

Automatic compaction therefore has three failure layers:

  • Repeated summary failures open an instance-level circuit breaker with exponential cooldown;
  • An explicit provider context-overflow error may bypass the breaker for one necessary compaction attempt;
  • After summarization, the runtime estimates the resulting messages again and requires a new thread if they still exceed the limit.

The pipeline emits compaction_started, compaction_completed, or compaction_failed. The conversation UI uses these events to update state, insert a deduplicated notice, and retain an activity record. Users no longer see an unexplained gap, and operations can distinguish an ordinary answer from a summary call or compaction failure.

Honest limits of the current design

Context compaction improves the ability to continue long work. It is not lossless storage or a proof of semantic correctness.

  • A summary is still model-generated derived content and may omit detail; source history and recent raw messages are the main compensation.
  • File restoration reads current contents at compaction time, not a historical file snapshot.
  • Token estimates cannot be perfectly identical across providers, especially without complete usage metadata.
  • A history-offload failure does not block summarization, but compacted source messages cannot then be recovered through the history-file path.
  • If one recent message, restored file segment, or the summary itself is already too large, another compaction is pointless and the runtime requires a new thread.

These constraints are more honest than claiming “infinite context.” Compaction provides a budgeted, recoverable, and diagnosable degradation path. It does not turn a finite window into an infinite one.

Six lessons from the implementation

1. Compaction should generate a projection, not rewrite facts. Raw history, model context, and the UI need separate owners.

2. Apply deterministic reduction before paying for another model call. Collapse, truncate, and microcompact can remove substantial low-value load.

3. Recent verbatim context is worth more than a longer summary. Partial compaction reduces the risk of reinterpreting the latest instruction or active tool state.

4. Tool-protocol boundaries are hard constraints. No summary quality can rescue an orphan tool result, a drifting cutoff, or non-replayable provider state.

5. Compaction completion is not task completion. Manual compaction, automatic compaction, the original execution, and recovery require separate accounting.

6. Measure again after compaction. Producing a summary does not prove that the context is safe. The main model should continue only after the post-compaction check passes.

Long-running work is not about fitting ever more text into a model. It is about preserving correct working state inside a finite window. That is why context compaction in Open Nua Desktop is no longer a small “summarize conversation” feature. It is infrastructure connecting message protocol, checkpoints, files, provider adaptation, execution lifecycle, and observability.