Memory Scope Configuration for Long-Running Coding Agents

Long-running coding agents don't get worse because the underlying model degrades. They get worse because large language models are stateless by design: a model knows only what sits inside its current context window at the moment it generates a response, and nothing else persists unless someone builds a system to make it persist. The instinct to solve this by simply widening the context window turns out to make things worse, not better. On the BEAM benchmark, accuracy actually declines as context grows from 1 million to 10 million tokens, so a larger window doesn't dilute the cost of an attention failure; it amplifies it, since there is more irrelevant material for the model to weigh against the signal that actually matters.

A team that treats the context window as if it were storage rather than working memory runs into four distinct failure modes. Token bloat is the simplest: unmanaged conversation history inflates the context rapidly within a single session, and output quality degrades in ways that are difficult to trace back to a single cause. Preference dilution follows a more specific pattern: agents honor a user's stated preferences at high rates early in a session, but compliance with omission-type preferences (formatting rules, things the user asked the agent to suppress) falls sharply as the session runs long, even while commission-type constraints (explicit things the agent must do) tend to hold steady. Mid-session contradiction appears as the ratio of relevant signal to accumulated noise shifts with every added turn, and the agent starts contradicting its own earlier output with no change to the model underneath it. Instruction decay rounds out the set: system prompts that are supposed to govern behavior throughout a session lose relative weight as the context fills with everything else, and the agent drifts from its own operating constraints without any identifiable trigger.

A coding agent built on retrieval-augmented generation layered over a growing context window doesn't escape these failure modes; it just manages them reactively. Such a system behaves like a capable search interface: it can pull relevant snippets on demand, but it cannot accumulate decisions over time, cannot recall that a given approach was already tried and rejected, and cannot honor a preference a user stated three sessions ago unless that preference happens to resurface in a retrieval query. None of this reflects a weakness in any particular model. It reflects the absence of a system built to carry information across the boundary of a single context window. If the problem is architectural, the fix has to be architectural too.

The two-tier memory architecture that production coding agents have converged on

Production coding agents have largely converged on a two-tier architecture that treats the context window the way an operating system treats RAM: fast, immediately accessible, and deliberately small, backed by a persistent layer that holds everything the agent might need later and feeds it in on demand.

Tier 1, the context window itself, holds only what the current turn actually requires: the recent exchange, the active task, and a narrow set of memories retrieved specifically because they're relevant to this prompt. This tier has to be actively managed rather than left to grow. Running summaries replace raw transcript, older messages get trimmed, and scratchpads hold working notes that don't need to survive past the current task.

Tier 2 is the persistent memory layer sitting behind it: a SQL database, a vector index, a graph store, or a dedicated memory framework built to hold durable facts, decisions, and preferences for as long as they remain relevant. This tier feeds Tier 1 through structured retrieval keyed to the task at hand, not by dumping its contents wholesale into the prompt.

The MemGPT and Letta pattern gives this architecture a formal shape. The context window functions as RAM, and the agent manages the paging between tiers itself through tool calls rather than relying on an external process to decide what belongs in context. Memory lives in labeled, size-bounded blocks pinned directly in context, while recall memory and archival memory sit on disk, searchable whenever the agent needs them.

Four functional categories of memory call for distinct retrieval paths, and collapsing them into one undifferentiated vector store produces retrieval noise rather than useful recall. Working memory is the active context window itself: bounded, costly to fill, and the only memory the model actually sees at generation time. Episodic memory holds past experience, things like which fix resolved a particular class of bug or which design got rejected and why, so you retrieve it best by matching the current situation against similar past examples. Procedural memory covers how things get done on a given team: conventions, deployment checklists, response style, and this category works better injected as evolving rules in the system prompt than pulled in as retrieved facts on the fly.

One more constraint governs the whole architecture: writing to memory cannot sit on the critical path of a response the user is waiting on. Memory writes that add latency to that response are a design flaw, not a tradeoff. Letta's sleep-time compute pattern handles this by consolidating memory asynchronously, which keeps the agent responsive in the moment while still allowing more careful organization of what gets stored, when there's no user waiting on the other end.

Memory scope as an infrastructure decision

Memory scope answers three questions for every piece of information an agent touches: what gets retained, at what boundary, and for how long. The model itself has no built-in concept of scope. It cannot tell on its own whether a fact belongs to one user, one project, or one organization. Every scope boundary has to be defined and enforced by the infrastructure surrounding the model, not by the model's own reasoning.

Three scope levels matter for coding agents in particular. Session scope covers facts relevant only to the current task run: the file paths currently in play, the active branch, working hypotheses the agent is in the middle of testing. These should expire at the end of a session, or get evaluated for promotion into a more durable scope if they turn out to matter beyond that one run. Project scope covers decisions, conventions, and constraints that belong to a specific repository or service: architectural choices, naming conventions, alternatives that were tried and rejected, known constraints on the system. This layer persists across sessions and across whichever agents happen to be working that codebase.

The mechanism that makes this work is scope tagging applied at the moment of every write. Each memory record carries identifiers, covering user, agent, session or run, project, and organization, and those identifiers govern which future queries are even allowed to surface that record. Metadata like repo ID, branch name, file path, and timestamp makes memory queryable along the exact dimensions that matter for code work, the same way a database index makes a query fast along the column it's built on.

Scope cannot be a model feature, because the model has no mechanism to enforce a boundary it doesn't control. The scope identifier attached to a query has to come from the authenticated session making the request, never from text inside the model's own prompt. A model that gets to decide whose memory it's allowed to query isn't making a design choice; it's exhibiting a security failure. That puts scope configuration in the same layer as authentication, routing, and access control. It belongs to platform engineering, in the same place a database access boundary belongs, not to prompt engineering.

Representing project-scoped memory in practice: repo-level memory files and structured metadata

The most common production pattern for project-scoped memory is a plain, human-readable file checked into the root of the repository, acting as a standing set of instructions for any agent that works in that codebase. Anthropic's documentation describes CLAUDE.md files as giving Claude Code persistent context loaded into every session. The AGENTS.md convention has spread even further: the official agents.md site reports it's in use across more than 60,000 open-source projects, read by agents including OpenAI Codex, Amp, Jules from Google, Cursor, and Factory. Google's Gemini CLI documentation describes its own GEMINI.md files the same way, as a feature for providing instructional context to the model, loaded automatically from the project root.

Underneath that file-level convention sits a finer-grained layer of structured metadata records, each tagged in a way that makes it queryable rather than just readable. Repo facts, like service dependencies, entry points, and build system layout, get tagged with repo ID and file path. Design decisions get recorded as discrete statements, something like "feature flags must be read from ConfigService, not environment variables directly," tagged with scope, author, and timestamp so a future agent (or a human reviewer) can tell when the decision was made and who made it. Rejected alternatives get stored with the reasons they were abandoned, so an agent working the same problem months later doesn't propose the same design a previous session already ruled out. User preferences get scoped to an individual engineer's ID within the project: preferred patterns, frameworks, sensitivities particular to how one person likes code written.

What makes this approach work is that it stores structured semantic records rather than raw transcript. A record that says a particular design decision should be treated a certain way captures the meaning of a decision. An agent retrieving that record gets a fact it can act on directly, instead of having to reconstruct the reasoning from a wall of past dialogue. That distinction is what lets an agent behave consistently across sessions and across the different engineers who might invoke it on the same repository.

Memory bloat and write discipline

Memory systems that write without discipline run into trouble even when their retrieval logic is sound. Persisting raw transcripts, trivial comments, and half-finished reasoning steps degrades retrieval quality as the store grows, and the result is an agent that performs worse the longer it operates, defeating the entire premise of giving it persistent memory.

The failure has a structural cause. When a retrieval query returns a wall of low-signal, partial thoughts alongside the handful of facts that actually matter, the context window fills with noise. The high-value decisions, the ones that are supposed to be shaping how the agent behaves, get buried under the surrounding volume rather than surfaced clearly.

Avoiding this requires explicit rules about what gets written and what gets dropped. Durable facts, explicit decisions, stated preferences, confirmed constraints, and rejected alternatives with their reasons attached belong in persistent storage. You should discard raw turn-by-turn transcript, superseded working hypotheses, and intermediate tool call results, or fold them into a summary rather than keep them verbatim. Session-scoped observations occupy a middle category: they should get evaluated at the end of a run to see whether they proved durable enough to deserve promotion to project scope.

None of this works without metadata attached to every record at write time. Source, timestamp, and confidence level, recorded alongside each fact, make it possible to filter and expire entries later as circumstances change. Without that metadata, there's no principled way to tell which stored facts have gone stale or simply turned out to be wrong. Treating temporal management as a core operation rather than an afterthought means a memory pipeline has to handle extraction, consolidation (merging related facts and resolving contradictions between them), selective retrieval, and ongoing tracking of when a fact was learned, whether it's still valid, and whether it needs review. Write discipline, in other words, is not a secondary concern behind retrieval design. Write discipline keeps the memory store useful as it grows instead of letting it slow the agent down.

Security boundaries that memory scope must enforce in multi-agent and multi-tenant environments

Once multiple agents work the same codebase, or one platform serves multiple teams or customers, memory scope is part of the authorization boundary itself, not a tuning parameter. A misconfigured scope at that point is a data boundary failure, the same category of problem as a misconfigured database permission, not merely a drop in retrieval quality.

The scope identifiers attached to a query, the user ID and organization ID that determine which records a given request can surface, have to come from the authenticated session making that request. They can never be derived from text inside the model's own prompt. A model that gets to choose whose memory it queries represents an exploitable vulnerability, since any input that can influence the model's reasoning becomes a potential path to someone else's data.

Memory poisoning compounds this risk in a way a single-turn prompt injection doesn't. A single poisoned prompt affects one exchange and then it's gone. A poisoned memory record persists, and for as long as it stays in the store, it can quietly steer agent behavior across every future session that retrieves it, potentially surviving across agent versions or even model upgrades.

If memory calls for this much seriousness, you need governance requirements closer to what already exists for code and infrastructure access. Each agent needs its own identity, with credentials that can be rotated and a permission scope independent of any human user's access level, so that every action can be attributed to a specific actor and audited after the fact. Memory records benefit from the same version history, review, and rollback workflows applied to code changes. Every read and write against the memory store should be logged with enough surrounding context to reconstruct why an agent behaved the way it did. And the memory store itself shouldn't be an opaque, vendor-controlled database that a security or compliance team has no way to inspect.

Applying these principles, ownership, scope boundaries, version history, review workflows, audit trails, and portability, gives security, compliance, and operations teams the same visibility into memory that they already expect to have into source code. Once memory scope reaches this level of consequence, it needs infrastructure-grade tooling behind it, not an ad hoc retrieval script bolted onto a vector database.

Agents that generate PRs at industrial volume once memory scope is configured well

Memory scope configured correctly isn't valuable on its own terms. It functions as a multiplier: an agent that retains the right decisions at the right boundary and discards the right noise can sustain output across sessions in a way an unscoped agent cannot, and that compounding advantage raises how much work the agent produces over time.

At that volume, a new constraint appears downstream, in the infrastructure built to receive the agent's output. Merge pipelines, review queues, and CI systems were sized for a world where pull requests arrive at the pace of human typing. An agent operating with well-scoped memory, carrying forward project conventions, rejected alternatives, and per-engineer preferences across sessions without needing to relearn them each time, can generate pull requests at a volume that human-paced merge infrastructure was never built to absorb. Once memory is solved, the queue becomes the next bottleneck.