Large File Handling in Agent-Driven Code Repositories
Agents need separate storage for ephemeral outputs while keeping project assets in Git LFS.
Large file handling in agent-driven code repositories requires separating durable project history from temporary run output. Git remains the right source of truth for code and reviewable changes, while Git LFS serves binaries that must follow those changes. Ephemeral agent artifacts need a separate storage layer built around targeted access and expiry.
Why Git's architecture becomes a liability when agents replace human developers
Git was designed to track text files efficiently, and its architecture reflects that choice. It is a content-addressable filesystem in which objects are hashed and stored. Delta compression saves space by storing the differences between similar objects. Binary files do not compress as effectively through this mechanism, so changes can leave large blobs in history. The cost compounds across repository size, clone speed, and branch switching.
That trade-off becomes more consequential when agents move from suggesting lines of code to writing whole pull requests, running unit tests, and opening the next change. Their work produces logs, build outputs, screenshots, exported PDFs, and intermediate datasets as a normal byproduct. These outputs are high-volume, high-churn files whose usefulness often ends well before the source history does. Keeping them in that history makes the repository carry temporary execution output for the long term.
How agent-generated artifacts differ from the large files Git LFS was built for
Large assets that are part of the project itself have long needed storage that preserves their relationship to source changes. A PSD file, a model checkpoint under active development, or a compiled test fixture may need to stay in lockstep with a commit. Its diff may be unhelpful, but its provenance still matters. Git LFS serves this category because the file belongs within the project’s branch, review, merge, and revert model. A clone that needs the asset should get the matching version through the same Git remote.
Agent run output has a different purpose. A raw log records an execution, but maintaining its version history is not ordinarily a product requirement. Intermediate datasets and run screenshots can help people check a run without becoming build inputs tied to source commits. Agents can generate hundreds of such artifacts per day, and that volume grows with adoption. Preserving each one as a permanent project asset mistakes evidence of work for the work product itself.
The distinction is therefore not simply between small and large files. It is between files whose history belongs to the project and files whose value belongs to a run.
What context window pressure reveals about agent file access patterns
Large files also impose a cost when agents consume them. Tool outputs, including file contents, command results, and search hits, consume 70–80% of the context in a typical agent session. That leaves less room for the system prompt and the agent’s own reasoning. Excess output consequently shortens the useful life of a session and degrades per-turn accuracy. Even when files are available in storage, their contents may need to stay out of context.
Claude Code addresses this problem with hard limits rather than relying on the model to regulate every read. A byte cap checked through a stat call can reject a file before it is opened, with an error directing the model toward offset/limit or grep. A token limit on returned output also catches token-dense files that fall below the byte cap. These protections work together as one check on a read. They prevent oversized reads, but navigating around large files still consumes tool calls and selection logic.
This behavior exposes the access pattern that storage must support. Agents query for the bytes relevant to a task. A full working tree can give the agent more data than it needs and still leave it with extra work to find the useful content. The mismatch concerns retrieval as much as capacity.
How Git LFS changes the storage equation without changing the abstraction
For binaries that genuinely belong with code, Git LFS separates history from payload without abandoning Git. It uses Git’s clean and smudge filter hooks to keep a small pointer file in the repository. That pointer records a content hash and size, while the actual bytes live on a dedicated LFS remote. On clone or checkout, LFS replaces the pointer with the real file. History retains its link to the asset without placing the binary content in every packfile.
The configuration that makes this consistent is .gitattributes. Tracking patterns determine which files LFS intercepts, and committing that configuration gives forks and fresh clones the same rules. Agent-adjacent repositories may use patterns including *.safetensors, *.h5, *.bin, *.psd, and *.mp4. A pre-commit hook that checks staged file sizes and warns about large untracked files is a low-cost way to catch mistakes. The purpose is to keep the storage boundary explicit rather than depend on each contributor remembering it.
LFS therefore changes where bytes reside while retaining the familiar project abstraction. The repository stays leaner, and the working tree materializes the assets it needs. Plans still need to allow for the hosting platform’s storage and bandwidth quotas. Moving a binary out of Git’s object storage does not make its transfer or retention free.
Where Git LFS breaks down at agent-scale artifact volumes
The pointer-and-payload split is well suited to versioned binaries, but it is not a reason to place every large output under LFS. With as much output as agents produce, the costs of bandwidth and metadata overhead grow large. CI systems fetch binary assets again and again, often across regions and clouds. LFS objects are content-addressed and highly cacheable, yet without caching the transfer cost recurs on each pipeline run. Workers that need only source can still pay to materialize unrelated assets.
Retention creates a separate problem. Although pointers are small, their history entries persist across the repository’s lifetime. High-churn run output therefore leaves metadata debt even when its payload sits elsewhere. Storage charges also stay tied to the hosting platform’s quotas, while object storage has per-byte pricing that scales with use.
These limits make LFS the wrong default for generated PDFs, screenshots, and dataset dumps that nobody needs in every clone. Treating LFS as unlimited object storage inside the repository reproduces the original category mistake at a different layer.
The case for external object storage with lifecycle rules as the home for ephemeral agent artifacts
Run-scoped object storage fits artifacts whose value is inspectable and time-bounded. A folder for an agent run gives humans a place to look through logs, build outputs, intermediate datasets, and screenshots without adding them to repository history. Lifecycle rules automate expiry when those artifacts are no longer needed. Where supported, per-file version history preserves an inspection model without Git’s checkout semantics.
An agent can access this external workspace through a Model Context Protocol server. This integration lets multi-megabyte binaries live in dedicated workspace storage while remaining available to the agent’s tools. Artifact size is no longer coupled to clone cost or Git storage quotas. Object storage also offers greater elasticity and lower cost per GB than LFS quotas tied to hosting plans. When runs produce a lot of output, those savings are large enough to shape how the system is run.
External storage lacks Git’s branch, diff, review, merge, and revert workflow. For a large artifact that must be reviewed as part of a code change, LFS remains the appropriate choice.
Why the merge queue becomes the pressure valve when large-file discipline fails
Artifact decisions propagate into the merge queue because validation must handle whatever the pipeline brings into scope. Large working trees and LFS downloads take up disk space on runners and put more pressure on I/O. As more teams use agents, they have more pull requests that still need review, CI, and merge. More changes can therefore reach a downstream constraint without improving delivery. Artifact handling and queue capacity are parts of the same infrastructure problem.
Plaid encountered that connection as its monorepo grew to hundreds of commits per week. Enabling a merge queue with a naive “build everything” configuration caused CI runners to exhaust disk space and hit I/O throttling. Plaid removed container image builds from the queue and narrowed validation to compilation. Median merge queue time fell to 1.8 minutes. The decisive change was the scope of work and artifacts in the queue.
Linear’s CI suite grew roughly 4× since January 2026, and agents wrote most of the new tests. Linear moved from default GitHub Actions hosts to runners with faster CPUs, faster storage, and cache infrastructure, which made jobs faster on average. An agent-operated queue could take on conflict resolution, CI failure triage, and queue management while reserving high-judgment decisions for humans. How much work it could handle would still depend on keeping artifact traffic that it does not need out of each run.
Choosing storage strategies for different file types in an agent-driven repository
The storage choice follows from a file’s churn rate, review requirement, access pattern, and required lifetime. Source code, configuration, small documentation, .gitattributes, and pipeline definitions belong in plain Git. Repositories should ideally stay under 1 GB and strongly under 5 GB to keep clones manageable. These files form the durable, reviewable description of the project. Their place in source history is part of their value.
Versioned binaries require a different treatment because their bytes are large but their relationship to a commit is important. Model checkpoints under active development, test fixture assets, and design files tied to release states fit LFS. The choice depends on whether every clone that needs the file should obtain it through the same Git remote. If so, preserving that Git flow is useful. Run logs, temporary build outputs, intermediate datasets, screenshots, and exported PDFs instead fit run-scoped external storage when they are ephemeral.
Shipped binaries and package dependencies form another category. Consumers download them rather than edit them in a working tree, so they belong in artifact registries or release assets. For ML pipelines, DVC offers a further distinction by treating data as a first-class pipeline dependency and decoupling data versioning from source control. For agent-driven ML work, that model introduces another tool surface, remote configuration, and mental model. LFS remains the lower-friction choice for code projects with some large binaries.
Other version-control workflows also need checks to see what they support. Jujutsu’s compatibility documentation lists Git LFS and .gitattributes as unsupported. A different interface to version control therefore does not automatically preserve the large-file behavior a repository depends on. The storage decision must account for both the format in which history is kept and the workflow that actually reads and writes the files.
What code storage designed for cloud-native agents actually requires
An agent filesystem must support cheap forks for parallel attempts while keeping changes legible to humans. A changed block device is not a useful review artifact, whereas a diff on a named branch is. Merge, rollback, and promotion must work without copying the whole machine state. Large binaries and generated outputs must coexist with source without degrading those operations. Git supplies the essential project primitives.
The performance requirements extend beyond those primitives. Agents need faster random-access reads for targeted retrieval, separation of hot and cold artifact tiers, and GitHub Sync designed around agent commit patterns. Meeting those needs spans repository structure, runner provisioning, lifecycle policy, and merge queue design. It is an ongoing infrastructure decision.
The AIDev dataset contains 932,791 agent-authored pull requests from OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code across a large number of repositories. That scale places agent workflows firmly in production. The software factory needs a storage layer designed for those workflows. Teams that separate project state from run output and design retrieval around agent tasks will be better positioned for throughput than teams that inherit human checkout habits unchanged.
