Agentic Pipeline Architecture for Large-Scale Codebases

A codebase that spans 400,000 files cannot fit inside any context window a model ships with today, and no increase in window size will change that fact, because the problem is not how much text a model can read but how a single sequential process can reason about a system that large, which is the structural ceiling that forces the move from one agent doing everything to a hierarchy of agents doing distinct, coordinated things.

Why a Single AI Agent Cannot Handle a Large Codebase

Picture an agent asked to change a permission check. Sourcegraph's 2026 guide frames this gap as the place where most output quality gets decided: context gathering, not code generation, is where an agent's plan either matches reality or doesn't.

Teams who have watched this play out call it the 80% problem, because the missed tail is nearly always the same handful of categories: auth middleware, serialization boundaries, audit trails, cross-repo tests, frontend guards, migration scripts. That consistency matters, because a problem that fails the same way each time is a design flaw, not noise that better prompting will average out.

Better retrieval alone does not fix this. A smarter search index might find thirty-one files instead of three, but the deeper issue is coherence: can one sequential process hold a plan that spans seven architectural layers, execute it correctly, and know when it has drifted from the original intent? At the scale of a 400,000-file monorepo, a single agent working step by step runs into a ceiling that no amount of context window expansion removes, because the limit is not storage, it's coordination. The field's response has been to stop asking a single model to do everything and instead build hierarchical systems of specialized, orchestrated agents. Coding, at this scale, has shifted from typing to orchestration.

The three-role hierarchy that makes multi-agent pipelines functional

The architectural answer that has emerged across production systems is a three-role hierarchy: a coordinator, one or more specialists, and a verifier. The coordinator decomposes a high-level task into bounded pieces of work and assigns them; it holds the plan and delegates, and it never touches a file directly. The verifier evaluates whether the output actually satisfies the original intent, not merely whether tests pass, and decides at the end of each cycle whether the work continues or needs to be redone.

So this division of labor is what lets a multi-agent pipeline produce coherent output at a scale no single process could manage. Gas Town runs twenty to thirty of these agents in parallel, and the system holds together because the roles are separated.

A source-code-level taxonomy of 13 open-source coding agent scaffolds backs this up empirically: eleven of the thirteen compose multiple loop primitives (ReAct, generate-test-repair, plan-execute, multi-attempt retry, tree search) as their control structure. That finding says something important: the three-role hierarchy is not one design choice among several equally valid options. When you examine production coding agents at the level of their actual control logic, this is close to how they already behave.

The obvious objection is that hierarchy adds coordination overhead and opens new failure points: a coordinator can misassign work, and a verifier can block progress on a false negative. That overhead buys something horizontal coordination cannot: when specialists talk directly to each other without a coordinator mediating, the result is architectural drift, each specialist making locally reasonable decisions that collectively pull the codebase in incompatible directions. So vertical delegation through a coordinator prevents that drift, because the coordinator is the one place where the plan stays whole.

What the scaffold around the model does to govern agent behavior

The code that surrounds the language model, the control loop, the tool definitions, the state management, and the context strategy, determines far more of an agent's behavior than which model sits inside it. That scaffold is where architectural decisions actually get made, and the taxonomy of 13 open-source scaffolds reveals wide variation underneath a seemingly uniform category of "coding agent": control strategies range from fixed pipelines to Monte Carlo Tree Search, tool counts range from zero to thirty-seven, and context compaction spans seven distinct strategies.

Two agents can both be described as "tool-using, planning, reflective" under a capability-based taxonomy and still differ enormously in cost, reliability, and the specific ways they fail, because those differences live in the scaffold that surrounds the model. Other dimensions remain unsettled and genuinely contested: context compaction strategy, state management, and multi-model routing are open design questions where teams are still making different bets.

So if you evaluate or build an agentic pipeline, that distinction has a practical consequence. The informative questions are about the scaffold: how does the control loop handle failure, how is state passed between agents, how is context compacted when a session runs long. Which model powers the system matters far less than these architectural answers, a point Sourcegraph's guide makes from the practitioner side as well, noting that many tools draw from the same frontier model families and that the real differentiator is the system built around the model: retrieval, tool use, planning, sandboxing, and review.

Managing context at codebase scale, from repository maps to just-in-time retrieval

Context infrastructure at scale has moved away from chunking and embedding an entire codebase ahead of time, because pre-inference retrieval like that cannot capture the dependency relationships that determine which files actually matter for a given change. Just-in-time, agent-driven retrieval has become the dominant answer: rather than guessing at indexing time what will matter later, agents search for what they need at the moment they need it.

The Repository Map pattern, pioneered by Aider, is one concrete version of this. Tree-sitter parses source code into an abstract syntax tree so it can pull out function signatures and class definitions, then a dependency graph gets built from those relationships, PageRank ranks which symbols matter most, and the resulting map fits dynamically within whatever token budget you have. The strategic move is to build a map of what connects to what, and let the agent query that map instead of trying to hold the whole codebase in memory at once.

Intent's Context Engine extends this approach with semantic dependency analysis across more than 400,000 files, surfacing functionally equivalent code that keyword search alone misses and extending architectural reasoning across the boundaries between separate repositories.

Sourcegraph's CodeScaleBench work treats this retrieval question as a cost problem. The benchmark measures how external code-retrieval tooling changes what agents produce on large-codebase and multi-repo tasks, and retrieval quality directly affects execution time, how many retries a task takes, and what the task ultimately costs to run. If an agent finds the wrong three files, it doesn't just produce a worse plan; it burns additional cycles discovering the other twenty-eight files it missed the first time, and every one of those cycles has a price.

Maintaining state across long-running tasks without losing earlier decisions

Multi-agent pipelines run into what amounts to a "50 First Dates" problem: agents with no memory between sessions, each one waking up fresh and capable of creating a state that conflicts with what a previous session already decided. Left unmanaged, agents tend to handle this by writing their own notes to themselves, and that habit produces what practitioners describe as a swamp of markdown files, each one slightly out of sync with the others, none of them a reliable record of what actually happened. It is an organizational failure as much as a technical one, and it is what a tool like Beads is built to solve: structured issue tracking designed for how agents actually need to read and write state.

Git worktrees have become the standard mechanism for isolating agents from each other at the file-system level. A Writer/Reviewer pattern often sits on top of this isolation: one agent writes code, a separate agent reviews it, and context gets cleared between sessions so that the reviewer isn't simply rubber-stamping the writer's own assumptions. So separation of concerns applies to state the same way it applies to role.

Intent's workspace model builds on the same worktree isolation, but adds something further: living specifications that function as self-updating contracts, continuously read from and written to by both humans and agents. So that structure is what stops requirement-implementation drift on tasks that run long enough for the original intent to get lost along the way. State management is a structural decision about the pipeline's architecture, shaping whether agents working in sequence build on each other's progress or quietly undo it.

The two protocols that now govern how agents talk to tools and to each other

Two protocols have stabilized to serve as the infrastructure layer for multi-agent systems: MCP for how agents talk to tools, and A2A for how agents talk to each other. A2A, the Agent-to-Agent Protocol from Google, standardizes how one agent delegates a task to another. IBM's competing Agent Communication Protocol merged into A2A in August 2025, consolidating what had been a fragmented layer into one standard, and A2A reached version 1.0 in March 2026, shipping alongside signed Agent Cards that give agents a verifiable identity. A large number of organizations now support it.

The production pattern that falls out of these two protocols maps directly onto the coordinator/specialist/verifier hierarchy described earlier: A2A governs communication between agents, MCP governs communication between agents and tools. So teams get something to build enterprise architecture around, rather than a collection of proprietary, one-off integrations that break every time a tool or framework changes.

That reach is also where the exposure sits. MCP servers increasingly connect language models directly to databases and other corporate resources, and the access an agent has through MCP is only as safe as the governance watching it. Beam AI's 2026 data found that only 47.1% of organizations with AI agents actively monitor or secure them, so the protocol now reaches well beyond what most organizations are watching. Prompt injection compounds the exposure: when an agent retrieves external data, processes it, and then acts on what it found, a single injected instruction embedded in that data can cascade through an entire workflow, turning a retrieval step into an attack surface. Enterprise observability systems such as SIEM, APM, and log aggregation were built to monitor infrastructure and application behavior, not the decision-making of an autonomous agent, so the protocol stack needs its own monitoring layer built specifically to watch what agents decide, not just what they execute.

Context compaction as a silent correctness failure in long-horizon agents

Context compaction, the practice of summarizing a running conversation once it nears its context limit, looks like resource management but behaves like a correctness risk. Claude Code implements compaction by summarizing conversations as they approach context limits, while trying to preserve architectural decisions, unresolved bugs, and implementation details. So that summarization is an editorial act performed on the agent's own running memory, and an editorial act can cut the wrong thing.

Research published in 2026 gives this failure a name: governance decay (arXiv:2606.22528), describing how context compaction can silently erase safety constraints in agents operating over long horizons. This failure is invisible to unit tests and appears later, in downstream behavior, often landing in the same categories the 80% problem already flags: auth middleware that no longer enforces what it used to, audit logs that stop recording a transition they were meant to record, permission guards that quietly stop matching their backend counterpart.

Compounding the problem, the scaffold taxonomy found that context compaction spans seven distinct strategies across the agents it examined, with no single approach treated as the default. The structural answer available today is to stop relying on conversation history as the sole place constraints live at all: living specifications and persistent state stores, such as Beads, hold constraints outside the compacted context, so that what an agent is required to do doesn't depend on whether a summarization pass happened to preserve it.

Verification loops: how the verifier prevents agents from executing efficiently against the wrong goal

The verifier role, introduced earlier as one leg of the coordinator/specialist/verifier hierarchy, is what closes the loop between what was specified and what actually got built. Its job is to check output against the original intent, not just against whether a test suite passes, because an agent can execute with complete efficiency toward a goal that quietly diverged from the one it was given. A specialist can write correct, well-tested code that solves the wrong problem, or build a permission check that passes every test while it mirrors an already-stale rule, or complete a task faithfully against a spec that compaction silently trimmed somewhere upstream. Tests alone cannot catch that category of failure, because they check consistency with the code as written, not fidelity to the intent the code was supposed to serve.

That is why the verifier sits at the end of every cycle: it decides whether the work actually satisfies the request before it ever reaches a developer, and it functions as a necessary structural component of the pipeline rather than an optional quality check layered on afterward. Every other piece of architecture covered here, the role hierarchy, the scaffold decisions, the context infrastructure, the state management, the protocol layer, the handling of compaction, exists to get an agent to a plausible answer. The verifier's entire function is to ask whether that answer was correct.