Overview
Yarchi reviewed more than 70 engineering posts from teams at Stripe, Uber, Spotify, DoorDash, Shopify, Meta, Microsoft, Cloudflare, and others, then grouped the working systems into ten distinct agent setups. The central claim is that none of these companies rely on a single AI agent. Each uses a set of specialized agents, and each agent is designed differently depending on the task.
The model itself is the smallest part of the system. What companies spend months building is the harness around it: the loop that calls the model and runs tools, the context and memory that persist across turns, the sandbox that isolates execution, the MCP gateway that controls tool access, the guardrails that enforce read-only defaults and spend caps, and the verification layer that checks work before it ships.
Most production setups mix fixed workflows with agent loops. Stripe calls these blueprints: predetermined steps such as lint and push, with agent loops inserted only where judgment is required. The five recurring patterns are prompt chaining, routing, parallelization, orchestrator/workers, and evaluator/optimizer. The practical rule is to start with one agent and split only when instructions become tangled or tools start to overlap.
Why This Matters
Reliable agents fail or succeed on the harness, not on the choice of model. The companies in this write-up have already paid the cost of learning which architectures hold up under real load. Their published metrics -- thousands of merged pull requests per week, median review times under four minutes, mitigation times dropping from hours to minutes -- provide concrete evidence of what works.
The six rules that appear across every setup are immediately usable: keep a separate checker so the agent that does the work never grades itself, fetch data and evidence with plain code before the model sees it, default to read-only and require human approval for anything that changes state, require evidence for every claim, grant narrow permissions per task, and replay real past cases after every change. These constraints turn experimental loops into systems that can ship.
For anyone building daily automation, research agents, or second-brain knowledge systems, the ten setups offer a menu of proven starting points rather than a blank canvas. Pick the one that matches the most annoying recurring task, keep it narrow and read-only at first, add a checker, measure results, then expand.
Key Points
- Companies do not use one AI agent. They use sets of specialized agents, each designed for a different task such as code review, incident investigation, or data questions.
- The model is the smallest part. Months of work go into the harness: loop, context and memory, sandbox, tool access via MCP, guardrails, and verification.
- Most production systems mix workflows (fixed code paths) with agents (model-determined steps). Stripe calls the combination blueprints.
- Five patterns appear repeatedly: prompt chaining, routing, parallelization, orchestrator/workers, and evaluator/optimizer.
- Start with one agent. Split only when instructions become tangled or tools overlap. Microsoft reduced Azure SRE from 100+ tools and 50+ subagents after chains of more than four transfers failed.
- Background coding agents at Stripe merge more than 1,000 pull requests a week. Shopify River co-authors one in eight merged PRs. Spotify Honk vetoes about a quarter of sessions that go off scope.
- Code review agents keep only comments worth reading. Cloudflare runs 131,246 reviews in one month at a median cost of $0.98. Engineers override the bot on 0.6 percent of merge requests.
- Migration agents apply one change across thousands of files using parallel worktrees and fixed gates. Airbnb completed 97 percent of a 3,500-file test migration automatically.
- On-call and SRE agents investigate while humans are still opening laptops. Microsoft cut time-to-mitigation from a 40.5-hour average to 3 minutes on one service. Meta requires every conclusion to cite a specific tool query.
- Security agents build evidence packages with plain code before any model runs, then use different models for hunters and checkers. Cloudflare drops any finding without real evidence.
- Data analyst agents rely more on the knowledge layer (table descriptions, metric definitions, business rules) than on the model. DoorDash Vera raised pass rate from 43 percent to 90 percent on 1,900+ evals.
- Scheduled automation agents wake on a cron, read from bookmarks rather than “last 24 hours,” split memory into user preferences and agent state, and enforce a spend cap per run.
Quotable
AI-generated from source material. Verify important details against the original source.
Yarchi
The model is the smallest part. What companies actually spend months on is everything around it, the harness.
This is the central claim of the entire piece. Model choice is secondary to loop, sandbox, guardrails, and verification.
Yarchi
Anthropic’s definition is even shorter: agents “are typically just LLMs using tools based on environmental feedback in a loop.”
Strips the definition down to the essential loop that every production setup implements.
Yarchi
The agent never decides it’s done, the tests do.
Describes the migration-agent pattern where fixed gates, not the model, control progress.
Yarchi
A conclusion without a log line, query or source behind it gets dropped.
One of the six rules that appear in every successful setup. Evidence is enforced by architecture, not by prompt.
Yarchi
Start narrow, read-only, with one checker on top. Every company in this article started the same way: one boring task, measured, then the next one.
Practical advice for anyone beginning to build production agents rather than demos.
Concepts
The Harness
Everything around the model: the loop that calls the model, runs the requested tool, and returns the result; context and memory that survive across turns; an isolated sandbox; tool access typically mediated by an MCP gateway; guardrails such as read-only defaults, spend caps, and human approval; and verification via tests, linters, or a second agent.
Workflow versus Agent
Workflows run models and tools through predefined code paths. Agents let the model determine its own steps and tool calls. Production systems almost always mix both. Stripe’s blueprints fix steps such as lint and push while inserting agent loops only where judgment is required.
Five Recurring Patterns
Prompt chaining (each step processes the prior result), routing (classify input and send to a specialized handler), parallelization (break work into parts or run the same task multiple times and compare), orchestrator/workers (lead agent decomposes, workers execute, lead merges), and evaluator/optimizer (one agent creates, another critiques until approved).
Separate Checker
The agent that does the work never grades it. Spotify uses an LLM judge, DoorDash runs a “disprove it” pass, Shopify uses a deliberately different model. This prevents the system from rubber-stamping its own output.
Code Before the Model
Fetch data, run deterministic checks, and collect evidence with plain code first. The model only handles the parts that require judgment. This reduces hallucination surface and makes every conclusion traceable.
Read-Only by Default
Anything that changes the real world goes through a human: draft pull requests, approval buttons, or plan sign-off. Agents can look at everything and recommend actions, but they do not execute destructive or state-changing steps without explicit approval.
Evidence for Every Claim
A conclusion without a specific log line, query result, or source is dropped. Meta and Cloudflare enforce this in architecture so the model cannot invent supporting data.
One Agent First
Maximize a single agent before splitting. Microsoft’s first Azure SRE Agent had 100+ narrow tools and 50+ specialized subagents; chains of more than four transfers almost always failed. The rebuilt system uses a few generalists with wide CLI access.
Knowledge Layer
For data agents, table descriptions, metric definitions, example queries, and business rules matter more than the model. DoorDash keeps new data sources switched off until they pass an eval threshold. Notion runs nightly agents that find stale or conflicting definitions.
Expiring Facts and Self-Improvement
In second-brain systems, facts expire unless reinforced. When an answer is wrong, sub-agents draft a fix, an adversarial agent reviews it, and old cases are replayed before the change ships. Meta reduced tokens per question by about 80 percent with this approach.
Implementation
AI-generated from source material. Verify important details against the original source.
Pick the Single Most Annoying Recurring Task
Do not build all ten setups. Choose the one that matches the pain that currently consumes the most time: small tickets, pull-request reviews, bulk migrations, on-call alerts, repeated data questions, customer support, or morning checks across multiple tools. Every company started with one boring task, measured results, then expanded.
Default to Read-Only and Add a Separate Checker
The agent can inspect and recommend but cannot change state. Any write, merge, restart, or rollback requires explicit human approval. Run a second agent (or a different model) whose only job is to challenge the first agent’s output before anything is shown to a human.
Collect Evidence with Plain Code First
Before the model is called, run deterministic scripts that gather logs, diffs, dashboards, or past incidents into a versioned package. The model only receives this package and must cite specific items from it. Findings without matching evidence are dropped by code, not by prompt.
Use a Fresh Sandbox and Narrow Tool Permissions
Give each run an isolated environment that starts from a known state. Expose only the tools required for that task through an MCP gateway or equivalent. Prefer a small number of general tools over dozens of narrow ones. Microsoft’s experience shows that excessive specialization creates failure chains.
Mix Fixed Workflow Steps with Agent Loops
Hard-code the parts that must always happen (lint, test, format, open draft PR). Insert an agent loop only where judgment is required. The tests or gates, not the model, decide when a step is complete. This is the blueprint pattern used by Stripe and the state-machine approach used in migration agents.
Replay Real Past Cases After Every Change
Keep a set of historical tickets, incidents, questions, or pull requests with known correct outcomes. Rerun them after any change to the harness, prompts, or tools. DoorDash and Notion both use this to keep pass rates high and to catch regressions before they reach production.
Add Parallel Workers and a Citation Pass Only When Needed
For research or migration work, spawn sub-agents that each handle one angle or one file, then compress their results. Finish with a separate citation or reconciliation step. Track token cost; multi-agent research can use 15 times the tokens of a single agent.
Instrument and Cap Every Run
Log every tool call, decision, and cost. Set a hard spend cap per session so a stuck loop cannot burn money. For scheduled agents, remember the last bookmark per source so the next run continues where the previous one stopped rather than re-reading a sliding window.
Tools & Resources
These resources are curated in two groups. Mentioned Resources are pulled directly from the source material, and Suggested Resources are added to help you expand and apply the ideas beyond the original.
The following resources may contain affiliate links. As an Amazon Associate I earn from qualifying purchases at no extra cost to you. This does not influence the placement of links on this page.
| Resource | Notes |
|---|---|
| anthropics/claude-code-action | Mentioned starting point for background coding agents. Mention @claude on a GitHub issue or PR and it opens a pull request via GitHub Actions. |
| obra/superpowers | Contains code-review skills and the dispatching-parallel-agents plus using-git-worktrees skills recommended for migration agents. |
| trailofbits/skills | 42 security plugins for code auditing, smart contracts, and reverse engineering. Recommended starting point for security agents. |
| assafelovic/gpt-researcher | Planner, execution agents, and publisher for multi-agent research. Recommended starting point for research agents. |
| anthropics/claude-quickstarts (managed-agents/daily-brief) | Reference implementation for scheduled automation agents, including per-source bookmarks and spend caps. |
| Stripe Minions post | stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents -- 1,000+ merged PRs per week via blueprints and Toolshed MCP. |
| Cloudflare AI code review | blog.cloudflare.com/ai-code-review -- risk-based specialists, 131k reviews in one month, 0.6% override rate. |
| Resource | Notes |
|---|---|
| Building LLM Powered Applications | Practical patterns for tool use, evaluation, and production deployment that complement the harness ideas in the source. |
| AI Engineering (Chip Huyen) | Covers evaluation, monitoring, and the engineering practices required to move agents from prototype to production. |
| Anthropic Claude Quickstarts | Additional reference implementations beyond the daily-brief example, including support and data-agent patterns. |
| OpenAI SRE Agent cookbook | Approval-gated rollbacks and investigation state kept in a Slack thread. Mentioned as a starting point for on-call agents. |
AI Prompt
AI-generated from source material. Verify important details against the original source.
AI Implementation Prompt
CONTEXT You are working from Yarchi’s October 2026 summary of production AI agent systems drawn from more than 70 engineering posts by Stripe, Uber, Spotify, DoorDash, Shopify, Meta, Microsoft, Cloudflare, and others. The central thesis is that the model is the smallest part of a working agent. Reliability comes from the harness: the loop, sandbox, context and memory, MCP-mediated tool access, guardrails, and verification. Companies use sets of specialized agents rather than one general agent, and they mix fixed workflows with agent loops. KEY PRINCIPLES 1. Harness over model. Spend design effort on the loop, isolation, permissions, and verification, not on swapping models. 2. Separate checker. The agent that produces work never evaluates it. Use a second model, a disprove-it pass, or deterministic gates. 3. Code before the model. Gather evidence and run deterministic checks with plain code. The model only judges. 4. Read-only by default. Human approval is required for any state change. Drafts and recommendations are fine; merges and restarts are not. 5. Evidence required. Every claim must cite a specific log, query, or source. Architecture, not prompt, enforces this. 6. Start narrow. One boring recurring task, measured, then expand. Split agents only when instructions or tools become tangled. 7. Replay real cases. Maintain a set of past tickets, incidents, or questions and rerun them after every change. LEVERS - Sandbox freshness and isolation level - Number and breadth of tools exposed per task - Presence and strictness of the separate checker - Ratio of fixed workflow steps to free agent loops - Quality and coverage of the knowledge layer (for data agents) - Spend cap and bookmarking strategy (for scheduled agents) - Parallelism versus sequential depth for research or migration WHAT THIS IS NOT This is not a prompt for generating novel agent architectures from scratch or for writing marketing copy about AI. It is a prompt for designing or auditing a concrete agent system against the patterns that have already shipped at scale. MODES 1. Audit an existing agent against the six rules and ten setups. 2. Design the smallest viable harness for a specific recurring task. 3. Choose which of the ten setups matches a given workflow and list the first three implementation steps. 4. Identify missing verification or evidence requirements in a proposed design. OPERATING INSTRUCTIONS Stay grounded in the source patterns. Prefer concrete harness choices over abstract advice. When the user’s current system is described, map it explicitly to one or more of the ten setups and name the specific rule that is missing or weak. Challenge designs that let the producing agent grade itself or that give the model unrestricted write access. Ask clarifying questions when the task, tools, or success criteria are ambiguous. Do not recommend building all ten setups at once. GUIDED DISCOVERY Ask me up to three questions, one at a time, to determine: (1) which recurring task currently consumes the most time or creates the most risk; (2) whether the environment already has isolated sandboxes, MCP or equivalent tool gateways, and a way to replay past cases; (3) what the acceptable human-in-the-loop step is for any state-changing action. Once you understand the situation, propose the single setup to start with, the minimum harness, and the first verification check to put in place.