How to design AI agents: a practical guide

Quick answer

Designing an AI agent system means defining six things before you write a line of code: the job the system needs to do, the roles each agent fills, the execution pattern (sequential or parallel), the tools each agent can access, the model assigned to each role, and where humans stay in the loop. Get those six right in the design phase and the implementation is straightforward.

Designing AI agents is not the same as prompting a single model. A single model is a tool you configure. An agent system is a team you design — with roles, responsibilities, handoffs, and accountability structures. The design work happens before implementation, and skipping it is the most common reason agent systems fail.

This guide covers each design decision in order: what it is, why it matters, and how to make it well. By the end you will have a mental model for designing any AI agent system — and a checklist you can use on your next one.

1. Start with the job, not the agent

The most common mistake in AI agent design is starting with the agents. Teams open an orchestration framework, define a few agents, connect them together, and then ask: does this do what we want?

It usually doesn't — because nobody defined what "what we want" means precisely enough.

Start with the job. Before you name a single agent, answer these questions:

• What is the outcome? What does success look like at the end of a successful run? Be specific. "Summarise support tickets" is not specific. "Produce a one-paragraph summary of each support ticket, categorised by issue type, within 30 seconds of ticket creation" is.
• What are the inputs? What does the system receive at the start of each run? Where does that input come from?
• What are the outputs? What does the system produce? Who or what consumes it?
• What decisions does the system need to make? Which of those decisions require human judgment, and which can be automated?
• What can go wrong? What are the failure modes, and what should the system do when they occur?

These questions force clarity before any technology choices are made. They also reveal whether the task is actually a multi-agent problem at all. If the answers are simple — one input, one output, no decisions, no failure modes — a single agent or even a simple workflow automation may be the right tool. Not every problem needs a team.

If the answers are complex — multiple phases, different capabilities required at different steps, decisions that need human review — then agent design is warranted. That complexity is exactly what agent systems are built to handle.

2. Break the job into roles

Once the job is clear, decompose it into roles. A role is the unit of design in an agent system. Each role represents one agent with one responsibility.

The principle is simple: one agent, one job. An agent that does research does research. An agent that drafts content drafts content. An agent that reviews output reviews output. The moment an agent does two things that could be done independently, you have a design problem — one that will surface as unpredictable behaviour, unclear accountability, or both.

How to decompose a job into roles:

Start with the phases of the work. If the job is "research, draft, and review a proposal," those are three phases — and three roles. Then ask: are any of those phases compound? Does "research" actually mean "search for information, evaluate sources, and synthesise findings"? If so, it might be three roles, not one.

The test for whether a phase should be split is whether the two halves require meaningfully different capabilities, tools, or models. If a single agent can do both well with the same model and the same tools, keep it as one role. If splitting would allow you to use a cheaper model for the simpler half, or give each half more focused instructions, split it.

If you're weighing whether you need multiple agents at all, one AI agent vs. many covers that earlier decision — and why the instinct to add more agents is usually wrong.

Two types of agent roles:

• Pipeline agents pass their output to a downstream agent. They are not the end of the workflow — they produce something that another agent needs.
• Leaf agents produce a final result. They are the end of a branch. Their output goes to a human or an external system, not to another agent.

Knowing which type each agent is shapes how you write its role definition. A pipeline agent needs to produce output in a format the downstream agent can consume. A leaf agent needs to produce output a human or external system can consume. These are different requirements and should be designed explicitly.

For a detailed guide to writing each component of an agent role, see Sub-agent design: the five components every agent needs and How to write a good AI agent role definition.

3. Choose the execution pattern

Once you have a set of roles, you need to decide how they connect. The execution pattern is the architecture of the system — how outputs move between agents and in what order.

There are two fundamental patterns, and most real workflows combine them:

Sequential (pipeline): Agent B waits for Agent A. Use this when B genuinely needs A's output to do its job. A drafter cannot start without a research brief. A reviewer cannot start without a draft. The dependency is real, and the sequencing reflects it.

Parallel (fan-out): Agents A, B, and C run at the same time, because their work is independent. Three research agents covering different data sources. Two classifiers checking the same input against different rubrics. Parallel execution is the right choice when there is no genuine dependency — running sequentially just wastes time.

Beyond these two fundamentals, a few common patterns are worth knowing:

The routing pattern: A classifier agent reads the input and sends it to the appropriate specialist. A support ticket goes to the billing agent, the technical agent, or the accounts agent depending on what the classifier decides. The classifier is cheap and fast; the specialists are capable and focused. Routing lets you handle heterogeneous inputs efficiently.

The reflection loop: An agent produces output; a separate reviewer evaluates it against a rubric; if it fails, the original agent revises and the reviewer checks again. The loop continues until the reviewer passes it or a maximum iteration count is reached. Always define a maximum — a loop without a termination condition is a production incident waiting to happen.

The hierarchical team: A coordinator agent breaks a complex goal into subtasks, delegates each to a specialist, collects results, and synthesises a final output. Best for open-ended complex tasks where the decomposition is part of the work.

Choosing the right pattern requires understanding the dependencies between roles. If you mapped the phases correctly in step 2, the execution pattern follows naturally: phases with genuine dependencies connect sequentially; phases without dependencies can run in parallel.

For a decision framework with worked examples, see Parallel vs. sequential agents: when to use each.

4. Assign tools and access

Each agent should have access to exactly the tools it needs — no more. This is the principle of least privilege applied to agent design, and it matters for three reasons:

Safety. An agent with access to tools it doesn't need has a larger blast radius when something goes wrong. An agent that can only read a database cannot accidentally write to it. An agent that can only send internal notifications cannot accidentally email a customer.

Debuggability. When an agent produces unexpected output, the first question is always "what did it have access to?" If the answer is "everything," the investigation is much harder. Narrow tool access makes failures easier to trace.

Quality. Agents with access to many tools often try to use them when they shouldn't. An agent that has a web search tool will use it even when the answer is already in its context. Narrow access keeps agents focused on their role.

Map tool access per agent, not per workflow. A research agent might need web search and a knowledge base. A drafting agent needs only the research output — no external access required. A reviewer needs the draft and a rubric — no tools at all beyond reading its inputs.

This mapping should happen during the design phase, not during implementation. By the time you're writing code, the tool assignments should already be decided. If you're discovering tool requirements during implementation, it's a sign the design phase was incomplete.

For how tool credentials and access policies work in practice, see the Tool credentials and permissions guide.

5. Route models to roles

Not every agent needs the most powerful — or most expensive — model. Model routing is the practice of assigning each agent the model that matches its requirements, rather than using a single model for the whole workflow.

The assignment logic is straightforward once you understand your roles:

For classification and routing: speed and cost matter more than capability. A classifier reading a support ticket and choosing between five categories is a pattern-matching task. A small, fast model handles it well and costs a fraction of a frontier model.

For synthesis and judgment: capability matters. An agent drafting a legal summary, producing a strategic recommendation, or deciding whether to escalate a complex case needs a capable model that can reason carefully.

For specialised domains: domain-specific models may outperform general frontier models on tasks like code review, medical record summarisation, or multilingual output. It is worth evaluating, not just defaulting to the biggest general model.

The model assignment should be explicit in the design — not left to runtime defaults or assumed from infrastructure. Every agent's model should be a conscious decision with a rationale, just like its role and tool access.

This matters at scale. A workflow that runs ten thousand times a day with five agents each using a frontier model has a very different cost profile than the same workflow using small models for classification and routing steps. Model routing is one of the most impactful cost levers available in agent system design.

For a full breakdown of routing strategy and worked examples, see Model routing in multi-agent workflows.

6. Add human checkpoints

Human checkpoints — gates — are places in the workflow where a person must review and approve before the system continues. They are not a sign that the system does not work. They are a design feature that keeps humans accountable for decisions with real consequences.

Gates should be designed in from the start, not added after something goes wrong. An afterthought gate is a patch on a workflow that was built to run without oversight. A designed-in gate is a deliberate decision: "at this point in the workflow, a human makes the call."

Where to put gates:

Before consequential, irreversible actions. Before a customer email is sent. Before a CRM record is updated. Before a contract is generated. Before a payment is processed. Before any output that would be embarrassing, costly, or legally significant if it were wrong.

A useful heuristic: if a human would be uncomfortable finding out the action happened without being told in advance, it needs a gate.

What a gate needs:

Every gate needs three things defined explicitly:

• On approve: what happens next? The workflow continues, the approved output is passed to the next step.
• On reject: what happens? The workflow stops, or a specific step reruns, or the output is returned to the originating agent for revision.
• On timeout: what happens if no one responds? The workflow pauses, escalates, or stops — but it should not proceed as if approved.

Timeout behaviour is the most commonly forgotten gate specification. In production, approvers are on holiday, miss notifications, or simply deprioritise the review. Define the timeout behaviour during design so the system handles it gracefully rather than stalling silently.

For a full treatment of gate placement and specification, see Human-in-the-loop: how to design approval gates.

7. Export and implement

When the six design decisions are made, you have a complete specification:

• The job the system does
• The roles, with scope, responsibilities, and skills
• The execution pattern connecting them
• The tools each agent can access
• The model assigned to each role
• The gates, with approve, reject, and timeout behaviour defined

This specification is the handoff to implementation. Whether you implement it yourself using an orchestration framework, hand it to an engineering team, or use a tool like Envelope that generates a structured spec directly, the implementation should follow the design — not discover the design as it goes.

A well-formed spec should be legible to non-technical stakeholders. The product lead should be able to read it and confirm the workflow matches what the business needs. The compliance team should be able to identify which steps have human review and which do not. The engineering team should be able to implement from it without inferring design decisions that were never made.

Once you have a spec, the next step is testing. How to test an AI agent before deploying it covers the five testing layers — isolation, integrations, handoffs, failure modes, and human gates. If you want to complete this design process without writing any code, designing a multi-agent system without writing code walks through it start to finish.

If the spec is not legible to a non-technical reader, it is not complete. Go back to step 1.

Design your AI agents in Envelope

Envelope turns a plain-language description of your workflow into a complete AI agent system — roles, model assignments, tool access, handoffs, and human review gates, all in a structured spec you can implement or export. Free to start, no code required.

Start designing →

Frequently asked questions

What is the difference between designing AI agents and prompting a single AI model?

Prompting a single model is a configuration task — you write instructions, the model responds, you iterate. Designing an agent system is an organisational design task — you define roles, responsibilities, dependencies, and accountability structures across multiple agents that work together. The skills are related but distinct. Prompt engineering produces better single-agent output; agent design produces better multi-agent systems.

How many agents does a typical AI agent system need?

Most production systems have between three and seven agents. Fewer than three is usually a single-agent task with unnecessary coordination overhead. More than seven is usually a sign that roles are not clearly defined — you end up with agents that overlap or that exist only to pass context from one place to another. Start with the minimum number required by the task structure and add agents only when there is a specific reason.

Do I need to be technical to design AI agents?

No. The design decisions — roles, execution pattern, tool access, model routing, gates — are organisational and process decisions, not engineering decisions. A business analyst, operations lead, or product manager can design a complete agent system. The implementation requires engineering. The design does not. This is why designing before building matters: non-technical stakeholders can review and approve the design before any code is written.

What is the most common mistake teams make when building AI agent systems?

Skipping the design phase. Teams reach for an orchestration framework, define a few agents, connect them together, and discover the design problems during implementation — or in production. The problems are usually architectural: unclear role boundaries, missing handoff specifications, no gates, no failure handling. These are expensive to fix after the fact. A design phase that takes a few hours saves days of rebuilding.

How do I know which execution pattern to use — sequential or parallel?

Follow the data dependencies. If agent B genuinely needs agent A's output to do its job, run them sequentially. If their work is independent — they can run at the same time without affecting each other's output — run them in parallel. Do not default to sequential because it is simpler to reason about. Unnecessary sequencing introduces latency and adds no reliability benefit. See Parallel vs. sequential agents: when to use each for a decision framework.

What model should each agent use?

Match the model to the agent's requirements, not the workflow's overall budget. Fast, cheap models are appropriate for classification, routing, and pattern-matching tasks. Capable frontier models are appropriate for synthesis, judgment, and high-stakes decisions. Specialised models may be appropriate for domain-specific tasks. Every assignment should be a conscious decision with a rationale — not a default. See Model routing in multi-agent workflows for the full framework.

What should go in an agent role definition?

A role definition should include: a name that describes the function (not the action), a scope statement covering what the agent is responsible for, explicit out-of-scope boundaries, the success condition for a completed run, and the skills — the instructions, rules, and operating procedures the agent runs on. The role defines what the agent is; the skills define how it does its job. See How to write a good AI agent role definition for a full guide.

How do I handle failures in an AI agent system?

Design failure handling during the design phase, not after. Every agent that calls an external tool needs a defined failure response — what the agent does when the tool returns an error or times out. Every gate needs a defined timeout behaviour. Every sequential step needs a defined recovery path if the upstream agent produces unusable output. Failure handling is not an edge case — it is a regular operating condition. Systems that handle failures gracefully were designed for them.