Hierarchical AI agent teams: how to build a supervisor that actually works

The moment you need more than one AI agent working on a problem, you need someone in charge. Not a democratic committee – a supervisor. Get the supervisor wrong and your multi-agent system either spins in circles or produces outputs that nobody actually asked for. I’ve been in both situations, and the second is somehow worse than the first.

Here’s what I’ve learned about building supervisors that don’t just look good in a diagram but actually hold up under production load.

What a hierarchical agent architecture looks like

The core pattern isn’t complicated: a top-level supervisor receives the overall task, breaks it into sub-tasks, routes each one to a specialist agent, collects the results, and synthesises them into a final output.

Think of it the way you’d think about a competent team lead. The supervisor is the project manager. The sub-agents are specialists – one handles research, one handles writing, one handles data analysis, one handles API calls. The supervisor doesn’t need deep expertise in each domain. It needs to know which specialist handles which type of job, how to define a task clearly enough for that specialist to execute it, and how to judge whether the result is actually good before passing it forward.

The LangGraph documentation includes a hierarchical_agent_teams notebook that demonstrates this architecture well. In their implementation, a top-level supervisor routes between different teams, each team has its own supervisor, and each team supervisor coordinates its own specialist agents. This nested hierarchy scales cleanly – you can add depth without redesigning everything.

For most use cases you’ll encounter as a founder, though, you don’t need three levels. A single supervisor with three to five specialist sub-agents handles most complex workflows. Start there before adding hierarchy.

Defining the supervisor’s decision criteria

The most common mistake in supervisor design is vague routing logic. If the supervisor doesn’t have clear criteria for which sub-agent to use when, it will either default to one agent for everything or make inconsistent decisions that produce incoherent outputs. I’ve debugged both failure modes more times than I’d like.

Good supervisor routing criteria have three characteristics.

They’re specific about task type, not topic. “Use the research agent when you need information from external sources” is a good criterion. “Use the research agent for anything research-related” is too vague – it causes the supervisor to misroute tasks involving internal data processing, which looks like research but isn’t.

They handle edge cases explicitly. What does the supervisor do when a task overlaps between two specialists? Define the tiebreaker upfront. Usually the right answer is decomposing the task into two sub-tasks rather than trying to force it into one agent’s lane.

They include a fallback. Some tasks will fall outside the defined specialisms. The supervisor needs a default behaviour for these: either attempt the task itself, surface it to a human, or return an explicit “cannot handle” response. A supervisor with no fallback will try to force every task into an existing category. That produces bad results and is hard to debug because the failure looks like poor output rather than a routing error.

In practice, this means writing the supervisor’s system prompt with explicit routing rules – not relying on the model to infer them. Be specific. List the sub-agents, describe exactly what each one handles, and give the supervisor clear decision criteria for ambiguous cases.

Avoiding loops

Loops are the most common production failure in multi-agent systems. Hierarchical architectures are particularly susceptible. The supervisor routes a task to a sub-agent. The sub-agent returns an incomplete result. The supervisor routes it again. The sub-agent returns the same incomplete result. Repeat until your API bill arrives.

There are several ways this happens, and each needs a different fix.

Unclear success criteria. The sub-agent doesn’t know what “done” looks like, so it keeps returning partial results. Fix: define explicit completion criteria in the task definition the supervisor sends. Include specific output format requirements, not just a description of what you want.

The supervisor re-routes failed tasks without modifying them. If a sub-agent fails, the supervisor needs to diagnose why before routing again. Simply re-sending the same task will produce the same failure. Fix: build explicit failure handling into the supervisor. When a sub-agent returns an error or a clearly inadequate result, the supervisor should modify the task, try a different sub-agent, or escalate.

Circular dependencies between sub-agents. Sub-agent A produces output that sub-agent B needs, but sub-agent B also produces output that sub-agent A needs. Fix: map your dependencies before you build. If you have circular dependencies, the workflow design is wrong. Restructure so information flows in one direction.

The practical safeguard is a maximum iteration count per task. If the supervisor hasn’t successfully completed a task within five routing attempts, surface the partial results to a human and ask for guidance. This prevents infinite loops from running up your costs and gives you visibility into where the system is actually struggling – which is usually a different place than you expect.

What to delegate vs what to keep at supervisor level

This design question affects every hierarchical system, and the answer isn’t always obvious until you’ve got it wrong a few times.

Keep at supervisor level: decisions that require context across multiple sub-tasks, quality assessment of final outputs, routing logic, and anything that requires synthesising results from multiple agents into a coherent whole. The supervisor has the broadest view of what’s happening. Decisions that require that breadth should stay there.

Delegate to sub-agents: specialised execution, tool use, data retrieval, domain-specific analysis, and any task that can be clearly defined and completed in isolation. Sub-agents work best when they have a specific, bounded task and the resources to complete it without needing to understand the bigger picture.

The mistake I see most often is delegating synthesis. Systems where the supervisor sends all sub-agent outputs to one more sub-agent and asks it to produce the final result. This rarely works well. Synthesis requires understanding the overall goal and the relationships between all the partial outputs – that’s supervisor-level reasoning. Keep it at the supervisor level.

Practical implementation: what the supervisor’s system prompt needs

The system prompt is where the supervisor’s behaviour gets defined. Here’s what it actually needs to contain.

A clear definition of the overall task type. What kinds of tasks is this supervisor built to handle? What’s explicitly out of scope?

A roster of sub-agents with specific capability descriptions. Not “the research agent does research” but “the research agent retrieves information from web search, Wikipedia, and the company knowledge base. Use it when you need factual information from external sources.”

Explicit routing rules with decision criteria. Include examples where the distinctions are subtle. A concrete example of a task that goes to sub-agent A versus one that goes to sub-agent B is worth more than a paragraph of abstract description.

Quality evaluation criteria. How does the supervisor assess whether a sub-agent’s output is good enough to proceed with? What specific markers should it check before moving forward?

Failure handling instructions. What should the supervisor do when a sub-agent fails, times out, or returns an obviously wrong result?

Output format requirements. What does the final synthesised output need to look like? Being specific here prevents the supervisor from returning whatever format feels natural on a given run, which will vary.

Connecting to the broader multi-agent picture

Hierarchical architecture is one pattern within a broader multi-agent design space. The multi-agent playbook covers the full landscape of patterns and when to use each. If you’re building a system that involves multiple agents, start there before committing to hierarchical as your architecture.

The supervisor pattern is the right choice when you have distinct specialisms that genuinely benefit from separation, a clear orchestration need, and tasks complex enough that a single agent can’t handle them reliably. It’s overkill for simple workflows. For complex ones, it’s often the only thing that scales.

Getting the handoffs right matters as much as getting the supervisor right. How the supervisor formats tasks for sub-agents, what metadata it passes along, and how it parses sub-agent responses all have significant effects on system reliability. I’ve covered AI handoff patterns in detail – worth reading alongside this.

The difference between a working supervisor and a broken one

A working supervisor makes routing decisions consistently, evaluates sub-agent outputs before passing them forward, handles failures without getting stuck, and produces coherent final outputs that reflect the combined work of all its sub-agents.

A broken supervisor routes tasks based on superficial features rather than genuine requirements, accepts any output from a sub-agent without checking quality, gets stuck in loops when sub-agents fail, and produces final outputs that don’t actually synthesise anything – just concatenating sub-agent results in sequence and calling it done.

The gap between the two is almost entirely in the design of the supervisor’s decision logic and the specificity of its system prompt. The models are capable of doing this well. The question is whether you’ve given them clear enough instructions.

Build the supervisor carefully. Test it against real tasks. Watch where it makes bad routing decisions and update the criteria. Iterate until the routing is consistent enough to trust. The supervisor is the brain of your multi-agent system – if that part is flaky, everything built on top of it will be flaky too.

For more on building AI agent systems that hold up under real-world conditions, see building autonomous workflows with AI agents and how AI agents are changing business operations.

Hierarchical AI agent teams: how to build a supervisor that actually works