Beyond the Prompt: The 7 Architectural Building Blocks of Production AI Agents
Reliability in agentic AI rarely comes from a better model alone—it comes from the deterministic architecture built around it: tools, state, oversight, and evaluation.
Rick Low · Hong Kong
A standard chatbot can explain how to fix a failing test. An AI agent, however, can inspect a codebase, edit a file, run tests, interpret execution errors, and decide what to try next. While this capability sounds transformative, it introduces fundamental software engineering questions:
- What boundaries restrict what the agent can modify in your infrastructure?
- Where does its context come from, and how is memory safely managed without polluting future executions?
- How can you programmatically verify that the agent actually solved the problem correctly rather than claiming success without proof?
Reliability in agentic AI systems is rarely achieved by simply upgrading the underlying Large Language Model (LLM). Instead, it depends on the system architecture built around the model—the deterministic framework controlling tool execution, state isolation, human oversight, and evaluation.
Below is a detailed guide to the seven architectural building blocks senior AI engineers use to turn probabilistic models into dependable production software.
Visualizing the 7-Layer Agentic Architecture Framework
The following diagram shows how these seven patterns function as composable building blocks. They are not competing frameworks; they wrap around a core execution task—such as diagnosing and patching an authentication bug—to form a safer, more deterministic system:
Detailed Breakdown of the 7 Architectural Building Blocks
1. Tool Calling Architecture
- Core Principle: The LLM requests actions; the host application validates, scopes, and executes them.
- The Door Metaphor: A tool acts as a small, well-defined door into another system. The agent can request to open the door, but the trusted host application decides whether it is allowed and controls what can be reached once opened.
- Key Implementation Details:
- Direct access to production databases or raw system credentials must never be handed directly to the model.
- Tools need clear, descriptive names, single-purpose responsibilities, and easily validatable parameters (e.g.,
search_repository_codeinstead of vague tools likefind_stuff). - Scope permissions tightly—a tool designed to read repository contents must not have permissions to mutate data or issue network requests.
2. Planning Architecture
- Core Principle: Complex tasks require dynamic, step-by-step discovery rather than rigid, upfront scripts.
- Key Implementation Details:
- Incremental Planning over Upfront Planning: Rather than forcing the agent to map out a 20-step plan at the start, it is safer to plan 2–3 steps ahead, take an action, observe the result, and adapt. If an unexpected authentication library is discovered mid-task, the agent can revise its route.
- When NOT to Use Agents: If a task follows the same predictable steps every time (e.g., validate input → fetch record → parse JSON → return), write deterministic code instead of using LLM planning.
- Stopping Conditions: Planning systems need hard boundaries—step limits, timeout windows, and fallback behaviors—to prevent infinite loops when an agent repeatedly retries a broken step.
3. Memory Architecture
- Core Principle: Separate ephemeral short-term working context from durable long-term institutional knowledge.
- The Notebook Metaphor: Memory should act like a selective notebook, not a second brain. Information is saved because a future execution explicitly needs it, not simply because the model happened to see it.
- Two Memory Domains:
- Working Context: Ephemeral context for the current run—the prompt, inspected file snippets, active code diffs, and recent test logs.
- Durable Memory: Selective, persistent knowledge (project conventions, architectural decisions, unresolved issues) loaded across sessions.
- Governance and Security: Loading thousands of past chat logs bloats context windows, raises latency, and risks introducing outdated or conflicting instructions. Stored preferences can go stale; one instruction can conflict with a newer one; information from one user must not leak into another user's session. Stored text must never be automatically treated as a trusted system instruction, or you open the door to prompt injection. Memory is therefore a governance problem as well as a storage problem: define what can be saved, who can access it, how long it remains useful, and how it can be corrected or deleted.
4. Multi-Agent Architecture
- Core Principle: Divide broad objectives into specialized roles coordinated by an orchestrator agent.
- Key Implementation Details:
- Orchestrator + Specialist Pattern: A primary orchestrator assigns tasks to sub-agents with narrow focus areas—such as a Code Inspector, a Security Reviewer, and a Test Analyzer—and synthesizes their outputs.
- Avoiding Agent Bloat: Adding more agents does not automatically make a system smarter; each agent adds instruction overhead, communication complexity, context switching, and potential failure points. Naming an agent "Security Reviewer" does not make it one—it still needs suitable instructions, information, tools, and evaluation.
- The Decision Rule: Ask: "What becomes clearer, safer, or more reliable when this work is given to another agent?" If there is no clear security or clarity benefit, keep a single agent. Creating a separate agent just to locate a file or run one test is usually unnecessary.
5. Human Approval Architecture
- Core Principle: Pause execution before performing high-risk, hard-to-reverse actions.
- Approval vs. Notification: Approval occurs before an action takes place. Sending a chat alert after an action has already executed is only a notification, not human gating.
- Risk Spectrum Gating:
- Low Risk (Auto-Approved): Reading files, searching codebases, editing in a limited workspace, running local sandbox unit tests, and often creating a draft pull request.
- High Risk (Human-Gated): Merging pull requests, deploying services to production, deleting production data, changing permissions, modifying live database schemas, issuing financial transactions, or sending a message on someone's behalf.
- Deterministic Policy Enforcement: Application code must enforce the human approval policy, not the model. The LLM can justify why it wants to merge a pull request; host software decides whether human sign-off is required. Approval is one limit layer; narrow tool permissions are another.
6. Sandboxed Execution Architecture
- Core Principle: Decouple the reasoning control plane from the execution environment.
- Key Implementation Details:
- Control Plane vs. Execution Environment: The control plane manages prompt logic, permissions, approvals, memory retrieval, and audit records. The execution environment (sandbox) is where model-generated shell commands run and local code diffs are applied.
- Sandboxing vs. Human Approval: Human approval decides whether an action should proceed; sandboxing restricts the blast radius if an executed command goes wrong.
- Sandbox Requirements: Isolated environments (for example containers, MicroVMs, or WebAssembly sandboxes) must enforce strict resource limits, restrict network connectivity, isolate local file systems, and strip access to production credentials. If the agent can reach every secret and production service from inside the sandbox, the boundary is not doing much.
7. Evaluation Architecture
- Core Principle: Evaluate both the final outcome and the full step-by-step path taken to reach it.
- The Process Path Principle: The path an agent takes matters as much as its final answer. A correct patch produced by calling an unauthorized endpoint or bypassing security protocols is still a system failure.
- Two-Tier Verification:
- Precise Automated Checks: Did the unit test suite pass? Did the agent stay within its pre-approved tools and file directories?
- Judgment-Based Checks: Is the code fix maintainable and idiomatic? Does the PR explanation accurately reflect the bug? Evaluated via human code review or defined LLM rubrics.
- Run Records and Benchmark Suites: Record what happened on each run—tools called, results returned, where the agent paused, duration, and completion status. Maintain a suite of representative tasks and re-run it whenever models, prompts, tools, or permissions change, so regressions surface early.
End-to-End Deep Dive: Resolving an Auth Bug
To see how these seven building blocks work together, trace how an agent resolves a token validation bug in an authentication microservice:
| Execution Step | Architecture Layer | Detailed System Operation |
|---|---|---|
| 1. Request & Context | 3. Memory | Developer submits the task. The host retrieves relevant coding conventions and token validation rules from Durable Memory and loads them into Working Context. |
| 2. Discovery & Search | 1. Tool Calling & 2. Planning | The agent plans its first step and calls something like search_repository_code for authentication / login validation. The host checks permissions, runs the search, and returns matching file lines. |
| 3. Sandboxed Edit | 6. Sandboxed Execution | The agent generates a fix inside an isolated container with a temporary git clone. It has no network path to production or developer credential stores. |
| 4. Multi-Agent Audit | 4. Multi-Agent | When it adds value, the orchestrator routes the proposed patch to a Security Reviewer agent that inspects the diff for issues such as missing null checks or token-handling gaps. |
| 5. Verification Run | 7. Evaluation | The agent runs unit tests in the sandbox. Automated checks confirm tests pass; audit logs confirm no unauthorized tool calls. |
| 6. Escalated Release | 5. Human Approval | Creating the PR may be auto-allowed. When the agent attempts to merge or deploy, the host intercepts and requires explicit developer sign-off. |
Strategic Roadmap: Phased Architecture Adoption
Building all seven blocks on day one often produces over-engineered, hard-to-debug systems. Senior engineers add complexity as task risk and autonomy grow:
- Phase 1 (Core Tools & Evals): Prefer deterministic code where the path is fixed. Add simple Tool Calling with strict arguments and basic Evaluation checks to measure success.
- Phase 2 (Dynamic Workflows): Introduce Planning when sequences depend on runtime discovery. Add Memory so working run state stays separate from persistent conventions.
- Phase 3 (Enterprise Hardening): Add Sandboxed Execution to isolate side effects, enforce host-driven Human Approval for high-stakes actions, and split work into Multi-Agent roles only when responsibilities divide cleanly.