Design the agent
before you deploy it.
DMADV for AI forward-deployed engineering. The phase map, the artefacts, and two composite cases. The cases are not client results. The figures are illustrative. The sequence is the point.
Rick Low · Hong Kong · 2026
DMAIC improves a process you already have. Most AI forward-deploy work does not have one. The engineer is sitting inside a customer’s operation, the requirement is still tacit, and the thing being built, an agent, a retrieval path, a decision workflow, does not exist yet. That is a design problem. Design for Six Sigma, run as DMADV, is the right frame: identify the problem before the solution, turn it into measurable critical-to-quality targets, design against those targets, and verify the design on real work before you call it deployed.
When to use DMADV, and when not to
Use DMADV when at least one of these is true.
- There is no stable process to improve. The work lives in inboxes, side chats, and senior judgement.
- The sponsor has named a solution, but the defect, what actually fails for the customer of the process, has not been defined.
- Meeting the target means a new mechanism: a new decision path, a new handoff, a new system of record for the judgement, not a faster version of the old form.
- The cost of a wrong answer is asymmetric. A payment release, a credit decision, a claims adjudication, a clinical coding suggestion.
Use DMAIC once the agent is in production and you are chasing drift: retrieval recall falling, tool-call errors rising, override rate climbing, cycle time slipping back. DFSS designs the thing. DMAIC keeps it capable. Running DMAIC on a greenfield agent is how teams spend a quarter improving a prompt that was aimed at the wrong job.
Define: separate the problem from the solution
Define is where most AI programmes skip ahead. The charter should forbid a solution in the problem statement.
Bad. Deploy an agent that reads payment instructions and emails the PSP.
Better. Same-day release of valid disbursement requests fails when the instruction pack is incomplete or inconsistent. Target: a defined share of clean cases released same day, with every reject traceable to a named rule.
Sponsor’s sentence
Deploy an agent that reads payment instructions and emails the PSP.
Solution word. Not a problem.Problem statement
Same-day release fails because incomplete or inconsistent packs are treated as clean. No product, vendor, or model in the sentence.
Four artefacts, and no more.
- A one-page charter. Business owner, process customer (the person who feels the defect, not the sponsor), in-scope and out-of-scope, and the explicit non-goals. For an agent, non-goals matter more than goals: no autonomous release above a threshold, no write to the core ledger, no use of customer free-text as a system instruction.
- Voice of the customer from the desk, not the slide. Sit with the people who do the work. Capture verbatim complaints and the workaround they actually use. A requirements document written offsite is a lossy summary of knowledge that only exists in the operation.
- A CTQ tree. A CTQ (critical to quality) is a measurable target the design must hit. Typical CTQs for an operations agent: first-time-right rate on the decision, cycle time from request to disposition, exception escape rate, abstain rate, and cost per case. Each CTQ needs an operational definition. “Accurate” is not a CTQ. “Agrees with the senior reviewer on the release, hold, or reject label, on a labelled set, with the evidence span cited” is a CTQ.
- The Y. The output is Y and the design parameters are the x’s: Y = f(x). Name Y before you name the model. If you cannot write Y, you are not ready to design.
A forward-deployed engineer who finishes Define with a model shortlist and no CTQ tree has not finished Define.
Measure: baseline the work, even though the agent does not exist yet
You cannot measure the variance of an architecture that has never run. The mistake is to treat that as a reason to skip Measure. The baseline is the current human process, not the future system.
Measure three things.
- Baseline capability of the work as done today. Cycle time distribution, rework rate, defect types, where judgement sits, how often the official SOP is not what the desk does. Pull this from cases, not interviews alone. Twenty-five to fifty recent cases, stratified by the types that hurt, beat a workshop.
- A measurement system for the future Y. Before any design, agree how a correct answer will be scored. Build the start of a golden set: inputs, expected disposition, the evidence that justifies it, and the cases that must be refused. If two seniors disagree on the label, you do not have a CTQ yet. You have an undefined standard. That disagreement is a finding, not an annoyance.
- The data the design will actually see. Documents, system fields, permissions, latency, what is missing, what is stale. The corpus on the share drive is not the corpus the desk uses.
Do not spend the phase simulating theoretical interfaces. A tracer set of real cases, scored against an agreed label, is the measurement system. Everything later is a comparison to that set.
Analyze: choose the concept before you choose the stack
Analyze turns CTQs into design requirements and kills weak concepts while they are still cheap.
- Quality function deployment, kept small. Rows are CTQs. Columns are design requirements: what the solution must do, not which vendor provides it. “Must cite the clause it relied on.” “Must refuse when the beneficiary name does not match the KYC record.” “Must finish inside the desk’s decision window.” The empty rows are the requirements you are about to forget.
- Concept selection, not a bake-off. Two or three concepts only. Score each against the CTQs and against kill-criteria: auditability, permission model, reversibility, time to a verified pilot. The Pugh matrix (each concept scored +1, 0, or −1 against the current desk) is enough. A weighted scoring model with false precision is not.
- Design FMEA (failure mode and effects analysis) for the ways it can actually fail. Hallucinated evidence. Wrong tool called with a valid-looking argument. Stale retrieval. Prompt injection from a document the agent was asked to read. Silent abstain that looks like success. A permission wider than the human role it replaces. The high-severity, low-detection rows become design constraints, not backlog items.
- The transfer gap. Current capability versus required capability, stated in the CTQ’s unit. If seniors are at 92 percent agreement with each other, do not set the agent a 99.9 percent target and call it Six Sigma. The measurement system caps the claim.
Analyze ends with one selected concept, a short list of design requirements, and a failure list the design is obliged to address. It does not end with an architecture diagram of every service you might use.
Design: parameters, tolerances, and the eval as the control
- Design parameters are the x’s you can set. Retrieval scope and chunking. Which tools exist and what arguments they accept. The policy on when to abstain. The human gate and what the human is shown. Temperature and model class matter less than these, and they are the parameters teams leave implicit.
- Tolerance design is what close enough means. A payment agent that is 97 percent label-accurate is not a 97 percent release agent. Outside the tolerance, low confidence, missing evidence, amount above the gate, beneficiary mismatch, the only acceptable output is a structured handoff, not a best guess. Write the tolerance as a rule the system enforces. Hope is not a tolerance.
- Designed experiments, kept narrow. Vary one family of parameters at a time against the golden set. Log the CTQ, not a vibe. Ten well-labelled cases that move the metric beat a hundred unscored demo prompts.
- The eval harness is the design control, built in this phase, not after launch. Every CTQ that can be scored offline gets a test. Every high-severity FMEA row gets a red-team case. The harness is a deliverable of Design, reviewed by the business owner.
Governance is part of the architecture: permissions, audit trail, and the human role are drawn before the first tool is wired. And the agent should sit where the work already happens. A side portal that staff must remember to open will lose to the inbox they already live in, however strong the eval score.
Verify: capability on live work, then the control plan
Verify is not a UAT script that checks the happy path. It answers one question: does this design meet the CTQs on work it did not train its story on?
Run a time-boxed pilot on live or recent-live cases, with the desk’s actual reviewers as the measurement system. Report each CTQ against its target, with the sample size and the disagreement rate between reviewers. Publish the escapes, bad cases that passed, by failure mode, mapped back to the FMEA. A pilot that only reports that users liked it has not verified anything.
Then write the control plan. Who owns the golden set. How often it is refreshed. What signal trips a rollback: override rate, abstain collapse, a new escape class. Who may change a design parameter. The hypercare window and the named owner after the forward-deployed engineer leaves.
Verification also includes the transfer. At least one pattern, a skill, an eval set, a gate design, should be fit to reuse. An engagement that ends as a bespoke agent and a Slack channel has not finished Verify.
Case study: a disbursement desk that asked for an email agent
A composite, built from the pattern of payment-instruction and disbursement work rather than a single named client. Figures are illustrative. The sequence is the point.
A regional lender’s disbursement desk released SME drawdowns to a payment partner by email. Volume sat at roughly 60 to 80 requests a day. The sponsor’s brief, written before anyone sat with the desk, was an agent that would read the instruction pack, draft the payment email, and send it. A pilot date was already on the steering slide.
Define
The forward-deployed engineer refused the brief as a problem statement and spent the first week on the desk. The process customer was the desk lead, not the sponsor. The defect the desk actually felt was a queue that breached same-day release by mid-afternoon, concentrated in packs that looked complete and were not: a beneficiary name that did not match the KYC record, a missing invoice reference, an amount that had been amended in a trailing email. Clean packs were not the problem. A senior could clear a clean pack in a few minutes. The dirty pack consumed the afternoon and, worse, two mismatches had been released in the prior quarter and reversed the next day.
The charter named three CTQs and three non-goals. The Y was same-day disposition of a valid request with no escaped mismatch. “An agent that emails the PSP” did not appear in the charter.
| CTQ | Operational definition | Pilot target |
|---|---|---|
| Same-day disposition of clean cases | Clean pack labelled release, with evidence cited, before the desk’s 16:00 cut-off. | 90% of cases the golden set called clean |
| Escape rate | A mismatch or incomplete pack labelled release. | Zero on the labelled mismatch class |
| Abstain quality | Missing required field or low match produces a structured hold, not a blank or a guess. | 100% of planted incomplete packs |
| Non-goal | Why it is refused |
|---|---|
| No autonomous send to the payment partner | Detection after send is too late for the escape CTQ. |
| No write to the loan system | The checker does not own the system of record. |
| No release above a stated amount gate | Severity is asymmetric. Outside tolerance, the only output is a hold. |
Non-goals are design constraints, not a later backlog.
Measure
Forty recent packs were pulled, stratified: twenty the desk had called clean, ten mismatches, ten incomplete. Two seniors labelled them independently. They agreed on 36 of 40. The four disagreements were all the same shape, a trading name on the invoice against a legal name on KYC, and were pulled out of autonomous scope until the rule was written down. That disagreement was the finding. Baseline on the human process: about 70 percent of all requests dispositioned same day, and the two reversed mismatches in the quarter sat inside the class the desk had called clean. The share-drive SOP was not what the desk used. The working rule lived in a checker’s notes and in the lead’s memory.
Analyze
Three concepts went onto a Pugh matrix against the CTQs and three kill-criteria: auditability, reversibility, and time to a verified pilot. The datum is the current desk, not the sponsor’s slide. Scale: +1 better than the desk, 0 the same, −1 worse. A kill row scored −1 fails the concept even if the sum looks acceptable. A kill means worse than the desk, not worse than perfect. The two reversed payments sit on the escape row. Auditability is whether the decision can be replayed, and the desk already can, because a person sends.
| Criterion | Kill? | A Email agent | B Drafting assistant | C Checker |
|---|---|---|---|---|
| Same-day disposition of clean cases | No | +1 | 0 | +1 |
| Escape rate on mismatch | Yes | −1 | 0 | +1 |
| Abstain quality | No | −1 | 0 | +1 |
| Auditability | Yes | −1 | 0 | +1 |
| Reversibility after a wrong release | Yes | −1 | +1 | +1 |
| Time to a verified pilot | No | +1 | +1 | 0 |
| Sum, ignoring the kill rule | −2 | +2 | +5 | |
| Decision | Killed | Not selected | Selected |
A · Email agent
B · Drafting assistant
C · Checker
A dies because it is faster on clean cases and quicker to demo, which is why the sum is not the point. A wrong release is detected after the payment partner has it. That fails the escape CTQ, auditability, and reversibility. B is not killed. The human still sends, so reversibility holds, but it moves the writing, which was not the bottleneck. C is the only concept that can hit the escape CTQ without pretending the model is a control: extract, compare to the KYC record and the request, release only inside a written tolerance, otherwise a structured hold naming the missing evidence. The human still sends.
The design FMEA put three rows at the top: hallucinated evidence span, a beneficiary match on a trading name, and a trailing email that amended the amount after the pack was read. Each became a design constraint. Seniors agreed with each other 90 percent of the time on this set. A 99.9 percent release claim was not available. The pilot target stayed inside what the measurement system could support.
Design
The parameters that mattered were not the model. They were the match rule (legal name, with trading name as a flagged exception), the required-field list, the amount gate, and the abstain message the desk saw in the queue tool they already used. A side portal was rejected in the design review. Two narrow comparisons were run on the forty packs: match-on-legal-name versus match-on-any-alias, and abstain-on-missing-field versus best-guess-and-flag. The alias rule cleared more cases and failed the escape CTQ. It was dropped. The eval harness, reviewed by the desk lead, held the forty packs, the four disputed trading-name cases, and a red-team set for the three FMEA rows. Changing a parameter without a harness rerun was written into the design as a blocked action.
Verify
The pilot ran beside the desk for two weeks, on live packs, with the lead as the measurement system. In the composite, clean cases dispositioned same day moved from the baseline near 70 percent of all requests to just over 90 percent of the cases the golden set called clean. Planted incomplete packs all abstained. In the composite, one new escape shape appeared in week one: a beneficiary match that passed because the KYC record itself was stale. That row went back into the FMEA, the golden set gained a staleness check, and the amount gate was not widened.
| Control item | Worked value |
|---|---|
| Label-standard owner | Desk lead. Disputed trading-name cases stay out of scope until this owner writes the rule. |
| Sample cadence | Weekly, ten live packs, stratified clean, mismatch, and incomplete. |
| Rollback trigger | Any escape on the labelled mismatch class, or a jump in the override rate on clean releases. |
| Who may change a parameter | No one, without a harness rerun signed by the desk lead. |
| Owner after the engineer leaves | Desk lead for the label standard. Operations for the queue. Engineering for the harness only. |
The weekly sample runs on live packs.
An escape on a kill CTQ.
Back to hold-only. The new row stays on the live list.
No parameter change until the desk lead signs the rerun.
The reusable piece taken out of the engagement was the checker pattern, extract, compare to the system of record, release only inside tolerance, structured hold otherwise, not the prompt. The sponsor still got an agent. They did not get the agent they had named. The one they got could be verified.
Case study: a claims intake queue that asked for a summariser
Same rule as the disbursement case: a composite, not a named client. The sponsor’s solution is different. The failure mode is the same. Design started at the tool.
A life insurer’s claims intake team handled death, disability, and medical claims across Hong Kong and a regional servicing hub. Volume was about 40 to 50 new claims a day. The brief on the steering slide was a summariser: read the claim pack, produce a one-page brief, drop it in the assessor’s queue. A vendor demo had already shown a fluent summary of a sample file.
Define
A week on the desk changed the problem. Assessors did not lose time because packs were long. They lost time because the pack was missing the document that changed the decision, and because two hubs applied the complete-pack rule differently. The process customer was the intake lead. The defect was a claim accepted into assessment that bounced back for a missing medical, a beneficiary form, or a policy exclusion the summary had smoothed over. Rework, not reading time, was the queue breach.
The Y was a correct complete-or-return disposition. “A one-page summary” was a design parameter, not the problem. Non-goals: no claim decision, no liability wording, no write to the claims system.
| CTQ | Operational definition | Pilot target |
|---|---|---|
| Right-first routing | Pack labelled complete or incomplete in line with the senior assessor, with the missing item named. | Match the agreement Measure records, and not above it. |
| Escape rate | An incomplete pack labelled complete and released to assessment. | Zero on the planted missing-document class |
| Summary fidelity | Every material fact in the brief traceable to a cited page. No fact added. | No unsupported fact on the red-team set |
Measure
Thirty-five recent packs, stratified across the three claim types and both hubs. Two seniors labelled complete or incomplete and listed the missing item. They agreed on 29 of 35. The target is to match that agreement. The six disputes were all disability claims where one hub treated a clinic letter as enough and the other required the full medical. That split was written down as an undefined standard and pulled out of autonomous scope. Baseline: about a quarter of packs accepted into assessment bounced within five days, and the bounce reason was a missing document in most of those cases. The SOP on the intranet was not the rule either hub used.
Analyze
Concept A, the sponsor’s summariser, was fluent and blind to the bounce. A summary can omit a missing document by saying nothing about it. It failed the escape CTQ and was killed. Concept B, a checklist bot that ticked documents present against a fixed list, was strong on the missing-document escape and weak on claim types whose complete rule was not a fixed list. It was kept as a component, not the concept. Concept C, a completeness checker with a cited brief attached, compared required documents by claim type to what was actually in the pack, returned a structured incomplete with the gap named, and only then attached a brief whose sentences each cite a page. The human still accepts the claim into assessment.
The design FMEA put three rows at the top: a summary that invents a diagnosis not in the file, a document present but for the wrong insured, and the hub rule split on disability medicals. Seniors agreed 83 percent of the time. An autonomous accept target above that was not available.
Design and verify
The parameters were the required-document list by claim type, the rule that a document must match the insured and the policy number, the abstain on disability medicals until the hubs signed one rule, and the brief schema that rejected a sentence with no page cite. One comparison was run on the thirty-five packs: free summary first, versus completeness check first and brief only if complete. The free summary cleared more packs and failed the escape CTQ. It was dropped. The checker sat in the intake queue the hubs already used.
Two weeks beside intake, with the lead as the measurement system. In the composite, packs released to assessment that later bounced for a missing document fell sharply on the claim types in scope, and every planted missing-document case returned a named gap. In the composite, one new escape appeared in week two: a medical in the pack for a dependent, not the insured, that the name match had passed on a shared surname. That row went back into the FMEA, the match rule was tightened to insured plus policy number, and disability claims stayed on abstain. The control plan named the intake lead as owner of the document standard, set a twice-weekly sample, and rolled back if an incomplete pack was labelled complete.
The piece taken out for reuse was the checker-then-brief pattern, not the summary prompt. The sponsor still got a brief on every complete pack. They did not get a summariser that could accept a claim by omitting what was missing.
What the two cases have in common
One sponsor asked for an email agent, the other for a summariser. In both, Measure found that seniors did not agree with each other, Analyze killed the named solution on the escape CTQ, and Verify refused to widen the gate when a new escape showed up. That is the case the method needs: not a model that scored well, a design that was not allowed to ship until the defect it was built for had a number.
Where classic DFSS breaks, and the patch
DFSS grew up on parts and processes with Gaussian variation. Agents fail differently: wrong state, ungrounded claim, a tool call that is syntactically valid and operationally false. Do not force a sigma conversion onto a 200-case labelled set and call it a capability study. Report the CTQ in its own unit, with a confidence interval, and keep the Six Sigma discipline, operational definitions, baselines, designed comparisons, a control plan, without the theatre of a sigma number the sample cannot support.
The other break is measurement latency. Waiting for a perfect baseline before any prototype is how DMADV earns a bad reputation in software. The patch is a thin tracer, not a skipped phase. A small labelled set and one concept built against it will teach you more in ten days than a quarter of interface simulation. The gate is still the gate: you do not widen autonomy until Verify clears the CTQs you wrote in Define. In the disbursement composite, that gate is what stopped Concept A from shipping on the strength of a demo.
What to take into the next engagement
Write the problem with no solution in it. Name the CTQs and the measurement system before you name the model. Score two concepts against those CTQs and against the failures you refuse to ship. Treat the eval harness and the abstain rule as design parameters, not as cleanup. Verify on the desk’s cases, then leave a control plan with an owner who is not you.
Appendix: the three sheets you leave behind
These are not the four Define artefacts. They are the sheets you leave behind. Blank fields are the template. The worked rows are the disbursement composite, so a reader can see a named solution die on the escape CTQ. Clear the worked figures before a live engagement, and do not describe them as a result.
Charter
| Field | What belongs here |
|---|---|
| Engagement | The work, not the tool. |
| Business owner | The person who can stop the engagement. |
| Process customer | The person who feels the defect, not the sponsor. |
| Problem statement | No product, vendor, or model in this sentence. |
| Y | The output the design must move, in the customer’s unit. |
| In scope / out of scope | Including the non-goals that are design constraints. |
| Three CTQs | Operational definition and pilot target on each. Accurate is not a CTQ. |
Gate out of Define: a reviewer who was not in the room can state the problem without using the sponsor’s solution word.
Pugh matrix
Score +1, 0, or −1 against the current desk. Mark kill rows before scoring starts. A kill-row failure fails the concept regardless of the sum. Selected is the best sum among concepts with zero kill failures. Ties are not selected.
Gate out of Analyze: one selected concept, the kill rows written as design constraints, and no architecture diagram of vendors you might use.
Control plan
| Field | What belongs here |
|---|---|
| Label-standard owner | A named person. Not a team. |
| Sample cadence | A rhythm the desk will actually keep. |
| Rollback trigger | A signal, not a feeling. One escape on a kill CTQ is enough. |
| Who may change a parameter | No one, without a harness rerun signed by the label owner. |
| Owner after the engineer leaves | The desk for the standard. Engineering for the harness only. |
Gate out of Verify: each CTQ reported against its target, with the sample size and the reviewer disagreement rate, and a named owner who is not the engineer.