Back
A practitioner article

Design the agent
before you deploy it.

DMADV for AI forward-deployed engineering. The phase map, the artefacts, and two composite cases. The cases are not client results. The figures are illustrative. The sequence is the point.

Rick Low · Hong Kong · 2026

DMAIC improves a process you already have. Most AI forward-deploy work does not have one. The engineer is sitting inside a customer’s operation, the requirement is still tacit, and the thing being built, an agent, a retrieval path, a decision workflow, does not exist yet. That is a design problem. Design for Six Sigma, run as DMADV, is the right frame: identify the problem before the solution, turn it into measurable critical-to-quality targets, design against those targets, and verify the design on real work before you call it deployed.

When to use DMADV, and when not to

Use DMADV when at least one of these is true.

Use DMAIC once the agent is in production and you are chasing drift: retrieval recall falling, tool-call errors rising, override rate climbing, cycle time slipping back. DFSS designs the thing. DMAIC keeps it capable. Running DMAIC on a greenfield agent is how teams spend a quarter improving a prompt that was aimed at the wrong job.

Define: separate the problem from the solution

Define is where most AI programmes skip ahead. The charter should forbid a solution in the problem statement.

Bad. Deploy an agent that reads payment instructions and emails the PSP.

Better. Same-day release of valid disbursement requests fails when the instruction pack is incomplete or inconsistent. Target: a defined share of clean cases released same day, with every reject traceable to a named rule.

Figure 1 · The charter gateComposite

Sponsor’s sentence

Deploy an agent that reads payment instructions and emails the PSP.

Solution word. Not a problem.

Problem statement

Same-day release fails because incomplete or inconsistent packs are treated as clean. No product, vendor, or model in the sentence.

Gate out of DefineA reviewer who was not in the room can state the problem without using the sponsor’s solution word.

Four artefacts, and no more.

A forward-deployed engineer who finishes Define with a model shortlist and no CTQ tree has not finished Define.

Measure: baseline the work, even though the agent does not exist yet

You cannot measure the variance of an architecture that has never run. The mistake is to treat that as a reason to skip Measure. The baseline is the current human process, not the future system.

Measure three things.

Do not spend the phase simulating theoretical interfaces. A tracer set of real cases, scored against an agreed label, is the measurement system. Everything later is a comparison to that set.

Analyze: choose the concept before you choose the stack

Analyze turns CTQs into design requirements and kills weak concepts while they are still cheap.

Analyze ends with one selected concept, a short list of design requirements, and a failure list the design is obliged to address. It does not end with an architecture diagram of every service you might use.

Design: parameters, tolerances, and the eval as the control

Governance is part of the architecture: permissions, audit trail, and the human role are drawn before the first tool is wired. And the agent should sit where the work already happens. A side portal that staff must remember to open will lose to the inbox they already live in, however strong the eval score.

Verify: capability on live work, then the control plan

Verify is not a UAT script that checks the happy path. It answers one question: does this design meet the CTQs on work it did not train its story on?

Run a time-boxed pilot on live or recent-live cases, with the desk’s actual reviewers as the measurement system. Report each CTQ against its target, with the sample size and the disagreement rate between reviewers. Publish the escapes, bad cases that passed, by failure mode, mapped back to the FMEA. A pilot that only reports that users liked it has not verified anything.

Then write the control plan. Who owns the golden set. How often it is refreshed. What signal trips a rollback: override rate, abstain collapse, a new escape class. Who may change a design parameter. The hypercare window and the named owner after the forward-deployed engineer leaves.

Verification also includes the transfer. At least one pattern, a skill, an eval set, a gate design, should be fit to reuse. An engagement that ends as a bespoke agent and a Slack channel has not finished Verify.

Case study: a disbursement desk that asked for an email agent

A composite, built from the pattern of payment-instruction and disbursement work rather than a single named client. Figures are illustrative. The sequence is the point.

A regional lender’s disbursement desk released SME drawdowns to a payment partner by email. Volume sat at roughly 60 to 80 requests a day. The sponsor’s brief, written before anyone sat with the desk, was an agent that would read the instruction pack, draft the payment email, and send it. A pilot date was already on the steering slide.

Define

The forward-deployed engineer refused the brief as a problem statement and spent the first week on the desk. The process customer was the desk lead, not the sponsor. The defect the desk actually felt was a queue that breached same-day release by mid-afternoon, concentrated in packs that looked complete and were not: a beneficiary name that did not match the KYC record, a missing invoice reference, an amount that had been amended in a trailing email. Clean packs were not the problem. A senior could clear a clean pack in a few minutes. The dirty pack consumed the afternoon and, worse, two mismatches had been released in the prior quarter and reversed the next day.

The charter named three CTQs and three non-goals. The Y was same-day disposition of a valid request with no escaped mismatch. “An agent that emails the PSP” did not appear in the charter.

CTQOperational definitionPilot target
Same-day disposition of clean casesClean pack labelled release, with evidence cited, before the desk’s 16:00 cut-off.90% of cases the golden set called clean
Escape rateA mismatch or incomplete pack labelled release.Zero on the labelled mismatch class
Abstain qualityMissing required field or low match produces a structured hold, not a blank or a guess.100% of planted incomplete packs
Non-goalWhy it is refused
No autonomous send to the payment partnerDetection after send is too late for the escape CTQ.
No write to the loan systemThe checker does not own the system of record.
No release above a stated amount gateSeverity is asymmetric. Outside tolerance, the only output is a hold.

Non-goals are design constraints, not a later backlog.

Measure

Forty recent packs were pulled, stratified: twenty the desk had called clean, ten mismatches, ten incomplete. Two seniors labelled them independently. They agreed on 36 of 40. The four disagreements were all the same shape, a trading name on the invoice against a legal name on KYC, and were pulled out of autonomous scope until the rule was written down. That disagreement was the finding. Baseline on the human process: about 70 percent of all requests dispositioned same day, and the two reversed mismatches in the quarter sat inside the class the desk had called clean. The share-drive SOP was not what the desk used. The working rule lived in a checker’s notes and in the lead’s memory.

Analyze

Three concepts went onto a Pugh matrix against the CTQs and three kill-criteria: auditability, reversibility, and time to a verified pilot. The datum is the current desk, not the sponsor’s slide. Scale: +1 better than the desk, 0 the same, −1 worse. A kill row scored −1 fails the concept even if the sum looks acceptable. A kill means worse than the desk, not worse than perfect. The two reversed payments sit on the escape row. Auditability is whether the decision can be replayed, and the desk already can, because a person sends.

CriterionKill?A Email agentB Drafting assistantC Checker
Same-day disposition of clean casesNo+10+1
Escape rate on mismatchYes−10+1
Abstain qualityNo−10+1
AuditabilityYes−10+1
Reversibility after a wrong releaseYes−1+1+1
Time to a verified pilotNo+1+10
Sum, ignoring the kill rule−2+2+5
DecisionKilledNot selectedSelected
Figure 2 · A faster demo is not a selection. Headline rows onlyComposite

A · Email agent

Clean cases, time to a demo+1
Escape · kill−1
Auditability · kill−1
Reversibility · kill−1
Killed. Three kill rows. Sum −2 is not the point.

B · Drafting assistant

Clean cases0
Escape · kill0
Reversibility · kill+1
Sum+2
Not selected. It moved the writing, not the bottleneck.

C · Checker

Escape · kill+1
Kill-row failures0
Sum+5
Who sendsThe human
Selected. The model is not the control.

A dies because it is faster on clean cases and quicker to demo, which is why the sum is not the point. A wrong release is detected after the payment partner has it. That fails the escape CTQ, auditability, and reversibility. B is not killed. The human still sends, so reversibility holds, but it moves the writing, which was not the bottleneck. C is the only concept that can hit the escape CTQ without pretending the model is a control: extract, compare to the KYC record and the request, release only inside a written tolerance, otherwise a structured hold naming the missing evidence. The human still sends.

The design FMEA put three rows at the top: hallucinated evidence span, a beneficiary match on a trading name, and a trailing email that amended the amount after the pack was read. Each became a design constraint. Seniors agreed with each other 90 percent of the time on this set. A 99.9 percent release claim was not available. The pilot target stayed inside what the measurement system could support.

Design

The parameters that mattered were not the model. They were the match rule (legal name, with trading name as a flagged exception), the required-field list, the amount gate, and the abstain message the desk saw in the queue tool they already used. A side portal was rejected in the design review. Two narrow comparisons were run on the forty packs: match-on-legal-name versus match-on-any-alias, and abstain-on-missing-field versus best-guess-and-flag. The alias rule cleared more cases and failed the escape CTQ. It was dropped. The eval harness, reviewed by the desk lead, held the forty packs, the four disputed trading-name cases, and a red-team set for the three FMEA rows. Changing a parameter without a harness rerun was written into the design as a blocked action.

Verify

The pilot ran beside the desk for two weeks, on live packs, with the lead as the measurement system. In the composite, clean cases dispositioned same day moved from the baseline near 70 percent of all requests to just over 90 percent of the cases the golden set called clean. Planted incomplete packs all abstained. In the composite, one new escape shape appeared in week one: a beneficiary match that passed because the KYC record itself was stale. That row went back into the FMEA, the golden set gained a staleness check, and the amount gate was not widened.

Control itemWorked value
Label-standard ownerDesk lead. Disputed trading-name cases stay out of scope until this owner writes the rule.
Sample cadenceWeekly, ten live packs, stratified clean, mismatch, and incomplete.
Rollback triggerAny escape on the labelled mismatch class, or a jump in the override rate on clean releases.
Who may change a parameterNo one, without a harness rerun signed by the desk lead.
Owner after the engineer leavesDesk lead for the label standard. Operations for the queue. Engineering for the harness only.
Figure 3 · The control loop after the engineer leavesComposite
01

The weekly sample runs on live packs.

→
02

An escape on a kill CTQ.

↓
03

Back to hold-only. The new row stays on the live list.

←
04

No parameter change until the desk lead signs the rerun.

↑
Then the sample runs againThe engineer has left. Widening the gate goes around this loop. It is not a follow-up.

The reusable piece taken out of the engagement was the checker pattern, extract, compare to the system of record, release only inside tolerance, structured hold otherwise, not the prompt. The sponsor still got an agent. They did not get the agent they had named. The one they got could be verified.

Case study: a claims intake queue that asked for a summariser

Same rule as the disbursement case: a composite, not a named client. The sponsor’s solution is different. The failure mode is the same. Design started at the tool.

A life insurer’s claims intake team handled death, disability, and medical claims across Hong Kong and a regional servicing hub. Volume was about 40 to 50 new claims a day. The brief on the steering slide was a summariser: read the claim pack, produce a one-page brief, drop it in the assessor’s queue. A vendor demo had already shown a fluent summary of a sample file.

Define

A week on the desk changed the problem. Assessors did not lose time because packs were long. They lost time because the pack was missing the document that changed the decision, and because two hubs applied the complete-pack rule differently. The process customer was the intake lead. The defect was a claim accepted into assessment that bounced back for a missing medical, a beneficiary form, or a policy exclusion the summary had smoothed over. Rework, not reading time, was the queue breach.

The Y was a correct complete-or-return disposition. “A one-page summary” was a design parameter, not the problem. Non-goals: no claim decision, no liability wording, no write to the claims system.

CTQOperational definitionPilot target
Right-first routingPack labelled complete or incomplete in line with the senior assessor, with the missing item named.Match the agreement Measure records, and not above it.
Escape rateAn incomplete pack labelled complete and released to assessment.Zero on the planted missing-document class
Summary fidelityEvery material fact in the brief traceable to a cited page. No fact added.No unsupported fact on the red-team set

Measure

Thirty-five recent packs, stratified across the three claim types and both hubs. Two seniors labelled complete or incomplete and listed the missing item. They agreed on 29 of 35. The target is to match that agreement. The six disputes were all disability claims where one hub treated a clinic letter as enough and the other required the full medical. That split was written down as an undefined standard and pulled out of autonomous scope. Baseline: about a quarter of packs accepted into assessment bounced within five days, and the bounce reason was a missing document in most of those cases. The SOP on the intranet was not the rule either hub used.

Analyze

Concept A, the sponsor’s summariser, was fluent and blind to the bounce. A summary can omit a missing document by saying nothing about it. It failed the escape CTQ and was killed. Concept B, a checklist bot that ticked documents present against a fixed list, was strong on the missing-document escape and weak on claim types whose complete rule was not a fixed list. It was kept as a component, not the concept. Concept C, a completeness checker with a cited brief attached, compared required documents by claim type to what was actually in the pack, returned a structured incomplete with the gap named, and only then attached a brief whose sentences each cite a page. The human still accepts the claim into assessment.

The design FMEA put three rows at the top: a summary that invents a diagnosis not in the file, a document present but for the wrong insured, and the hub rule split on disability medicals. Seniors agreed 83 percent of the time. An autonomous accept target above that was not available.

Design and verify

The parameters were the required-document list by claim type, the rule that a document must match the insured and the policy number, the abstain on disability medicals until the hubs signed one rule, and the brief schema that rejected a sentence with no page cite. One comparison was run on the thirty-five packs: free summary first, versus completeness check first and brief only if complete. The free summary cleared more packs and failed the escape CTQ. It was dropped. The checker sat in the intake queue the hubs already used.

Two weeks beside intake, with the lead as the measurement system. In the composite, packs released to assessment that later bounced for a missing document fell sharply on the claim types in scope, and every planted missing-document case returned a named gap. In the composite, one new escape appeared in week two: a medical in the pack for a dependent, not the insured, that the name match had passed on a shared surname. That row went back into the FMEA, the match rule was tightened to insured plus policy number, and disability claims stayed on abstain. The control plan named the intake lead as owner of the document standard, set a twice-weekly sample, and rolled back if an incomplete pack was labelled complete.

The piece taken out for reuse was the checker-then-brief pattern, not the summary prompt. The sponsor still got a brief on every complete pack. They did not get a summariser that could accept a claim by omitting what was missing.

What the two cases have in common

One sponsor asked for an email agent, the other for a summariser. In both, Measure found that seniors did not agree with each other, Analyze killed the named solution on the escape CTQ, and Verify refused to widen the gate when a new escape showed up. That is the case the method needs: not a model that scored well, a design that was not allowed to ship until the defect it was built for had a number.

Where classic DFSS breaks, and the patch

DFSS grew up on parts and processes with Gaussian variation. Agents fail differently: wrong state, ungrounded claim, a tool call that is syntactically valid and operationally false. Do not force a sigma conversion onto a 200-case labelled set and call it a capability study. Report the CTQ in its own unit, with a confidence interval, and keep the Six Sigma discipline, operational definitions, baselines, designed comparisons, a control plan, without the theatre of a sigma number the sample cannot support.

The other break is measurement latency. Waiting for a perfect baseline before any prototype is how DMADV earns a bad reputation in software. The patch is a thin tracer, not a skipped phase. A small labelled set and one concept built against it will teach you more in ten days than a quarter of interface simulation. The gate is still the gate: you do not widen autonomy until Verify clears the CTQs you wrote in Define. In the disbursement composite, that gate is what stopped Concept A from shipping on the strength of a demo.

What to take into the next engagement

Write the problem with no solution in it. Name the CTQs and the measurement system before you name the model. Score two concepts against those CTQs and against the failures you refuse to ship. Treat the eval harness and the abstain rule as design parameters, not as cleanup. Verify on the desk’s cases, then leave a control plan with an owner who is not you.

Appendix: the three sheets you leave behind

These are not the four Define artefacts. They are the sheets you leave behind. Blank fields are the template. The worked rows are the disbursement composite, so a reader can see a named solution die on the escape CTQ. Clear the worked figures before a live engagement, and do not describe them as a result.

Charter

FieldWhat belongs here
EngagementThe work, not the tool.
Business ownerThe person who can stop the engagement.
Process customerThe person who feels the defect, not the sponsor.
Problem statementNo product, vendor, or model in this sentence.
YThe output the design must move, in the customer’s unit.
In scope / out of scopeIncluding the non-goals that are design constraints.
Three CTQsOperational definition and pilot target on each. Accurate is not a CTQ.

Gate out of Define: a reviewer who was not in the room can state the problem without using the sponsor’s solution word.

Pugh matrix

Score +1, 0, or −1 against the current desk. Mark kill rows before scoring starts. A kill-row failure fails the concept regardless of the sum. Selected is the best sum among concepts with zero kill failures. Ties are not selected.

Gate out of Analyze: one selected concept, the kill rows written as design constraints, and no architecture diagram of vendors you might use.

Control plan

FieldWhat belongs here
Label-standard ownerA named person. Not a team.
Sample cadenceA rhythm the desk will actually keep.
Rollback triggerA signal, not a feeling. One escape on a kill CTQ is enough.
Who may change a parameterNo one, without a harness rerun signed by the label owner.
Owner after the engineer leavesThe desk for the standard. Engineering for the harness only.

Gate out of Verify: each CTQ reported against its target, with the sample size and the reviewer disagreement rate, and a named owner who is not the engineer.

Back