Rick Low
Rick Low
Case study · UC-DR-01

Bank Doc-Risk GenAI Agent

For each named case folder, the agent flags document gaps and risk before a human or soft path writes the Case/GRC system of record.

Soft flag HITL SoR

Operating design · HITL · FMEA · circuit breakers · Asia/Shanghai

A reference architecture for the operating model. Not a live agent at a named bank. Targets, RPN scores, and breaker signals on this page are design estimates.

Glossary

SoR = Case/GRC system of record. HITL = human-in-the-loop. Soft flag = low-risk + complete pack may auto-write a non-clearing flag to SoR; it never clears a regulatory obligation. CP-* = named control point. CB-* = circuit breaker. FMEA = failure mode and effects analysis; RPN = S×O×D.

What it does

Flow: classify → extract → completeness → grounded insight → soft flag or HITL → SoR write.

In scope v1: named case folders, fixed doc-type taxonomy, soft auto for low+complete only. Out of scope v1: whole-estate crawl, auto-clear of regulatory obligation, public LLM endpoints, agent-held PII memory.

Platform path

01

BR-SIGN-OFF — freeze PACK-v1, taxonomy, Azure region, RACI.

02

Claude Code scaffolds LangGraph (redacted packs only).

03

PoC-GO on LangGraph — gold-set metrics; CB stubs trip in CI.

04

UAT-ACCEPT — Risk Ops on LangGraph HITL; optional Dify shell.

05

PROD-STEADY — LangGraph service; Claude Code off live folders; Dify off SoR.

PoC targets

Design estimates — not measured production KPIs. These are the PoC-GO gates, not a result.

Doc-type accuracy≥90%
Gap recall≥95%
Grounded cites, or the factor is dropped100%
Illegal auto-approves0

Production graph

Every live case runs on LangGraph. The model drafts the narrative and the citations. A deterministic scorer sets the tier. The soft path never clears a regulatory obligation.

LangGraph · Azure DI · private Azure OpenAI · LangfuseDesign
PlatformLangGraph
CP-INTAKEintake_folder
CP-CLSclassify_doc_type
CP-EXTextract_fields · Azure DI
CP-GAP · CP-HOOKcompleteness_check
CP-LLMrisk_summarize · Azure OpenAI
CP-SCOREscore_and_route

Soft · low + complete

auto_flag_soft. Non-clearing flag only. It never clears a regulatory obligation.

Else · CP-HITL

hitl_interrupt. Approve, edit, reject, or escalate. Timeout → quarantine, never approve.

CP-CASEwrite_case_packet
CP-EVALemit_langfuse_eval

Control points

Eleven named checks. The fail action is the contract. The checklist stays in the policy pack, not only in the prompt.

IDNameWhereCheckFail action
CP-INTAKEFolder allow-listintake_folderPath in approved case rootsReject the run
CP-CLSTaxonomy gateclassify_doc_typeType in the taxonomy, or unknownUnknown to HITL
CP-EXTField confidenceextract_fieldsConfidence ≥ 0.90Force HITL
CP-GAPCompletenesscompleteness_checkRequired docs and fields presentGap list to HITL
CP-HOOKRules outside the modelpolicy packChecklist is not in the promptBlock deploy if inlined
CP-LLMGrounded summaryrisk_summarizeJSON schema and citationsDrop uncited factors
CP-SCOREDeterministic tierscore_and_routePolicy table onlyUnit-test fail blocks deploy
CP-HITLHuman gatehitl_interruptApprove, edit, reject, or escalateTimeout to quarantine
CP-CASESoR writewrite_case_packetIdempotent case idRetry, then dead-letter queue
CP-NETPrivate egressplatformAllow-list endpoints onlyKill the run and open an incident
CP-EVALTrace and evalLangfusetrace_id on every caseAlert if missing

Failure modes

S / O / D are preliminary design scores pending PoC measurement — illustrative, not measured production KPIs. RPN = S×O×D. Six modes, highest RPN first. As of 25 Oct 2026.

IDFailure modeRPNControlsAction
FM-02OCR / extract field error180CP-EXT, CB-EXTRACTField confidence gate; cite the page
FM-01Wrong doc-type classification160CP-CLS, CB-CONFIDENCEUnknown type to HITL
FM-04LLM under-states risk150CP-SCORE, CP-HITL, CB-LLMDeterministic tier; never an LLM-alone tier
FM-05Missing-info rule bypassed81CP-GAP, CP-HOOKChecklist lives outside the model
FM-08PII egress to a public model endpoint80CP-NET, CB-DATA-EGRESSPrivate Azure OpenAI only; egress allow-list
FM-06HITL skipped, or timeout treated as approve40CP-HITL, CB-HITL-BACKLOGTimeout to quarantine, never approve

Circuit breakers

Six breakersIllustrative

Example signals only (ex: …) — illustrative design telemetry. Trip rules are the production contract. Soft auto-path is disabled whenever any breaker is open. Opening CB-HITL-BACKLOG or CB-CASE-WRITE never auto-approves. Opening CB-DATA-EGRESS kills the run.

CB-EXTRACT · Document Intelligence

Trip. Error rate >5% or p95 >30s over 5 min.

Degrade. Pause extract; queue files; page ops.

Example state. closed · ex: 1.2% err

CB-LLM · Azure OpenAI summarize

Trip. 5xx >3% or grounded-cite fail >10%.

Degrade. Stop the LLM node; HITL packet without narrative.

Example state. closed · ex: 0.4% 5xx

CB-CONFIDENCE · Extract confidence

half-open

Trip. Share of fields below 0.90 exceeds 25% of the pack.

Degrade. Force HITL; widen the eval sample.

Example state. half-open · ex: 18% low-conf

CB-HITL-BACKLOG · Analyst queue

Trip. Open HITL above SLA (50 cases or >4h p50).

Degrade. Throttle intake; alert the Risk Ops lead. Never approve.

Example state. closed · ex: 12 open

CB-CASE-WRITE · Case / GRC write

Trip. Write fail >2% or dead-letter queue >0 for 10 min.

Degrade. Freeze completions; retry the dead-letter queue. Never approve.

Example state. closed · ex: 0 DLQ

CB-DATA-EGRESS · Network egress

Trip. Any call outside the allow-list.

Degrade. Kill the run; open a security incident bridge.

Example state. closed · ex: allow-list OK

Gates

Four gates. Nothing skips one. Status as of 25 Oct 2026 (Asia/Shanghai): pending or not started. No gate is signed off.

BR-SIGN-OFF

Pending

Risk, Compliance, InfoSec, and Ops sign. PACK-v1 and the doc-type taxonomy freeze. BR-01 to BR-10, from the folder allow-list through a trace on every case.

Exit · before PoCSigned requirements. No build on an unfrozen pack.
PoC-GO

Not started

Gold-set targets met on redacted packs. No production PII. Breaker stubs trip in CI.

Exit · before UATPoC targets above. A chat demo is not this gate.
UAT-ACCEPT

Not started

Risk Ops signs UAT-01 to UAT-07: soft flag, missing field, name conflict, low confidence, HITL timeout, case API down, blocked egress. Dify is an optional HITL shell here, not the SoR.

Exit · before productionScripts signed. FMEA owners named. Breaker runbooks written.
PROD-STEADY

Not started

LangGraph service only for live cases. Claude Code stays off live folders. Dify stays off the system of record.

Exit · steady state30 days with no Sev-1. Breaker drills passed. Dual-run inside band.

Next build

Next build order: freeze PACK-v1 + taxonomy, scaffold LangGraph nodes with CP-* hooks, gold set, breaker stubs, then UAT with Risk Ops. Optional Dify only as HITL shell in UAT.

Related: Design the agent before you deploy it.

Rick Low