WF-13 — Integration Delivery

Process map for reliable asynchronous delivery of business events: transactional outbox, transform, signed dispatch, acknowledgment and reconciliation. Owner: Integration Owner — Platform Technology (on-call: Vikram Nair) · Version 5 (active since 28 Jun 2026).

WF-13 INT-02

Open Events & Webhooks

Events (24h)

12,480

Across 18 event types · 12 endpoints

Delivery success

99.6%

First-attempt 97.1% · after retries 99.6%

Median latency

1.8s

Commit → acknowledged

DLQ depth

▼ 1 poison · 6 outage-related

Endpoints healthy

11 / 12

Prime Link webhook degraded since 06:15

Process map — happy path Trigger: business event committed Per §8.2 Table 14

Business event committed (e.g. deal won)→ Outbox persist — same transaction as the write→ Transform — canonical payload → channel mapping vN→ Sign (HMAC) · idempotency key · replay guard→ Send webhook — delivery attempt captured→ Acknowledge — 2xx + receipt id→ Reconciled — nightly delivery report clean

Consumers deduplicate on the idempotency key: a retried create with the same key returns the original result without duplication (INT-01/INT-02 acceptance criteria). Every attempt, signature key id and response is retained as evidence.

Exception paths3 modelled exceptions

Outbox row dispatched→ Transform fails schema validation→ No retry — poison message (deterministic failure)→ DLQ (poison) — fix mapping version, then authorized replay

Send webhook→ 5xx / timeout from endpoint→ Bounded retries — backoff 1m · 5m · 15m · 1h · 3h · 6h→ Retries exhausted→ DLQ + owner-visible incident created (INT-02)

Nightly reconciliation 02:00→ Sent ≠ acknowledged counts for an endpoint→ Investigate delivery attempts + receipts→ Reconciliation exception — manual match or authorized replay · report annotated

Owner & definition (§8.3)Published v5

Workflow ID: WF-13
Owner: VNIntegration Owner — Platform Technology · on-call Vikram Nair
Version: 5 · effective 28 Jun 2026 · replay-safety simulation passed
Trigger: Any committed business event with a registered subscription (deal, party, invoice, listing…)
Preconditions: Subscription active · channel mapping version published · signing key valid · endpoint registered with secret
States: Committed → Outboxed → Transformed → Signed → Sent → Acknowledged → Reconciled
Terminal outcomes: Reconciled DLQ (poison) DLQ (retries exhausted) Reconciliation exception
Required evidence: Outbox row + transaction id · mapping version · signature key id · every delivery attempt with response · receipt id · reconciliation report
Policy snapshot: Replay requires authorization with reason; replayed events keep the original idempotency key

Controls (Table 14, expanded)INT-02

Control	Enforcement	Status
Transactional outbox	Event persisted with transaction state before dispatch — no event without commit, no commit without event	Enforced
Idempotency	Deterministic key per event (e.g. `deal-DL-2026-0142-won`); consumer duplicate-suppression verified in contract tests	Enforced
Signed webhooks	HMAC-SHA256, key id in header, rotation quarterly (current `kid wh-2026-q3`); replay protection via timestamp + nonce	Enforced
Bounded retry	6 attempts, exponential backoff 1m → 6h; deterministic failures skip retry	Enforced
DLQ + incident	Exhausted / poison events land in DLQ; owner-visible incident auto-created	7 in DLQ
Replay + reconciliation	Authorized replay with reason; nightly reconciliation report sent vs acknowledged per endpoint	Enforced

Exception paths & recovery

Exception	Detection	Recovery	Authority
Schema error	Transform validation failure (deterministic)	Straight to DLQ (no retry); publish fixed mapping version; authorized replay preserves idempotency key	Integration owner
Endpoint outage	5xx / timeout; retries exhaust after ~10h	DLQ + incident; provider status on Provider Health; bulk replay after recovery	Integration owner + provider contact
Manual reconciliation	Nightly count mismatch per endpoint	Attempt-level investigation; manual match or replay; report annotated with resolution + reason	Integration owner · reviewed by Grace Okafor (audit)
Duplicate receipt	Same idempotency key acknowledged twice	Consumer returns original result; flagged for contract-test review; no data duplication	System

Timers & escalation (§8.3)Calendar: 24/7 platform

Timer	Calendar	Warning	Breach	Escalation
Outbox dispatch lag	24/7 · poll 5s	1m	5m	Platform on-call page
Retry window	24/7 · backoff 1m → 6h	attempt 4	attempt 6 → DLQ	Incident to integration owner
Incident acknowledgment	Business hours (Mon–Fri)	15m	30m	On-call → Technology owner
DLQ item age	Business days	1d	3d	Weekly governance review · aging report
Reconciliation run	Nightly 02:00 GST	+30m	+2h	Platform on-call · report to RPT-24

Screens involved

Events, Webhooks & DLQ — primary console: outbox, deliveries, retries, replay
Provider Health — endpoint status, incidents and SLA
API Access & Credentials — signing keys, scopes, rate limits
RPT-24 Integration & Queue Health — reconciliation and DLQ trend reporting
Audit Explorer — replay authorizations with reasons

Dead-letter queue — current itemsOpen DLQ

Event	Endpoint	Cause	Age	Action
`lead.created` · dlq-3312	Bayut connector	Schema error — mapping v9	3h	Replayed 09:30 after v10 fix
`listing.delisted` · dlq-3313…18	Prime Link webhook	Outage — 503 since 06:15	≤ 11h	Awaiting endpoint recovery · incident INC-2026-088

Worked instance — deal.won for DL-2026-0142 · Amina Noor · unit PR-A-1103 Reconciled · 1 subscriber retried

16 Jul 2026 · 17:42:01
Deal marked Won in Deal Workspace
Event evt_01J2XQ8V… + outbox row obx-88213 committed in the same database transaction · payload includes deal id, unit, amount AED 1,850,000.00 (masked party contact data per policy)
17:42:03
Transform — mapping v12 (finance ERP payload)
Canonical event → channel schema validated · mapping version recorded on the delivery record
17:42:04
Signed & guarded
HMAC-SHA256 kid wh-2026-q3 · idempotency key deal-DL-2026-0142-won · timestamp + nonce for replay protection
17:42:05
Subscriber 1 acknowledged — Finance connector
POST → 200 · receipt rcp_5591 · commission voucher queued in Commissions
17:42:05
Subscriber 2 failed — Portal syndication delist (PR-A-1103)
503 Service Unavailable · retry schedule started (1m · 5m · 15m …) · every attempt captured
18:12:09
Subscriber 2 retry 3 succeeded
200 · receipt rcp_5604 · listing delist confirmed on Portal Syndication · same idempotency key, no duplicate delist
17 Jul · 02:00
Nightly reconciliation — clean for this event
Report REC-2026-0717: 12,480 sent = 12,480 acknowledged across healthy endpoints · Prime Link mismatch tracked separately under INC-2026-088

Idempotency in action When retry 3 delivered the delist event, the portal connector had partially processed attempt 1 before the 503. Because both attempts carried deal-DL-2026-0142-won, the consumer returned the original result — one delist, no duplication (INT-02 acceptance criterion).

Static render mockup — no live data. Source: ERP Modern Development Specification v2.0 · Covers: WF-13, INT-02.