Events, Webhooks, Retries & Reconciliation
Transactional-outbox operations for integration owners: outbound events are persisted with transaction state before dispatch, webhooks are signed with replay protection, retries are bounded, and failures land in a dead-letter queue with owner-visible incidents (INT-02).
Event envelope — evt_01J8Z6QF
{
"eventId": "evt_01J8Z6QF",
"type": "invoice.posted",
"schemaVersion": "2.1",
"occurredAt": "2026-07-17T05:42:18Z",
"businessDate": "2026-07-17",
"currency": "AED",
"amount": "78,750.00",
"tenant": "al-manara-llc",
"scope": "dubai-hq",
"correlationId": "c0a8-17ju-2026-8802",
"causationId": "evt_01J8Z5RD",
"definitionVersion": "fin-posting v14 (prior v13)",
"contentHash": "sha256:5b02…91cf",
"producerTs": "2026-07-17T05:42:18.240Z"
}
Envelopes carry event/correlation IDs, tenant/scope, definition version and prior version, effective time and a content hash — and never secret or unnecessarily sensitive values (§14). Payload PII is referenced by canonical ID, not embedded.
Outbox — persisted before dispatch
Last 6 h Failed only| Event ID | Type · schema | Scope | Correlation | Published (UTC) | State |
|---|---|---|---|---|---|
evt_01J8Z6QF |
invoice.posted · v2.1 INV-2026-0879 · Amina Noor · AED 78,750.00 |
dubai-hq | …-8802 |
17 Jul 05:42 | Confirmed |
evt_01J8Z6M2 |
lead.created · v3.0 LEAD-10236 via Bayut connector |
dubai-hq | …-8795 |
17 Jul 05:31 | Confirmed |
evt_01J8Z5PK |
unit.reserved · v2.0 PR-A-1104 · Prototype Residences A |
dubai-hq | …-8781 |
17 Jul 05:08 | Confirmed |
evt_01J8Z4TN |
TemplateBindingActivated · v1.2 Sale contract EN/AR binding v9 |
tenant-wide | …-8776 |
17 Jul 04:55 | Confirmed |
evt_01J8Z3RW |
listing.status.changed · v2.3 LST-2041 → PUBLISHED (Bayut) |
dubai-hq | …-8760 |
17 Jul 04:12 | Retrying 3/6 · next 06:12 |
evt_01J8Z2H8 |
ProviderProfileHealthChanged · v1.0 Dubizzle connector → DEGRADED |
tenant-wide | …-8741 |
17 Jul 03:38 | Confirmed |
evt_01J8Z1CD |
ConfigurationDriftDetected · v1.0 Runtime differs from approved baseline · comms-policy |
tenant-wide | …-8733 |
17 Jul 03:02 | Confirmed |
evt_01J8YZKQ |
receipt.allocated · v2.0 RCPT-5521 → INV-2026-0871 · AED 25,000.00 |
abu-dhabi | …-8719 |
17 Jul 02:47 | Confirmed |
evt_01J8YWB3 |
deal.stage.changed · v3.1 DEAL-3302 → CONTRACT (Falcon Trading FZ-LLC) |
dubai-hq | …-8694 |
17 Jul 01:20 | Dead-letter |
evt_01J8YV9A |
payroll.policy.activated · v1.0 Commission plan FY2026-H2 (announce-only) |
tenant-wide | …-8688 |
17 Jul 00:58 | Confirmed |
Delivery attempts — evt_01J8YWB3 → Bayut status webhook
| # | Endpoint | Dispatched (UTC) | Result | Latency | Signature | Backoff / next |
|---|---|---|---|---|---|---|
| 1 | hooks.bayut.partner/almanara/status | 17 Jul 01:20:04 | 503 Service Unavailable | 30,004 ms | HMAC-SHA256 sent · nonce n-8841 | +1 m |
| 2 | hooks.bayut.partner/almanara/status | 17 Jul 01:21:06 | 503 Service Unavailable | 30,002 ms | HMAC-SHA256 sent · nonce n-8842 | +5 m |
| 3 | hooks.bayut.partner/almanara/status | 17 Jul 01:26:11 | 503 Service Unavailable | 30,007 ms | HMAC-SHA256 sent · nonce n-8843 | +15 m |
| 4 | hooks.bayut.partner/almanara/status | 17 Jul 01:41:15 | 503 Service Unavailable | 30,001 ms | HMAC-SHA256 sent · nonce n-8844 | +1 h |
| 5 | hooks.bayut.partner/almanara/status | 17 Jul 02:41:19 | 503 Service Unavailable | 30,003 ms | HMAC-SHA256 sent · nonce n-8845 | +4 h |
| 6 | hooks.bayut.partner/almanara/status | 17 Jul 06:41:22 | 503 Service Unavailable | 30,005 ms | HMAC-SHA256 sent · nonce n-8846 | Exhausted → DLQ + INC-2231 |
Every webhook is signed (HMAC-SHA256 over body + timestamp) with a unique nonce for replay protection; receivers must reject stale timestamps and reused nonces. Retries use bounded backoff (1m, 5m, 15m, 1h, 4h, 12h — max 6 attempts), then the event quarantines to the DLQ. All attempts are captured for audit.
Inbound webhook verification (last 6 h)
| Received (UTC) | Source | Signature check | Replay check | Disposition |
|---|---|---|---|---|
| 17 Jul 09:41 | Property Finder lead webhook | Valid | Fresh nonce | Accepted → lead connector |
| 17 Jul 09:12 | Payment gateway settlement callback | Valid | Fresh nonce | Accepted → banking adapter |
| 17 Jul 08:57 | Unknown origin (IP 203.0.113.44) | Invalid — WEBHOOK_SIG_INVALID | — | Rejected, audited, no state change |
| 17 Jul 08:20 | Bayut lead webhook | Valid | Nonce reuse — replay blocked | Rejected · original delivery already processed |
| 17 Jul 07:44 | E-signature provider event | Valid | Fresh nonce | Accepted → document adapter |
Dead-letter queue — 7 quarantined events
| Event ID | Type | Subscription | Failure reason | Attempts | Quarantined (UTC) | Incident | Status |
|---|---|---|---|---|---|---|---|
evt_01J8YWB3 | deal.stage.changed | Bayut status webhook | Endpoint 503 on all attempts | 6/6 | 17 Jul 06:41 | INC-2231 | Replay requested — awaiting approval |
evt_01J8XR20 | listing.status.changed | Bayut status webhook | Endpoint 503 on all attempts | 6/6 | 16 Jul 10:12 | INC-2231 | Replay requested — awaiting approval |
evt_01J8XQZL | listing.status.changed | Bayut status webhook | Endpoint 503 on all attempts | 6/6 | 16 Jul 09:58 | INC-2231 | Replay requested — awaiting approval |
evt_01J8XP44 | listing.status.changed | Bayut status webhook | Endpoint 503 on all attempts | 6/6 | 16 Jul 09:21 | INC-2231 | Replay requested — awaiting approval |
evt_01J8W1D7 | invoice.posted | Partner ledger feed (Prime Link) | Receiver rejected signature (their key rotated) | 6/6 | 15 Jul 22:10 | INC-2229 | On hold — receiver fixing key |
evt_01J8V8SM | lead.created | Marketing DW sink | Schema mismatch — consumer pinned v2, event v3 | 6/6 | 15 Jul 14:02 | INC-2226 | Consumer migration in progress |
evt_01J8V2K9 | receipt.allocated | Partner ledger feed (Prime Link) | Receiver rejected signature (their key rotated) | 6/6 | 15 Jul 05:33 | INC-2229 | On hold — receiver fixing key |
Consequence: events evt_01J8YWB3, evt_01J8XR20, evt_01J8XQZL, evt_01J8XP44 will be re-dispatched to hooks.bayut.partner/almanara/status in original order with new signatures and nonces. If the receiver is not idempotent, downstream duplicates are possible. Source records are unaffected.
Requested by: Elena Petrova (Property Admin), 17 Jul 2026 08:47. Replay is an authorized, audited action; incident INC-2231 closes automatically when all 4 deliveries confirm.
Reconciliation report — window 16 Jul 2026 00:00–24:00 UTC
Variance 3 — investigating| Event type | Produced | Dispatched | Delivered | Confirmed | DLQ | Variance | Status |
|---|---|---|---|---|---|---|---|
| lead.created | 412 | 412 | 411 | 411 | 1 | 0 | Reconciled |
| listing.status.changed | 1,268 | 1,268 | 1,265 | 1,265 | 3 | 0 | Reconciled |
| invoice.posted | 184 | 184 | 184 | 184 | 0 | 0 | Reconciled |
| receipt.allocated | 203 | 203 | 203 | 203 | 0 | 0 | Reconciled |
| deal.stage.changed | 96 | 93 | 93 | 93 | 0 | 3 | Variance — 3 events stuck in PENDING; outbox scan re-queued them 17 Jul 03:05 |
| unit.reserved | 57 | 57 | 57 | 57 | 0 | 0 | Reconciled |
| ProviderProfileHealthChanged | 14 | 14 | 14 | 14 | 0 | 0 | Reconciled |
| Total (window) | 17,911 | 17,908 | 17,904 | 17,904 | 4 | 3 |
Delivery success — last 7 days
| Day | Dispatched | Delivered | DLQ | Success rate |
|---|---|---|---|---|
| 11 Jul | 16,882 | 16,730 | 0 | 99.1% |
| 12 Jul | 15,904 | 15,809 | 0 | 99.4% |
| 13 Jul | 17,240 | 16,930 | 0 | 98.2% |
| 14 Jul | 18,011 | 17,831 | 0 | 99.0% |
| 15 Jul | 17,655 | 17,002 | 3 | 96.3% |
| 16 Jul | 17,908 | 17,030 | 4 | 95.1% |
| 17 Jul (to 10:00) | 7,904 | 7,714 | 0 | 97.6% |
Static render mockup — no live data. Source: ERP Modern Development Specification v2.0 · Covers: INT-02 (with §10.3 event standards and §14 outbox envelope rules).