The outbox pattern with Service Bus

For mission-critical integrations between Dynamics 365 and external systems, naive direct calls have a problem: what happens when Dynamics 365 commits a transaction but the integration call fails? The change exists in Dynamics 365 but never reached the downstream system; data is now inconsistent. The outbox pattern solves this by combining a Dataverse "outbox" table with Azure Service Bus to guarantee that every committed change reaches downstream systems exactly once, eventually.

The problem with direct integration calls.

A typical pattern: a plug-in on Dataverse opportunity-close fires, calls an external API to push the sale to the back-office ERP, the API responds, the plug-in completes, the transaction commits.

What can go wrong:

• The API call succeeds but the Dataverse transaction rollbacks for unrelated reasons. The downstream system has the data; Dataverse doesn't. Inconsistency.
• The API call times out (5-minute plug-in limit). The plug-in fails; Dataverse rollbacks. From the user's perspective, the operation failed; but the downstream may have processed it.
• The API is temporarily unavailable. The plug-in fails; the user retries; but the rule is "do this only once".

These problems are why naive synchronous integration is unreliable for important data.

The outbox pattern.

The pattern decouples the recording of intent from the delivery of the message:

1. Within the Dataverse transaction: a plug-in (or flow) records a message into an outbox table in Dataverse. The outbox row holds the payload, target, and metadata. The Dataverse transaction commits atomically — either both the business change and the outbox row commit, or neither.

2. Out of band: a separate worker (Azure Function on a timer, a flow on schedule, a continuous-running service) reads new outbox rows, publishes them as Service Bus messages to the appropriate queue, then marks the outbox row as published.

3. Service Bus durably holds the message. The downstream consumer reads at its own pace.

4. Idempotent consumer: the downstream system processes each Service Bus message and is designed to be idempotent — receiving the same message twice produces the same result (no double-processing).

5. Confirmation: optional — the consumer acks back, and the outbox row marks complete.

Why this works.

• Atomic outbox commit — the business change and the message-to-send commit together. Either both happen or neither.
• Service Bus durability — once the message is on the queue, downstream can be offline for hours / days without losing it.
• Idempotency — the consumer can safely receive a message twice (e.g. if the publisher published, then crashed before marking the outbox row complete).
• Retry — failed deliveries retry from the queue; persistent failures land in the dead-letter queue for investigation.

Implementation in Dataverse.

• Outbox table — a custom Dataverse table with columns: payload (JSON), target, status (Pending / Published / Acked / Failed), created date, published date, retry count.
• Plug-ins / flows — insert rows into the outbox in the same transaction as the business change.
• Publisher worker — Azure Function on a timer, every 10–30 seconds, queries new outbox rows and publishes them.
• Service Bus — the durable channel.
• Downstream consumer — reads Service Bus messages, processes idempotently, optionally acks back.

The change tracking alternative. If Dataverse change tracking is enabled, the publisher worker can poll change tracking instead of an explicit outbox table. The pattern is similar but uses Dataverse's built-in delta mechanism rather than a custom outbox.

When to use the outbox pattern.

• Mission-critical integrations where data loss is unacceptable.
• High-volume integrations where Service Bus's durability and decoupling matter.
• Cross-system business processes that span Dataverse and other systems.

When not.

• Low-stakes integrations where occasional inconsistency is recoverable. Direct calls or basic webhooks suffice.
• Read-only integrations where the consumer pulls from Dataverse. The outbox is about push from Dataverse, not pull.
• Real-time UI-blocking calls — outbox is asynchronous; users don't wait for downstream success.

Trade-offs.

• Complexity — multiple components to design, build, monitor. The reliability benefit must justify the complexity.
• Latency — outbox-based integration has small delay (seconds to a minute typically). For most business processes that's fine; for some it isn't.
• Operational discipline — monitor outbox depth, dead-letter queue, retries. Without active monitoring, problems hide.

Common pitfalls.

• Outbox not in the same transaction — defeats the purpose. The outbox insert must be atomic with the business change.
• Consumer not idempotent — duplicates cause double-processing. Use message IDs or natural keys.
• No dead-letter handling — failed messages accumulate silently. Build alerting.
• Outbox not cleaned up — millions of completed rows degrade Dataverse performance. Archive or delete old rows.

Operational reality. The outbox pattern is canonical enterprise-integration practice; not specific to Dynamics 365. Adopting it for the integrations that matter is one of the highest-leverage reliability investments.