From scheduled polling to event-driven architectures: six patterns, when to use each, and the trade-offs we have hit in production.
What choosing wrong costs you
Picking the wrong integration pattern rarely breaks immediately. It surfaces six months later as "the data is always out of sync" or "rate limiting took the service down".
Here are six common patterns in order of increasing complexity, and when each one is the right call.
1. Scheduled polling
The simplest approach: a scheduled job periodically pulls a full or incremental dataset.
Use it when data volume is small, latency requirements are loose (hourly or daily), and the upstream API does not support incremental queries. Always pass an incremental timestamp — without it, growth will eventually crush both sides.
2. Webhooks
The upstream pushes events as they happen. Excellent latency, but it requires a publicly reachable, highly available receiving endpoint.
Three things are mandatory: signature verification, idempotency handling, and a retry queue for failures. Skip idempotency and a network blip that replays an event will double your data.
3. Queue in the middle
Land incoming events in a queue, then process them asynchronously with consumers. Good for absorbing spikes and guaranteeing no message loss.
Use it when traffic is bursty, downstream processing is slow, or you need to replay historical events. The cost is higher operational complexity.
4. Change data capture (CDC)
Listen to change events at the database layer instead of relying on the application to emit them.
Use it when you must sync a legacy system, the vendor offers no API, or you must guarantee no change is ever missed. It is the least invasive to the source system but the most demanding operationally.
5. Aggregation layer / BFF
Put a layer over multiple upstream systems that exposes one unified interface and model downstream.
Use it when you integrate three or more systems, or when upstreams are likely to be swapped. The core value is isolating change: upstream changes, downstream does not.
6. Event-driven architecture
All systems communicate over an event bus rather than calling each other directly.
The most flexible long term and the most expensive to start. It is worth it only with many systems, independently deploying teams, and a genuine business need. Do not adopt it early for architectural purity.
How we choose
Start with the simplest option and escalate only when it demonstrably fails the requirement. If polling works, do not build webhooks; if webhooks work, do not add a queue. Each step up roughly doubles the operational burden.
One universal recommendation
Whichever pattern you choose, keep a sync log in your system: what ran, when, what it synced, whether it succeeded, and why it failed. When someone asks why a record did not sync, that table saves you two hours of digging.