Cross-Region_Database_Replication Design

Below is the Design content created with the help of AI & feel it worthy to study so sharing here.

1. Executive Summary

This document specifies the architecture and a working reference implementation for cross-region, active-active database replication — a deployment where every region accepts local reads and writes against its own database, and changes are propagated to every other region in near real time. The goal is to give users in North America, Europe, or Asia the same fast, local read/write experience while keeping every region's data converging toward a single, consistent global state.
Cross-region replication only works if every node in every region can also act as a write master — in other words, a multi-master replication layer is the engine underneath a cross-region deployment. Part I of this document covers that engine: how multi-master replication works, the trade-offs between native database engines and an external CDC pipeline, and the building blocks (conflict resolution, transaction handling, loop prevention) that any cross-region system depends on. Part II is the primary deliverable: the cross-region architecture itself, a complete reference implementation built and tested across two simulated regions, and the production hardening — network tuning, security, and an incident walkthrough — required to run it for real.
Key Recommendations

• Use asynchronous, CDC-based replication for any cross-region deployment. Synchronous consensus across WAN links is not viable once round-trip latency exceeds a few milliseconds — and inter-region links typically run 100–250 ms.
• Build the replication backbone on Debezium + Apache Kafka, with one local Kafka cluster per region bridged by MirrorMaker 2 (or Confluent Cluster Linking), so a cross-region network outage degrades to buffered lag rather than blocked writes.
• Adopt a Data-Home routing pattern so that any given row is normally written from a single home region, reducing cross-region write conflicts to near zero while preserving full-cluster disaster recovery.
• Design schemas defensively (UUID primary keys for any row reachable from more than one write master — integer keys remain preferable for single-homed tables, see Appendix G.1, soft deletes, an explicit conflict-resolution timestamp column) so the cross-region consumer is rarely forced to make a judgment call.
• Treat circular-replication prevention as a first-class design concern, not an afterthought — it is the single most common production incident in cross-region active-active topologies, and Section 14 walks through diagnosing and fixing one end-to-end.

 
Part I
Foundational Concepts

The multi-master replication mechanics that any cross-region deployment is built on top of. Readers already comfortable with CDC, conflict resolution, and replication consistency models can skip ahead to Part II.

2. Architectural Overview
​
Unlike a standard primary-replica setup where only one node handles writes, a multi-master topology treats every database instance as a peer capable of accepting INSERT, UPDATE, and DELETE traffic directly from the application — the property a cross-region deployment depends on, since each region's database must be writable locally. Building or operating such a system means designing a replication layer that takes on four core responsibilities, regardless of which underlying technology implements it.
​
2.1 What the Replication Layer Must Do

2.2 Three Architectural Paths
​When implementing multi-master replication, there are three broad architectural patterns to choose from. Figure 1 summarises them; the comparison table and the rest of this document focus primarily on the external, log-based approach, which is the recommended default for new builds.

2.3 Recommendation
​For a multi-master relational setup confined to a single region with a homogeneous database engine, native or highly specialized engine-level replication is generally superior to a custom-built replication service — relational databases enforce strict guarantees (foreign keys, unique constraints, cascading deletes) that are difficult to replicate perfectly in an external application.
However, the moment requirements expand to cross-region deployment — which is the subject of this document — an external CDC pipeline becomes the architecture of choice. It tolerates the multi-hour network partitions that inter-region links occasionally suffer, it decouples replication transport from the database engine so each region can be tuned independently, and it is the path detailed for the remainder of this document, beginning with the consistency-model trade-off in Section 3 and culminating in the full cross-region reference implementation in Part II.

3. Replication Consistency Models
​
Every multi-master design ultimately chooses between two consistency models. The choice has a massive effect on application performance and on how much conflict-handling logic the rest of the system needs to carry.

3.1 Synchronous (Eager) vs. Asynchronous (Lazy) Replication
In the synchronous model, a transaction is only committed back to the application after it has been successfully written to all nodes in the cluster. In the asynchronous model, a node commits locally and responds immediately; the replication layer propagates the change to other nodes in the background.

4. Native Engine Comparison: MySQL Group Replication vs. PostgreSQL BDR

MySQL Group Replication (MGR) and PostgreSQL BDR (Bi-Directional Replication, part of EDB Postgres Distributed) both deliver multi-master writes, but they sit on opposite ends of the distributed-systems spectrum: pessimistic consensus versus optimistic asynchrony.
​
4.1 Architectural Comparison

The trade-off, simplified:

• Choose MySQL Group Replication if the application cannot tolerate any data divergence and you are willing to accept higher write latency in exchange for the database cleanly rejecting conflicting writes before they happen.
• Choose Postgres BDR if the application needs to write fast across distant geographical regions and you are prepared to resolve conflicts after they occur.

​
4.2 How Each Engine Handles Primary-Key Conflicts
​​
A primary-key conflict occurs when two nodes concurrently attempt to INSERT a row with the same key. Because the two engines handle time and network agreement completely differently, their strategies diverge entirely.

MySQL Group Replication: the “Certification” Gatekeeper​
MySQL forces every transaction through a global Paxos ordering step before allowing it to commit, a process called Certification. When a node writes a row, MySQL extracts a hash of the primary-key constraint (the “write set”) and broadcasts it to the group, which assigns a global, monotonic sequence number. Every node then checks its own deterministic transaction log: whichever transaction was ordered first is certified and commits; the losing node's transaction aborts and rolls back, surfacing a standard serialization/deadlock-style error to the application, which simply retries.

Postgres BDR: the Post-Facto Repair Engine​
BDR operates optimistically: it assumes conflicts are rare, lets every node commit immediately, and cleans up afterward during background replication. When BDR's logical replication worker tries to apply Node A's insert onto Node B and finds Node B already has a row with that key, it hits a native insert_exists constraint error and falls back to a predefined conflict-resolution policy:

• update_if_newer (default, Last-Write-Wins): BDR compares exact commit timestamps (track_commit_timestamp); the later write wins globally and the earlier row is silently overwritten.
• CRDTs (Conflict-free Replicated Data Types): if configured, compatible field types merge or accumulate values automatically without discarding data.
• Logging & dead-lettering: every conflict event is written to bdr.conflict_history for audit and review.

4.3 Best Practices to Avoid Primary-Key Conflicts Altogether
​​
Regardless of which engine is chosen, relying heavily on the engine to sort out key collisions leads to high rollback rates (MySQL) or lost updates (Postgres). It is far better to design the schema so collisions rarely occur:

• UUIDs (v4 or v7) as primary keys: the entropy space makes random cross-node collisions effectively impossible.
• Interleaved sequences: if integer IDs are required, configure auto-increment offsets per node — e.g. in a 3-node cluster, Node 1 generates 1, 4, 7, 10…; Node 2 generates 2, 5, 8, 11…; Node 3 generates 3, 6, 9, 12…
• BDR Global Sequences: Postgres BDR can allocate chunks of IDs to individual nodes through a thread-safe distributed mechanism, removing insert-collision risk completely.

5. External Log-Based Replication (CDC) Architecture​

Native engines like MGR or Postgres BDR are powerful, but they tightly couple replication logic directly inside the database kernel. An External Log-Based Replication Engine — Change Data Capture (CDC) combined with a distributed stream such as Apache Kafka — is considered the gold standard for enterprise infrastructure, for four concrete reasons.

5.1 Why CDC + Kafka Is the Enterprise Standard
​
1. Zero Impact on Database Write Performance
Native multi-master solutions force the database engine to do replication-related work on every write: MySQL runs a global Paxos round; Postgres BDR maintains a persistent logical-replication mesh to every peer. An external pipeline reads the WAL or binlog asynchronously and out-of-band, so the database spends 100% of its CPU cycles on queries, not on cluster coordination.

2. Dynamic Buffering and Outage Resilience
In a native cluster, losing a node or a network partition can force the remaining nodes to block writes entirely once quorum is lost. An external engine leverages a durable message broker (Kafka or Redpanda) as a shock absorber: if a node is offline for hours, events simply accumulate safely in Kafka, and the consumer resumes exactly where it left off once the node returns — without overwhelming the target.

3. Homogeneous vs. Heterogeneous Freedom
Native replication forces a monoculture — every server must run the same engine, often the same version. External replication treats data as abstract, structured events (JSON or Avro), so a single CDC stream can feed a Postgres node, a MySQL node, an analytics warehouse, a search index, and a cache simultaneously.

4. Sophisticated, Multi-Layered Conflict Resolution
​Native engines bind you to their built-in rules (MySQL's rollback, BDR's timestamp comparison). With an external pipeline, the replication consumer is just a microservice — it can implement custom business logic: route conflicting rows to a dead-letter queue for human review, or query external business state to merge two writes intelligently.

5.2 Debezium Pipeline Setup
​
Debezium runs inside the Kafka Connect framework and continuously tails the database's write-ahead log (WAL for Postgres, binlog for MySQL), streaming every row-level change as a structured message into a Kafka topic.
Step 1 — Prepare the Database Logs

• PostgreSQL: set wal_level = logical so logical decoding plugins can extract changes.
• MySQL: enable the binary log with binlog_format = ROW so individual row modifications are recorded instead of raw SQL statements.

Step 2 — Configure the Connector​
Connectors are configured with JSON, submitted via REST to the running Kafka Connect cluster. Example for PostgreSQL:

Once submitted, Debezium takes an initial snapshot of existing data, then streams new changes into a topic named [topic.prefix].[schema].[table] — e.g. cdc_events.public.customers.

5.3 Anatomy of a CDC Event Payload
​​
A Debezium event is explicitly split into a schema block (data types and constraints) and a payload block (the actual database state). Below is an annotated UPDATE event for a customer record.

5.4 Production Schema Optimization
​
By default, Debezium attaches a full JSON schema block to every message. At production scale this is wasteful: the example above carries roughly 2.5 KB per message, of which only about 8% is the actual data — the rest is repeated schema definition. At thousands of transactions per second, this inflates storage, network bandwidth (especially painful for cross-region mirroring), and consumer-side parsing cost.

The standard fix is to externalize the schema to a Confluent Schema Registry and switch the Kafka Connect converters to Avro or Protobuf, which strip the embedded schema and replace it with a 4-byte schema-registry reference. For a high-volume production CDC pipeline, Avro with a Schema Registry is the typical default, balancing compactness with safe schema evolution.


6. Consumer & Conflict-Resolution Architecture

A custom Kafka consumer cannot simply execute blind UPDATE or INSERT statements against the target databases. Because it reads from a unified topic containing events from every master, it has a global view of the cluster's timeline and must act as an idempotent, conflict-aware state replayer.

6.1 Pipeline Design​
A single-threaded consumer creates replication lag quickly. Production designs use a dispatched-worker pattern: events are partitioned by the row's primary key (so all changes to a given row always reach the same worker, eliminating intra-row races), fanned out to parallel workers, and funnelled through a single Conflict Resolution Engine before being applied or routed to a dead-letter queue.

6.2 Last-Write-Wins Resolution Logic
​​
To avoid race conditions at the database level, each worker wraps its check-and-write in a short transaction using an Optimistic Concurrency Control (OCC) pattern —
a conditional UPDATE rather than a plain one:

• Rows affected = 1: the write succeeded — the incoming data was newer than what was on disk.
• Rows affected = 0: a conflict occurred. The target already holds a newer value (from a local write or another node's sync). The consumer safely discards the change.

​
6.3 Critical Edge Cases

A. The “Delete vs. Update” Dilemma
If Node A deletes a row at 10:00 and Node B updates that same row at 10:01, a late-arriving delete event could wipe out the newer update. The fix is to implement soft deletes (is_deleted = true / deleted_at = timestamp) and treat deletions exactly like updates, preserving the row's timestamp so the conflict engine can correctly favour the 10:01 update.

B. Network Time Skew
Wall-clock timestamps are notoriously unreliable across distributed servers; a 2-second clock drift can make one node's writes always win Last-Write-Wins regardless of true physical order. Mitigate this by forcing all hosts to sync via NTP to sub-millisecond drift, or by replacing timestamps with vector clocks / logical counters that order events by causality rather than wall-clock time.
C. The Dead-Letter Queue for High-Value Data​
For financial or otherwise mission-critical records, silent overwrites or drops are unacceptable. When Rows Affected = 0 signals a conflict that business logic cannot confidently resolve, the consumer serializes the incoming payload alongside a snapshot of current database state and writes it to a dedicated conflict-DLQ topic (e.g. customers-conflict-dlq) for an operator to review and patch manually.

7. Scaling Multi-Table Transactions

Partitioning Kafka topics by a row's primary key (e.g. customer_id) gives massive parallel scaling for single-row updates. The moment an application executes a cross-row or multi-table transaction, however, that transaction is broken apart by Debezium into independent row-level events. Because different keys land on different partitions, those events are processed by different consumer threads entirely out of sync — breaking the Atomicity and Isolation guarantees of the original transaction on replica nodes.

7.1 The Core Solution: Transaction-Aware Partitioning
Instead of row-level independence, every row change that belonged to the same original transaction must be routed to the same Kafka partition and processed by the same consumer thread. This is achieved with Debezium's Transaction Metadata feature (provide.transaction.metadata = true), which emits a transaction block inside every payload:

"transaction": { "id": "99", "total_order": 1, "data_collection_order": 1 }

A routing component then partitions on hash(transaction.id) rather than the row's primary key, so every event tagged id: 99 lands in the same Kafka partition and is consumed by the same thread — preserving causal order across tables.

7.2 How the Consumer Replays the Transaction
​​
A dedicated consumer thread buffers events in memory, keyed by transaction.id, until it sees Debezium's explicit BEGIN and END boundary markers for that transaction. On END, it opens a single native transaction on the destination database and applies every buffered row change back-to-back:

If any row-level update fails a sanity check or constraint, the consumer can roll back the entire batch on the target, preserving cross-table atomicity.

8. Preventing Circular Replication​

In an active-active setup, avoiding circular replication — also called the “infinite loop” or “ping-pong” effect — is essential. Without a circuit breaker, a write on Node A streams to Node B; Node B applies it, generating a new local log entry; Node A sees that entry, copies it back, and the transaction bounces between servers indefinitely, consuming CPU and disk I/O without bound. This is, in practice, the most common production incident in active-active topologies.

8.1 Pattern 1 — Origin Server Tagging (Native Engines)
Native engines such as MySQL Group Replication and PostgreSQL BDR inject a hidden metadata tag specifying the server/node ID where a write originated. When a node receives a transaction from the replication stream, it compares the origin tag to its own ID; on a match, it silently drops the event and suppresses outbound log generation for it, terminating the loop at the source.
8.2 Pattern 2 — Client / Application Identification (Custom CDC + Kafka)​
When building an external replication layer, the database sees the Kafka consumer as an ordinary application connection — so without explicit handling, the consumer's own writes get picked back up by Debezium and the loop begins. The fix:

1. Name the consumer connection: pass a distinct application name in the connection string (e.g. ApplicationName=cdc_replicator for Postgres, or a session variable for MySQL).
2. Configure Debezium filtering: PostgreSQL connectors can filter by application name or replication origin; MySQL connectors can use binlog_ignore_db or filter on system user tags / channel names.

Because Debezium ignores log entries stamped with the replicator's signature, the loop is broken the instant the data lands on the target node.
8.3 Pattern 3 — Explicit Origin Columns (Schema-Level Fallback)​
If connection-level filters aren't available, add an operational tracking column to every replicated table:

Code Editor

​When the application writes on Node A, it stamps origin_cluster_id = 'CLUSTER_A'. The consumer replicating to Node B then issues a conditional upsert:

Part II
Cross-Region Architecture & Reference Implementation

The primary subject of this section: how to take the multi-master building blocks from Part I and assemble them into a production-grade, active-active replication system spanning multiple geographic regions.

9. Multi-Region Active-Active Architecture

Stretching an active-active layout across geographically separated regions (us-east, eu-west, ap-south, for example) introduces an unavoidable constraint of physics: cross-region fiber routes typically add 100–250 ms of round-trip latency. How that latency is managed determines whether the system remains usable.
​
9.1 Latency Impact on the Two Replication Models
Model A — Native Synchronous (e.g. MySQL Group Replication)
Spanning a single virtually-synchronous cluster across distant regions causes performance to collapse. A write must achieve Paxos consensus across regions before committing, so the node must wait for packets to cross oceans and return. Because relational databases hold row locks during this window, throughput drops sharply and connection pools starve.
​
Model B — Asynchronous (e.g. Postgres BDR or external Kafka CDC)
Local application servers talk only to their local regional database, which commits in 1–5 ms; a background process (Debezium streaming to a cross-region Kafka cluster) copies the data to other regions asynchronously. The trade-off is eventual consistency: a user who edits their profile in New York and immediately opens the app in London might see stale data for a few hundred milliseconds.

                            Figure 9. Illustrative write-latency impact by replication model and topology (approximate, for relative comparison only).

​9.2 Production Pattern: Data-Home Routing​
Because asynchronous multi-region replication means conflicts will happen across regions, enterprise architectures rarely allow unconstrained writes to any row from any region. Instead, application traffic is partitioned so that specific data subsets are “owned” by a home region — for example, us-east owns North American accounts and eu-west owns European accounts.

Why this is the gold standard for multi-region deployments:

• Local latency: users interact only with their home region's infrastructure, so writes stay fast.
• Near-zero cross-region conflicts: because European users never write to North American profiles, the probability of a row-level conflict drops close to zero.
• Disaster-recovery resilience: every region still holds a full, real-time read-only copy of the entire global dataset, so if us-east goes dark, eu-west can be promoted to primary writer for US accounts instantly.

9.3 Handling Global Shared Tables (the Exception)​
Some tables cannot be partitioned by region — a global product catalog or a currency-rates table, for example. These are handled with one of two methods:

• Method A — Single-Origin, Multi-Region Read: one region (e.g. corporate headquarters) is the source of truth for changes; all others treat the table as read-only, relying on async replication to refresh local caches.
• Method B — CRDTs: for tables that genuinely need concurrent writes from multiple regions (e.g. a global inventory counter), the engine replicates the delta operation (ADD -1) instead of the absolute value, so the database mathematically merges updates regardless of arrival order. This only works for values with no hard floor or ceiling to protect — a CRDT counter cannot itself stop a merged total from going negative, which is why account balances and other invariant-bound values need the account-homing / pre-allocation pattern in Appendix G.5 and G.6 instead.

10. Reference Implementation: A Local Two-Region Test Environment​

Before standing up independent Kafka clusters per region (Section 11) and the full network/security hardening that goes with them (Sections 12–13), it is worth proving the core replication logic end-to-end in a minimal, self-contained environment. This section builds exactly that with Docker Compose: two simulated regions, each with its own PostgreSQL instance and Debezium connector, bridged by a single shared Kafka broker and one Python consumer.

10.1 Directory Structure

10.2 Infrastructure (docker-compose.yml)​
Region A listens on host port 5431 and Region B on 5432; both Debezium workers register against a single Kafka broker shared between them. The two regions stay logically isolated by topic name (region_a.public.users vs. region_b.public.users), even though they share the same physical broker process.

10.3 Database Schema (init-db.sql)​
Both databases share an identical schema, including a last_replicated_ts metadata column used for conflict resolution:

10.4 Connector Registration (register-connectors.sh)​
Each region's Debezium worker is registered independently via a REST call to its own Kafka Connect port (8083 for Region A, 8084 for Region B):

10.5 The Replication & Conflict Consumer (consumer.py)​
This single Python process is the cross-region bridge for the test environment: it subscribes to both regions' topics, and for every event applies Last-Write-Wins logic to the region that did not originate the change. Install dependencies with pip install kafka-python psycopg2-binary.

10.6 Running and Testing the System

1. Start the stack: docker-compose up -d, then wait ~15–20 seconds for both Kafka Connect workers to initialize.
2. Register the connectors: ./register-connectors.sh
3. Start the bridge: python consumer.py
4. Verify basic sync: update a row in Region A's database and confirm the consumer log shows it being applied to Region B within a second or two.
5. Verify conflict resolution: stop the consumer (Ctrl+C) to simulate a dropped trans-Atlantic link, update the same row in Region A, wait two seconds, then update it again in Region B. Restart the consumer.

​VALIDATION RESULT
Expected result: because Region B's write happened later in time, the consumer applies Region B's value to Region A, but raises a Conflict Alert when it tries to apply Region A's now-outdated value to Region B — demonstrating Last-Write-Wins protecting the cluster from a dirty cross-region overwrite.

​REVIEWER NOTE
Read this result alongside Section 6.3.C, not in isolation: this document already recommends routing high-value/financial conflicts to a Dead-Letter Queue rather than letting wall-clock Last-Write-Wins decide silently. Treat this validation step as confirming the circuit breaker works, not as evidence that LWW is the right outcome for every conflicting write. See Appendix G.4.

11. Scaling to True Multi-Region: Independent Kafka Clusters per Region

The single shared Kafka broker from Section 10 proved the replication logic, but it is a single point of failure that does not reflect a real multi-region deployment — a genuine cross-region system places an independent Kafka cluster inside each region, so that a regional outage or network partition never takes down another region's ability to read or write locally. This section evolves the design accordingly, built around logical decoding, Debezium, and a per-region Apache Kafka deployment bridged across the WAN. The guiding principle is unchanged: network hiccups between regions must never block a local database write.
​
11.1 The Cross-Region Data Flow​
The architecture follows a “Mirror Kafka” pattern. Rather than having Debezium in Region 1 write directly to a database in Region 2 over the ocean, each region streams to its own local Kafka cluster first; a mirroring tool then transfers data between Kafka clusters, and each side consumes locally.

11.2 Critical PostgreSQL Configuration​
The following parameters must be applied to postgresql.conf on every regional database server:

11.3 Cyclic-Prevention Using PostgreSQL Replication Origins​
Because both regions stream to each other simultaneously, the consumer must tag its own writes so they are not picked back up by the local Debezium connector — the schema-agnostic version of Pattern 1 from Section 8, implemented natively in PostgreSQL.

The Debezium connector is then configured to skip transactions tagged with a replication origin, emitting only events generated by real application users:

​OPERATIONAL TIP
Production best practice: do not rely on a manual terminal command to create the replication origin. Have the consumer check for and create the origin programmatically on startup (SELECT 1 FROM pg_replication_origin WHERE roname = ...; create if missing) so the environment can be torn down and rebuilt at any time without manual steps.

12. Cross-Region Backbone Engineering

In production, the cross-region backbone is the most vulnerable point of a global active-active architecture — it absorbs erratic latency, packet loss, fiber cuts, and throughput bursts. Reliable low-latency replication depends on specific topology choices and network tuning.
​
12.1 Physical Placement: “Remote Consume, Local Produce”
Standard practice is to host the MirrorMaker 2 (or equivalent) compute cluster physically inside the destination region. Kafka producers are far more sensitive to network latency than consumers: a producer writing over a high-latency WAN link with acks=all frequently times out waiting for acknowledgment. By placing MirrorMaker 2 inside the destination region, it performs a remote read over the WAN (which Kafka's consumer protocol tolerates well via long-polling) and a local write over single-digit-millisecond data-center switches.

12.2 OS and TCP-Level Tuning
Default Linux and Java network settings throttle cross-region throughput because of the bandwidth-delay product. Without wider TCP windows, the sender spends most of its time idle waiting for acknowledgments across the ocean. Production deployments widen the kernel buffers on both ends of the link:

Larger buffers let TCP keep more data in flight without waiting for acknowledgments — essential for high-latency cross-region links — while auto-tuning ensures low-traffic connections don't waste memory. Because cross-region traffic transits public or semi-public networks, it must also be encrypted via mTLS; in production this encryption is offloaded to dedicated network proxies (an Envoy mesh, or a cloud provider's private interconnect) rather than run on the core Kafka broker I/O threads.

12.3 MirrorMaker 2 Production Tuning​
The default MirrorMaker properties must be customized for throughput, batching, and resilience:

12.4 Modern Alternative: Confluent Cluster Linking​
MirrorMaker 2 requires deploying and scaling an intermediate Kafka Connect cluster: it consumes, deserializes, re-produces, and re-serializes every message. Confluent Cluster Linking eliminates that middle layer, allowing brokers in the destination region to connect directly to brokers in the source region over native intra-broker channels.

13. Security: Access Control for Cross-Region CDC Topics
By default, an unconfigured Kafka cluster allows any client to read or write any topic. In a production multi-region setup, access is restricted using the principle of least privilege: every component is uniquely identified via mutual TLS or SASL/SCRAM, and granular ACL rules constrain what each identity may do.
​
​13.1 The Perimeter Matrix

13.2 Implementing Kafka ACLs
​
Apache Kafka's native kafka-acls utility enforces these boundaries. Assuming components authenticate as unique principals:

13.3 Advanced Guardrails: Strict Separation Patterns​

• Pattern A — Dedicated replication clusters (recommended): run CDC topics on a lean, isolated Kafka cluster used exclusively for replication. Application services connect to kafka.corp.internal; Debezium, MirrorMaker, and the conflict consumer connect to a separate kafka-cdc.corp.internal that other developers don't even have firewall routes to.
• Pattern B — Prefixed namespaces with default-deny: set allow.everyone.if.no.acl.found = false and prefix every CDC topic (e.g. sys.cdc.infra.), then add an explicit wildcard deny rule so that even a developer's wildcard subscription cannot read the underlying replication stream.

14. Operational Case Study: Diagnosing a Circular Replication Loop
​
During validation of the reference implementation described in Section 11, a single UPDATE on Region A's database triggered a continuous stream of identical “processing” log lines on both regions' consumers — the classic circular-replication symptom described in Section 8. This case study documents the diagnosis and fix as a worked example.
​
14.1 Symptom

14.2 Root Cause
The consumer connected using application_name=cdc_replicator, but the Debezium connector configuration in place did not actually filter transactions based on that connection string. Region B's UPDATE generated a new WAL entry; Debezium B captured it; the mirror sent it back to Region A; Region A's consumer applied it, generating a new WAL entry on Region A — and the cycle continued in both directions simultaneously, exactly as described in Section 8's loop diagram.

14.3 Fix: PostgreSQL Replication Origins
The bulletproof fix — implemented natively in PostgreSQL rather than via an application-side flag — was to register a replication origin per directional flow, bind every consumer-written transaction to it with pg_replication_origin_session_setup(), and configure Debezium with "slots.stream.params": "origin=usernone" so it only emits changes generated by genuine application users, skipping anything tagged with a replication-origin session. This is the same mechanism detailed in Section 11.3.
​
14.4 A Related Failure Mode: Missing Origin Registration​
Immediately after deploying the fix, a fresh environment produced a different, more informative error:

​This occurred because the origin had never been created on that fresh database — the consumer attempted to tag a transaction with a name PostgreSQL didn't recognize, and the write rolled back immediately. The immediate fix is to run pg_replication_origin_create() once per database; the durable fix, recommended for any environment where containers are rebuilt regularly, is to have the consumer check for and create the origin automatically on startup rather than relying on a manual one-time command.

14.5 OutcomeAfter registering replication origins and redeploying the origin-aware Debezium configuration, a write on Region A synchronized to Region B exactly once and came to a clean halt — confirming the circuit breaker was functioning as designed.

AppendixA. Components of the Multi-Cluster Stage (Section 11)
Section 10 validated the replication logic with a single shared Kafka broker. Scaling that environment to the independent-cluster design in Section 11 means adding the following components per region:

• kafka-a / kafka-b: independent KRaft-mode Kafka brokers, one per region, replacing the single shared broker from Section 10.
• debezium-a / debezium-b: Kafka Connect workers, now registered with slots.stream.params: origin=usernone so they ignore replication-origin-tagged transactions.
• mirror-maker: a MirrorMaker 2 instance bridging kafka-a and kafka-b in both directions, defined in mm2.properties as tuned in Section 12.3.
• consumer.py (upgraded): the Section 10 consumer, extended with the replication-origin tagging shown in Appendix B below, parameterized as --region region_a|region_b.

B. Reference Consumer Skeleton (Origin-Aware Version)​
The Section 10 consumer upgraded with the replication-origin tagging from Section 11.3 — origin registration on startup, followed by the Last-Write-Wins apply logic with origin tagging, used once each region has its own Kafka cluster and bidirectional flow becomes possible:

C. Kafka Topics Created by MirrorMaker 2​
Observed on the Region A Kafka container in the reference environment:

D. Reading Replica Status in Kafdrop​
Kafdrop's Partition Detail table can be misread on a per-region cluster. In the reference environment, the Leader Node, Replica Nodes, and In-Sync Replica Nodes columns for a region-b topic all showed the single value 2 — which is not a count of nodes, but the broker ID of kafka-b itself.

E. Sample Cross-Region CDC Message​
A raw message as it appears on the region_a.public.users topic, viewed from Kafdrop, before schema stripping (Section 5.4):