SaFi Bank Space : WIP - Data Exchange Between Database and Kafka

Created by Yuetong Yang, last modified on Nov 03, 2022

The Outbox Pattern
CDC Tool
Solution Architecture
Implementation details

The Outbox Pattern

http://www.kamilgrzybek.com/design/the-outbox-pattern/

https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/

CDC Tool

https://debezium.io/documentation/reference/2.0/

https://debezium.io/documentation/reference/2.0/architecture.html

https://docs.confluent.io/cloud/current/connectors/cc-postgresql-cdc-source-debezium.html

Solution Architecture

Implementation details

Sample workflow with CDC and outbox

Sample Outbox Table Schema

CREATE TABLE outbox
(
    id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregateid   varchar     NOT NULL, -- message key
    aggregatetype varchar     NOT NULL, -- kafka topic
    payload       bytea       NOT NULL, -- raw avro message binary value
    created_at    timestamptz NOT NULL,
    updated_at    timestamptz NOT NULL
);

Sample debezium connector config

{
  "name": "sample-outbox-connector",  
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector", 
    "plugin.name": "pgoutput",
    "database.hostname": "postgres", 
    "database.port": "5432", 
    "database.user": "postgresuser", 
    "database.password": "postgrespw", 
    "database.dbname" : "sample_db", 
    "database.server.name": "sample_db", 
    "table.include.list": "public.outbox",
	"key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "io.debezium.converters.ByteBufferConverter",
    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter"
  }
}

Attachments:

~drawio~557058:229b5867-a6cd-46a1-9572-1eb4b6e6294b~Outbox with cdc.tmp (application/vnd.jgraph.mxfile)

Outbox with cdc (application/vnd.jgraph.mxfile)

Outbox with cdc.png (image/png)

~drawio~557058:229b5867-a6cd-46a1-9572-1eb4b6e6294b~implementation details.tmp (application/vnd.jgraph.mxfile)

implementation details (application/vnd.jgraph.mxfile)

implementation details.png (image/png)

~implementation details.tmp (application/vnd.jgraph.mxfile)

implementation details (application/vnd.jgraph.mxfile)

implementation details.png (image/png)

~implementation details.tmp (application/vnd.jgraph.mxfile)

implementation details (application/vnd.jgraph.mxfile)

implementation details.png (image/png)