we have to kafka connector cluster, one for business process and integrated with confluent cloud you can check it from https://confluent.cloud
the other is for slacker manger to sync message from confluent kafka cluster to gcp pub/sub topics, check it from https://kafka-connect-utils-data.apps.brave.safibank.online/
Kafka connector related code
kafka connector depends on docker image, we use confluent-init-container:2.4.0 as the base image and add customized jar files(utils.jar, java-pub-sub.jar) to plugin paths.
docker image:
https://github.com/SafiBank/SaFiMono/blob/main/.github/workflows/download-utils-jar-build-image.yaml
https://github.com/SafiBank/SaFiMono/blob/main/devops/docker/utils_jar/Dockerfile
terraform code:
connector templates:
How to check the detail configuration for kafka connector cluster
Go to Grafana to check the detail log information, the connector configuration and detail logs can retrieve from Grafana Loki component.
Connector cluster configuration
rest.advertised.host.name=connect-utils-data-0.connect-utils-data.safi-confluent-operator.svc.cluster.local rest.advertised.host.port=8083 admin.bootstrap.servers=SASL_SSL://pkc-vk2jj.asia-southeast1.gcp.confluent.cloud:9092 admin.sasl.mechanism=PLAIN admin.security.protocol=SASL_SSL bootstrap.servers=SASL_SSL://pkc-vk2jj.asia-southeast1.gcp.confluent.cloud:9092 config.providers=file config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider config.storage.replication.factor=3 config.storage.topic=safi-confluent-operator.connect-utils-data-configs confluent.topic.replication.factor=3 consumer.bootstrap.servers=SASL_SSL://pkc-vk2jj.asia-southeast1.gcp.confluent.cloud:9092 consumer.sasl.mechanism=PLAIN consumer.security.protocol=SASL_SSL group.id=safi-confluent-operator.connect-utils-data key.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schema.registry.basic.auth.credentials.source=USER_INFO key.converter.schema.registry.url=https://psrc-knmx2.australia-southeast1.gcp.confluent.cloud key.converter.schemas.enable=true listeners=http://0.0.0.0:8083 offset.flush.interval.ms=10000 offset.storage.replication.factor=3 offset.storage.topic=safi-confluent-operator.connect-utils-data-offsets plugin.path=/usr/share/java,/usr/share/confluent-hub-components,/data producer.bootstrap.servers=SASL_SSL://pkc-vk2jj.asia-southeast1.gcp.confluent.cloud:9092 producer.sasl.mechanism=PLAIN producer.security.protocol=SASL_SSL request.timeout.ms=20000 rest.advertised.listener=http retry.backoff.ms=500 sasl.mechanism=PLAIN security.protocol=SASL_SSL status.storage.replication.factor=3 status.storage.topic=safi-confluent-operator.connect-utils-data-status value.converter=io.confluent.connect.avro.AvroConverter value.converter.schema.registry.basic.auth.credentials.source=USER_INFO value.converter.schema.registry.url=https://psrc-knmx2.australia-southeast1.gcp.confluent.cloud value.converter.schemas.enable=true
