SaFi Bank Space : Deploy Kafka-Connect in GKE Cluster for Confluent-Cloud

Kafka-connect deployed in dev cluster available here: https://kafka-connect.apps.dev.safibank.online/

Confluent for Kubernetes (CFK) is a cloud-native control plane for deploying and managing Confluent in a private cloud environment.

It provides a standard and simple interface to customize, deploy, and manage Confluent Platform through declarative API.

The following shows the high-level architecture of CFK and Confluent Platform.

Step-1: Deploy confluent-operator

Install the Confluent-operator using helm-chart which installs all the necessary CRDs to deploy kafka components like connect and connectors.

[~]$ kubectl get crds
NAME                                          CREATED AT
clusterlinks.platform.confluent.io            2022-09-14T04:59:11Z
confluentrolebindings.platform.confluent.io   2022-09-14T04:59:11Z
connectors.platform.confluent.io              2022-09-14T04:59:11Z
connects.platform.confluent.io                2022-09-14T04:59:12Z
controlcenters.platform.confluent.io          2022-09-14T04:59:14Z
kafkarestclasses.platform.confluent.io        2022-09-14T04:59:14Z
kafkarestproxies.platform.confluent.io        2022-09-14T04:59:14Z
kafkas.platform.confluent.io                  2022-09-14T04:59:15Z
kafkatopics.platform.confluent.io             2022-09-14T04:59:15Z
ksqldbs.platform.confluent.io                 2022-09-14T04:59:15Z
schemaexporters.platform.confluent.io         2022-09-14T04:59:15Z
schemaregistries.platform.confluent.io        2022-09-14T04:59:15Z
schemas.platform.confluent.io                 2022-09-14T04:59:16Z
zookeepers.platform.confluent.io              2022-09-14T04:59:16Z
[~]$

Deploy Confluent-Operator Helm chart using ArgoCD-Kustomization as mentioned here: ArgoCD-Deployment

Step-2: Deploy Kafka-connect

Pre-requisites:

Confluent-cloud environment/Cluster and Schema Registry should be deployed like this Confluent-Cloud-Environment
Confluent-cloud kafka Credentials should be pushed to hashicorp vault like this Confluent-cloud kafka Credentials
Confluent-cloud schema-registry credentials should be pushed to hashicorp vault like this Confluent-cloud schema-registry credentials

Deploy kafka-connect:

Create kubernetes secrets for the kafka cluster keys and Schema Registry keys

---
apiVersion: v1
kind: Secret
metadata:
  name: ccloud-credentials
  labels:
    app: ccloud
type: Opaque
data:
  plain.txt: <secret:secret/data/dev/confluent/kafka~jaasconfig-secret|base64>

---
apiVersion: v1
kind: Secret
metadata:
  name: ccloud-sr-credentials
  labels:
    app: ccloud
type: Opaque
data:
  basic.txt: <secret:secret/data/dev/confluent/schema-registry~basic-secret|base64>

Now Deploy the kafka-connect using the CRD’s Connect connects.platform.confluent.io deployed earlier.

---
apiVersion: platform.confluent.io/v1beta1
kind: Connect
metadata:
  name: connect
  labels:
    app.kubernetes.io/name: kafka-connect-dev
spec:
  replicas: 1
  image:
    application: confluentinc/cp-server-connect:7.2.0
    init: confluentinc/confluent-init-container:2.4.0
  build:
    type: onDemand
    onDemand:
      plugins:
        locationType: confluentHub
        confluentHub:
          - name: kafka-connect-jdbc
            owner: confluentinc
            version: 10.2.5
          - name: kafka-connect-ably
            owner: ably
            version: 2.0.3
          - name: kafka-connect-bigquery
            owner: wepay
            version: 2.3.4
          - name: kafka-connect-gcp-pubsub
            owner: confluentinc
            version: 1.2.0
  dependencies:
    kafka:
      bootstrapEndpoint: <secret:secret/data/dev/confluent/kafka~bootstrap-endpoint>
      authentication:
        type: plain
        jaasConfig:
          secretRef: ccloud-credentials
      tls:
        enabled: true
        ignoreTrustStoreConfig: true 
    schemaRegistry:
      url: <secret:secret/data/dev/confluent/schema-registry~endpoint>
      authentication:
        type: basic
        basic:
          secretRef: ccloud-sr-credentials
  mountedSecrets:
    - secretRef: bq-sinkconnector-creds
    - secretRef: pubsub-sourceconnector-creds

Kafka deployed as stateful-set as mentioned below.

To deploy source/sink connectors via this connect we need to mount the secrets using mountedSecrets that are required by those connectors .. for example, google pub/sub connector and BigQuery connectors needed google SA key.

We found an issue with HA for kafka-connect.. i.e. kafka-connect is getting down whenever new connector gets deployed via this connect

References

Introducing Confluent For Kubernetes(CFK)

Confluent For Kubernetes(CFK)

Demo: Confluent for Kubernetes(CFK)