Pre-requisites
Must already successfully produced messages to the Kafka Topic with correct Avro schema setup
Do not use Confluent Control Center to test producing message to the Kafka Topic because it will not work with this error
SerializationException: Unknown magic byte!.Infra team must already setup Kafka component including Kafka Connect Transform plugin.
Infra team / DKatalis must already setup Ably Kafka Connector for Customer Channel and Account Channel. http://172.16.48.7:9021/clusters/3ATuWyTpROidRjVX0mDmvA/management/connect/connect-default/connectors
Flow :
Producer > Kafka Topic > Ably Kafka Connector > ably.io > Mobile app
Overview
Below are the steps to configure sync from the Kafka Topic to Ably Channel:
Add the Kafka Topic to the Ably Kafka Connector Sink configuration.
Open ably.io DEV console and subscribe to the channel you are mapping your Kafka Topic to.
Produce Avro messages to the Kafka Topic, you should be able to see it going into the ably.io DEV console.
Inform FE team and add Jira ticket for them to add an Ably handler at the mobile app for that Kafka topic event payload.
Detail Steps
Add the Kafka topic to the Ably Kafka Connector Sink Configuration
: please do not use same topic for multiple connector. This will kill the entire connector. We will automate this Kafka topic to ably channel mapping in the near future.
Below example is using account_updated topic being mapped to ably customer channel. Postman script is attached to this page at the bottom.
Open http://ably.io DEV console and subscribe to the channel you are mapping your Kafka Topic to
Please contact Common squad to add your email into Ably as tester prior to this; then accept the invitation.
Choose your environment
Select DEV console Tab
Subscribe (attach) to the relevant channel for your test
If you map your Kafka Topic to customer channel, then use customer:customerId.
For example: customer:3fa85f64-5717-4562-b3fc-2c963f66afa6
Produce Avro messages to the Kafka Topic, you should be able to see it going into the http://ably.io DEV console.
For this test we use Postman, it is the same as if you use your MS to produce the message. kafka proxy rest api v2 not support header and key with different serialization. It successfully produce message with kafka proxy rest api v3. for key and headers value, string is not supported, so it using BINARY. We need to encode real value with Base64 first.
Check the message is received in ably dev console
check the message is processed by FE
Inform FE team and add Jira ticket for them to add an Ably handler at the mobile app for that Kafka topic event payload.
Every event/command/snapshot you wish to be read by FE will need FE Ably Handler. You will also need to inform FE team what screen will be impacted and need to be refreshed by this event/command/snapshot.
Attached postman collection that used when testing
Related articles
Attachments:
image-20220720-093817.png (image/png)
kafka connect test safi dev.postman_collection.json (application/json)
image-20220720-094252.png (image/png)
image-20220720-094534.png (image/png)
image-20220720-094720.png (image/png)
image-20220720-095430.png (image/png)
image-20220720-095650.png (image/png)
image-20220720-095729.png (image/png)
image-20220720-095747.png (image/png)
image-20220722-085457.png (image/png)
Screen Shot 2022-07-22 at 16.00.36.png (image/png)
Screen Shot 2022-07-22 at 10.39.23.png (image/png)
Screen Shot 2022-07-22 at 10.39.57.png (image/png)
Screen Shot 2022-07-22 at 10.40.18.png (image/png)
Screen Shot 2022-07-22 at 10.40.49.png (image/png)
Screen Shot 2022-07-22 at 10.41.18.png (image/png)
Screen Shot 2022-07-22 at 10.41.18.png (image/png)
Screen Shot 2022-07-22 at 10.41.38.png (image/png)
