This document details the Kafka topic mappings to Pub/Sub for all the relevant data environments. This connection is created largely for the purpose of data processing of Kafka messages.
The pub/sub topics associated with a Kafka topic, will inherit the same name and naming convention of the given Kafka topic. As long as the new name is within the 64 character restriction imposed by Terraform. If it exceeds 64 characters, then it will need to be renamed into something that is intuitive and allows one to identify the linked Kafka Topic.
Within 64 characters:
customers.device-fingerprint.snapshot.v1 → customer.device-fingerprint.snapshot
Exceeds 64 characters:
transactions.interbank-transaction-occurred.event.v3 → transaction.interbank-transaction-occur.event
Documentation for Pub/Sub Topic Mappings
Name | Documentation Link | Description |
|---|---|---|
Lark Summary File | Details the endpoints and pubsub topic names across all environments | |
GitHub Documentation for Topics | Details the existing list of topics as produced and consumed by the various microservices | |
Pub/Sub and Function interaction | Stream section of this documentation explains Pub/Sub use for FeatureStore ingestion |
Implementation
Creating new Topics in GCP Environment
In order to create new topics in the associated GCP Environment, one will need to edit the parameters as specified inside the .yaml as found on Github. See the link below. The .yaml config file is read upon deployment and the associated pub/sub topics will be deployed as per the specifications in the config file.
When creating new topics, it is best to reach out to DevOps with a ticket as well.
Name | Documentation Link | Description |
|---|---|---|
Pub/Sub config file | The pubsub.yaml file specifies the topic parameters, which is read by the terraform files and used to deploy the associated topic |