Recording Path: https://advancegroup.larksuite.com/drive/folder/fldus9dedXC2REO9A8FGHaDkxxc?from=space_persnoal_filelist
Attendees: Nov 25, 2022 Part 1
Peter Kmec (Unlicensed) Peter Luknár (Unlicensed) Gnanasekaran Gajendiran Lucky La Torre (Unlicensed) Fol Justin Lacsina (Unlicensed) Regin Villamor (Unlicensed) Joebert Jacaba (Unlicensed)
Attendees: Nov 28, 2022 Part 2
Responsibilities
Kafka
Confluent Cloud Kafka
Do you have any confluence doc created that will give us some idea of the current Confluent Cloud setup?
Confluent Kafka Implementation For setting up org and env
Doc for review if needs to be updated Peter Kmec (Unlicensed)
for env creation pls doc the stages Peter Kmec (Unlicensed)
For kafka-connect - Deploy Kafka-Connect in GKE Cluster for Confluent-Cloud but need to be updated
In setting up the topics and schemas, the same doc needs to be updated Peter Kmec (Unlicensed)
How is this deployed/created? Is it also in code?
everything is in tf
Access for Developers to be done today PK - Peter Kmec (Unlicensed)
Cant create user automatically on the Confluent side.
Add person manually by email , once accepted, then we can apply the role using the terraform
TF custom ccloud provider
Confluent Operator
https://github.com/SafiBank/safi-tf-providers/tree/main/terraform-provider-ccloud
Maintained by devs common/schema/topicSchemasDefinitions.json
Can you mention what are the manual stes that needed to be done and that may needs to be automated?
Retrieval of group id only, no manual config creation
Any possible scaling issues that we may encounter?
cku only way to scale
Do you see any issue that may arise in prod with the current package/subscription?
since we have dedicated, there should be no issues in re to the package/subscription
Any incompatibility issues we need to be aware of?
only the registry api key that lacks the support on tf
Any existing open issues with the provider we need to be aware of?
none
For Confluent Vendor, did you have any internal comms with the vendor, whats the name of the person/product manager/support if there is one?
slack channel - ask ion to invite the safi #confluent_kafka
Are all the applications now pointed to the confluent cloud kafka?
brave is the only one
old stage is using the kafka vm
old dev?
dedicated cluster in the old dev that we are using for TM 4.4.1 that is still using internal kafka
next step on tm 4.4.1 is to finalize the oauth comms and to test the connection to the ccloud kafka
Do we have any dangling kafka resources that needs to be deleted or not in use or still in use by dev but soon we have to delete them as they are redundant?
confluent marketplace - needs to be deleted from dispatcher
Is there anything that we need to change with the existing cloud kafka setup for it to work with Thought Machine in Dev?
nothing since the dedicated ccloud kafka cluster supports oauth and will be tested with tm 4.4.1
confluent operator
how the scaling will happen if theres only one confluent operator
no need to scale the operator, the connectors are the ones you need to scale
PostgreSQL
Were there any plans that you have discussed before when it comes to creating the Production instances.
Are there any plans/agreements for creating 1:1 instance (microservice to database instance ratio) Whats your recommendation?
try doing in lower env before prod
josef is working on the POC on Alloydb which we may get some info by next week
To address the following
Security
Scaling
High Availability
Disaster Recovery
If none, what would you recommend based on the current performance of the dbs and issues encountered on the dbs - not needed
Are all databases secrets stored in CICD Vault for all databases deployed in all of the Environments?
Yes
For Database Monitoring, what were the initial plan?
If there are none, would you recommend using the Google Cloud Monitoring for the Cloud SQL?
Do you suggest using the cloudsql grafana integration?
Database Migration
Do you see any migration that we need to do for any environments?
Do we have any github action that has any db migration script?
For manipulating prod db in the future, do you suggest doing this in GHA? (in respect with automation and security)
Outstanding Issues
Are we encountering any issues with connection pools right now? (concurrent connections)
Were there any reported db issue currently that is still open or in the past that were not addressed?
None that were aware of
On Postgres switching to Alloydb
What is the current status of this?
josef is doing a poc
What are the challenges of using this and migrating to Alloydb?
will be provided on the poc document
What was the reason why we need to switch to Alloydb?
because it was managed and other features not available in pg
What are the next plans on Alloydb?
still in poc
Are there any existing work being done on AlloyDB or testing being done by our team?
josef
Object storage
All GCS buckets are in Terraform (atleast for environment projects, excluding Safi Dev (dkatalis)), True?
Aside from the assets bucket, do we have or do we need to set any current buckets to public?
based on current requirements of apps, do we need to set or have we set any versioning or lifecycle policies in any of the buckets?
to be continued-- Nov 28, 2022
AlloyDB
A: Terraform provider is in beta - permission issues are pending
Data team (Big Data analysis)
Link to the Documentation on what is currently setup and implemented in GCP or outside GCP for Data Team. - BQ, CFN, Pub/Sub, Vertex AI(Jupyter notebook- API )
Did we configure anything manually for them? (them being data team) - everything is in terraform except firebase ios, android mobile apps with terraform provider - enable analytics is manual
Whats being done on the Data Team side right now, any open tickets? - kafka connectors tikets-
How are the (datasets) Bigquery being managed right now? (in relation to cost management) - no performance tuning
In relation to security and permissions, all of the access for data team gcp projects are also in terraform? - Mobile team assests gcs bucket
Can you provide us some details on how the following resources are deployed and being utilized right now? (for GCP Data)
Data Studio - Visualisation purpose -Nothing
Cloud Composer - Terraform 1. Data team 2. Applications team - cloud-composer-infra TFC -Workspace
Cloud Functions - Gnanasekaran Gajendiran did it in GHA
Pubsub Topics - Terraform
Backup
GKE Backup - New feature
Ably
Any Ably Documentation for DevOps related tasks? - creation of environments and keys has been automated - only manual is ios certs, firebase keys
Can you walk us through on what needs to be done if there’s still anything pending and what has been done so far? automation accounts for ably needs to be changed - TFC variable is manually added - dispatcher tfc variables
Sergei Tasks
encryption(cloud KMS- bucket/pubsub/BQ/GKE), DR, HA GKE Replicas, PostgreSQL, Data workspace, Firebase automation
High Availability for GCP resources & K8S apps
What are the outstanding tasks being done by Sergei? - Firebase mobile application config automation using terraform -
Is he handling the automation of DKatalis created resources in the
acquired-badge-348405project? - all the tickets are completed except Firbase automationStatus of the Firebase automations. (and related documentation)- still work in-progress
VAPT Tickets
Are there any outstanding tickets raised to you or any member of the VL devops that is in progress?
SAF-130 - SaFi-2022-9 SaFi Mobile Application Information Disclosure via Stack Trace Backlog
full VAPT vulnerabilities report to review
Security
Do we have any documentation on specific security processes we implemeted? (tf scan(trivy scan), kms etc) → check with Aleksandr Kanaev (Unlicensed) (Trivy, Threatmapper and others) and Gregor Zaťko (Unlicensed) (Sentinel policies)