SaFi Bank Space : Data Team: PubSub, BigQuery, Google Functions, Vertex AI Feature store, Google Cloud Run

PubSub

PubSub TF module - https://github.com/SafiBank/SaFiMono/tree/main/devops/terraform/_modules/gcp/pubsub/v1 - creates:

• topic

  • encrypts topic with KMS

• push & pull subscriptions

Current configuration described in tf-env-data-infra:

• pubsub.tf

  • creatable topics and their subscriptions described in _files/pubsub.yaml

• pubsub_ms_slacker_manager.tf

BigQuery

Current configuration is described in tf-env-data-infra:

• big-query.tf

  • creatable datasets and their tables are described in _files/google_bq.yaml

    • table schemas are described in json-files in bq_table_schemas dir

  • datasets and tables are encrypted with KMS

Google Functions

• GC-functions itself are placed in separate repo (described in terraform/tf-repos/data-gcs.tf): https://github.com/SafiBank/DataTest/tree/main/services

  • Also described by @Gnanasekaran Gajendiran : https://safibank.atlassian.net/wiki/spaces/ITArch/pages/176259079/Requirement+And+Status#Google-Functions%3A

• Current infra configuration is described in tf-env-data-infra:

  • cloud-fuction-github.tf

Vertex AI Feature store

Still pending (?)

Google Cloud Run

Note: new service to deploy must first exist under SaFiMono/run/.

• Project repo deployed as Cloud Run service:
  SaFiMono/run/dbt/*

• GitHub action that will build, push (container image) and deploy as Cloud Run service.
  (e.g. data-analytics .github/workflows/app-grun-data-analytics-brave.yml)

• Current configuration is described in /tf-env-data-infra/cloud_run.tf

  • /tf-env-data-infra/_files/google_cloud_run.yaml

• How to test via curl(must have the role/permission roles/run.invoker)
  e.g. data-analytics

  curl -iv https://data-analytics-uiidkzmqsa-as.a.run.app \
  -H "Authorization: bearer $(gcloud auth print-identity-token)"