SaFi Bank Space : Mobile GRPC Setup

Create DNS Records:

Enable in microservices.yaml file like below

https://github.com/SafiBank/SaFiMono/blob/main/devops/terraform/_files/microservices.yaml#L31-L32

tyk_grpc: true
app_grpc: true

Terraform code to create DNS records

https://github.com/SafiBank/SaFiMono/blob/main/devops/terraform/tf-dns-safibankonline/apps_grpc_records.tf

resource "google_dns_record_set" "grpc_apps" {
  for_each = { for k, v in setproduct( [for k, v in local.safi_microservices : k if lookup(v, "app_grpc", false) != false], [for key in keys(local.safi_environments) : key if lookup(local.safi_environments[key], "enabled",false) != false] ) : format("%s-%s", v[0], v[1]) => {app=v[0], env=v[1]} }
  name         = format("%s-grpc.apps.%s.safibank.online.", replace(each.value.app, "_", "-"), each.value.env)
  managed_zone = google_dns_managed_zone.env_safibank_online[each.value.env].name
  type         = "A"
  ttl          = 300
  rrdatas      = [local.dns_app_internal_loadbalancer[each.value.env]]
}

https://github.com/SafiBank/SaFiMono/blob/main/devops/terraform/tf-dns-safibankonline/tyk_grpc_records.tf

resource "google_dns_record_set" "grpc_tyk" {
  for_each = {for k,v in setproduct([for k, v in local.safi_microservices : k if lookup(v, "tyk_grpc", false) != false], [for key in keys(local.safi_environments) : key if lookup(local.safi_environments[key], "enabled",false) != false]) : format("%s-%s", v[0], v[1]) => {app=v[0], env=v[1]}}

  name         = format("%s-tyk-grpc.tyk.%s.safibank.online.", replace(each.value.app, "_", "-"), each.value.env)
  managed_zone = google_dns_managed_zone.env_safibank_online[each.value.env].name
  type         = "A"
  ttl          = 300
  rrdatas      = [local.dns_tyk_external_loadbalancer[each.value.env]]
}

Run the terraform workspace

https://app.terraform.io/app/safi/workspaces/safi-dns-safibankonline

Helm Chart Modification:

Need to modify kotlin chart to create ingress for GRPC

https://github.com/SafiBank/SaFiMono/blob/main/devops/charts/kotlin/templates/ingress-grpc.yaml

{{- if .Values.ingressGrpc.enabled -}}
{{- $fullName := include "kotlin.fullname" . -}}
{{- $svcPort := .Values.serviceGrpc.port -}}
{{- if and .Values.ingressGrpc.className (not (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion)) }}
  {{- if not (hasKey .Values.ingressGrpc.annotations "kubernetes.io/ingress.class") }}
  {{- $_ := set .Values.ingressGrpc.annotations "kubernetes.io/ingress.class" .Values.ingressGrpc.className}}
  {{- end }}
{{- end }}
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1
{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
metadata:
  name: {{ $fullName }}-grpc
  labels:
    {{- include "kotlin.labels" . | nindent 4 }}
  {{- with .Values.ingressGrpc.annotations }}
  annotations:
    {{- toYaml . | nindent 4 }}
  {{- end }}
spec:
  {{- if and .Values.ingressGrpc.className (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion) }}
  ingressClassName: {{ .Values.ingressGrpc.className }}
  {{- end }}
  {{- if .Values.ingressGrpc.tls }}
  tls:
    {{- range .Values.ingressGrpc.tls }}
    - hosts:
        {{- range .hosts }}
        - {{ . | quote }}
        {{- end }}
      secretName: {{ .secretName }}
    {{- end }}
  {{- end }}
  rules:
    {{- range .Values.ingressGrpc.hosts }}
    - host: {{ .host | quote }}
      http:
        paths:
          {{- range .paths }}
          - path: {{ .path }}
            {{- if and .pathType (semverCompare ">=1.18-0" $.Capabilities.KubeVersion.GitVersion) }}
            pathType: {{ .pathType }}
            {{- end }}
            backend:
              {{- if semverCompare ">=1.19-0" $.Capabilities.KubeVersion.GitVersion }}
              service:
                name: {{ $fullName }}-grpc
                port:
                  number: {{ $svcPort }}
              {{- else }}
              serviceName: {{ $fullName }}
              servicePort: {{ $svcPort }}
              {{- end }}
          {{- end }}
    {{- end }}
{{- end }}

Modify values.yaml

https://github.com/SafiBank/SaFiMono/blob/main/devops/argocd/environments/brave/apps/card-manager/values.yaml#L33-L49

  ingressGrpc:
    enabled: true
    annotations:
      cert-manager.io/cluster-issuer: zerossl-dns
      ingress.kubernetes.io/protocol: h2c
      traefik.ingress.kubernetes.io/router.tls: "true"
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
      kubernetes.io/ingress.class: traefik-internal
    hosts:
      - host: card-manager-grpc.apps.brave.safibank.online
        paths:
          - path: /
            pathType: Prefix
    tls:
      - secretName: card-manager-grpc.apps.brave.safibank.online-tls
        hosts:
          - card-manager-grpc.apps.brave.safibank.online

TYK Modification:

Deploy Traefik ingress with external load balancer

https://github.com/SafiBank/SaFiMono/blob/main/devops/argocd/environments/common/infra/base/traefik-external.yaml

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: traefik-external
spec:
  generators:
  - list:
      elements:
      - name: safi-brave-tyk-a
        server: https://172.21.111.236:8081
        values:
          override: |
            deployment:
              replicas: 1
            podDisruptionBudget:
              enabled: true
              maxUnavailable: 1
            ingressClass:
              enabled: true
              isDefaultClass: true
            ingressRoute:
              dashboard:
                enabled: false
            providers:
              kubernetesIngress:
                ingressClass: traefik
                publishedService:
                  enabled: true
            ports:
              metrics:
                expose: true
            resources:
              requests:
                cpu: "100m"
                memory: "50Mi"
              limits:
                cpu: "300m"
                memory: "150Mi"
            metrics:
              prometheus:
                serviceMonitor:
                  relabelings:
                    - sourceLabels: [__meta_kubernetes_pod_node_name]
                      separator: ;
                      regex: ^(.*)$
                      targetLabel: nodename
                      replacement: $1
                      action: replace
                  jobLabel: traefik-external
                  scrapeInterval: 30s
                  scrapeTimeout: 5s
                  honorLabels: true
  template:
    metadata:
      name: '{{name}}-traefik-external'
    spec:
      project: dev-infra
      source:
        chart: traefik
        repoURL: https://helm.traefik.io/traefik
        targetRevision: 20.3.0
        helm:
          releaseName: traefik-external
          values: | 
              {{values.override}}
      destination:
        server: '{{server}}'
        namespace: traefik-external
      syncPolicy:
        automated:
          prune: true
        syncOptions:
        - CreateNamespace=true
        - Replace=true
        retry:
          limit: 5
          backoff:
            duration: 5s
            factor: 2
            maxDuration: 3m

Create GRPC SVC in TYK

https://github.com/SafiBank/SaFiMono/blob/main/devops/argocd/environments/brave/tyk/tykapigw/grpc-svc.yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    traefik.ingress.kubernetes.io/service.serversscheme: h2c
  labels:
    app: gateway-svc-tyk-apigw-brave-tyk-pro
  name: grpc-svc
  namespace: tyk
spec:
  ports:
  - name: grpc
    port: 4444
    protocol: TCP
    targetPort: 4444
  selector:
    app: gateway-tyk-apigw-brave-tyk-pro

Create GRPC Ingress in TYK

https://github.com/SafiBank/SaFiMono/blob/main/devops/argocd/environments/brave/tyk/tykapigw/card-manager-grpc.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: zerossl-dns
    ingress.kubernetes.io/protocol: h2c
    kubernetes.io/ingress.class: traefik-internal
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
  labels:
    app: gateway-tyk-apigw-brave-tyk-pro
  name:  card-manager-grpc
  namespace: tyk
spec:
  rules:
  - host: card-manager-grpc.smallog.tech
    http:
      paths:
      - backend:
          service:
            name: grpc-svc
            port:
              number: 4444
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - card-manager-grpc.smallog.tech
    secretName: card-manager-grpc.smallog.tech-tls

Mapping in TYK

GRPC Testing

MS Level Testing

❯ grpcurl   -proto CardManagerV1.proto -d '{"cardId": "02ca3c4b-ba09-4aef-81e6-0419cd15ad4d", "isSyncTracking": "false"}' card-manager-grpc.apps.brave.safibank.online:443  cardmanagerv1.CardManagerV1/GetCardDetails
{
  "cardId": "02ca3c4b-ba09-4aef-81e6-0419cd15ad4d",
  "accountNo": "0275243584",
  "customerId": "ebe2e661-8e08-4b47-a938-97449aed1a78",
  "cardType": "DD1",
  "cardPicture": "https://storage.googleapis.com/test-safi-dev/ebe2e661-8e08-4b47-a938-97449aed1a78/ebe2e661-8e08-4b47-a938-97449aed1a78_02ca3c4b-ba09-4aef-81e6-0419cd15ad4d_7.png",
  "namePrinted": false,
  "dynamicCVV": false,
  "internationalBlockingFlag": false,
  "embossName": "Maria Santos Dela Cruz",
  "maskingCardNo": "436434XXXXXX5438",
  "expiration": "20271130",
  "cardStatus": "ACTIVE",
  "orderingNumber": 1,
  "key": "COrqNGD",
  "createdAt": "2022-11-03T08:56:54.802Z",
  "updatedAt": "2022-12-05T02:55:33.905Z"
}

TYK Level Testing

❯ grpcurl   -proto CardManagerV1.proto -d '{"cardId": "02ca3c4b-ba09-4aef-81e6-0419cd15ad4d", "isSyncTracking": "false"}' card-manager-grpc.smallog.tech:443  cardmanagerv1.CardManagerV1/GetCardDetails
{
  "cardId": "02ca3c4b-ba09-4aef-81e6-0419cd15ad4d",
  "accountNo": "0275243584",
  "customerId": "ebe2e661-8e08-4b47-a938-97449aed1a78",
  "cardType": "DD1",
  "cardPicture": "https://storage.googleapis.com/test-safi-dev/ebe2e661-8e08-4b47-a938-97449aed1a78/ebe2e661-8e08-4b47-a938-97449aed1a78_02ca3c4b-ba09-4aef-81e6-0419cd15ad4d_7.png",
  "namePrinted": false,
  "dynamicCVV": false,
  "internationalBlockingFlag": false,
  "embossName": "Maria Santos Dela Cruz",
  "maskingCardNo": "436434XXXXXX5438",
  "expiration": "20271130",
  "cardStatus": "ACTIVE",
  "orderingNumber": 1,
  "key": "COrqNGD",
  "createdAt": "2022-11-03T08:56:54.802Z",
  "updatedAt": "2022-12-05T02:55:33.905Z"
}

SaFi Bank Space : Mobile GRPC Setup

Create DNS Records:

TYK Modification:

GRPC Testing

Attachments: