SaFi Bank Space : Euronet VPN Implementation

For our connection to the Euronet Sandbox we have implemented IPsec VPN in the safi-env-x-sharedvpc project.

VPN Implementation:

The VPN is implemented trough terraform with the use of HCV for secret management.

The terraform can be found in the tf-environments folder, and is being run from the safi-env workspaces.
The necessary secrets can be found in the vault, they are different per environment but for brave they can be found here.

locals {
  udp_rules = {
    udp500  = "500"
    udp4500 = "4500"
  }
}

#Create the Public IP for the Gateway
resource "google_compute_address" "vpn-public-ip" {
  provider = google-beta.sharedvpc
  name     = "${local.prefix}-${var.env_name}-vpn-public-ip"
}
#Create the VPN Gateway
resource "google_compute_vpn_gateway" "vpn-gateway" {
  provider = google-beta.sharedvpc
  name     = "${local.prefix}-${var.env_name}-vpn-gateway"
  network  = google_compute_network.shared_vpc.id
}

#Create tunnel between Safi and Euronet
resource "google_compute_vpn_tunnel" "euronet-tunnel" {
  provider = google-beta.sharedvpc
  name     = "${local.prefix}-${var.env_name}-euronet-tunnel"
  peer_ip                 = data.vault_generic_secret.euronet.data["euronet_peer_ip"]
  shared_secret           = data.vault_generic_secret.euronet.data["euronet_shared_key"]
  local_traffic_selector  = [local.curr_env.projects["applications"].k8s.secondary_ranges["euronet-pods"]]
  remote_traffic_selector = [data.vault_generic_secret.euronet.data["euronet_remote_subnet"]]
  target_vpn_gateway = google_compute_vpn_gateway.vpn-gateway.id

  depends_on = [
    google_compute_forwarding_rule.forwarding_esp,
    google_compute_forwarding_rule.forwarding_udp
  ]
}
#Forwarding Rules for VPN
resource "google_compute_forwarding_rule" "forwarding_esp" {
  provider    = google-beta.sharedvpc
  name        = "${local.prefix}-${var.env_name}-firewall-rule-esp"
  ip_protocol = "ESP"
  ip_address  = google_compute_address.vpn-public-ip.address
  target      = google_compute_vpn_gateway.vpn-gateway.id
}

resource "google_compute_forwarding_rule" "forwarding_udp" {
  provider    = google-beta.sharedvpc
  for_each    = local.udp_rules
  name        = "${local.prefix}-${var.env_name}-firewall-rule-${each.key}"
  ip_protocol = "UDP"
  port_range  = each.value
  ip_address  = google_compute_address.vpn-public-ip.address
  target      = google_compute_vpn_gateway.vpn-gateway.id
}

#Create a Route that points towards Euronet Subnets
resource "google_compute_route" "euronet-route" {
  provider   = google-beta.sharedvpc
  name       = "euronet-route"
  network    = google_compute_network.shared_vpc.id
  dest_range = data.vault_generic_secret.euronet.data["euronet_remote_subnet"]
  priority   = 1000

  next_hop_vpn_tunnel = google_compute_vpn_tunnel.euronet-tunnel.id
}
#Firwall rules for Safi to Euronet Traffic
resource "google_compute_firewall" "euronet-rules" {
  provider    = google.sharedvpc
  name        = "${local.prefix}-${var.env_name}-euronet-vpn"
  description = "Managed by vpn.tf don't modify in console"
  network     = google_compute_network.shared_vpc.id
  direction   = "EGRESS"

  destination_ranges = [data.vault_generic_secret.euronet.data["euronet_remote_subnet"]]

  allow {
    protocol = "tcp"
    ports = [
      "443", #Euronet Provided Port
    ]
  }
}

data "vault_generic_secret" "euronet" {
  path = format("secret/%s/vpn/euronet", var.env_name)
}

GKE and Kubernetes Configuration:

The next requirement for Euronet other than to use IPSec VPN was the network size limit we got from them, which was at max /24 network, for this we have created a specific node-pool to which we deploy all things that have to have access to the sandbox.

The node_pool is configured in applications-project in applications_gke.tf

resource "google_container_node_pool" "standard-2vcpu-8gb_vpn" {
  provider          = google-beta.applications
  name_prefix       = "euronet-gtw-"
  location          = var.google_region
  cluster           = module.gke_shared_vpc_applications.cluster.name
  node_count        = 1
  max_pods_per_node = 32

  network_config {
    pod_range = "gke-euronet-pods-range"
  }
  node_config {
    preemptible  = false
    machine_type = "n2-standard-2"
    labels       = {
      "cloud.google.com/gke-role" = "euronet-gateway"
    }

    workload_metadata_config {
      mode = "GKE_METADATA"
    }

    taint {
      key    = "safi_nodegroup_class"
      value  = "euronet_gateway"
      effect = "NO_SCHEDULE"

    }
    service_account = module.gke_shared_vpc_applications.service_account.email
    oauth_scopes    = ["https://www.googleapis.com/auth/cloud-platform"]
    tags            = ["${local.prefix}-${var.env_name}-vpn-applications-node-pool"]
  }
  autoscaling {
    min_node_count = 0
    max_node_count = 4
  }

  lifecycle {
    ignore_changes = [
      node_count,
    ]
    create_before_destroy = true
  }
}

We then use tolerations, affinity and antiaffinity to deploy the necessary services and only those to this specific node pool:

  tolerations:
    - key: "safi_nodegroup_class"
      operator: "Equal"
      value: "euronet_gateway"
      effect: "NoSchedule"

  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: cloud.google.com/gke-role
                operator: In
                values:
                  - euronet-gateway
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: app
                  operator: In
                  values:
                    - euronet-gateway
            topologyKey: topology.kubernetes.io/zone