SaFi Bank Space : Cloudflare VPN Implementation

Overview:

VPN is provided by Cloudflare. The deployment of this VPN runs inside a GKE cluster. Necessary providers and resources are described bellow.

Workspaces:

safi-env-*-vpn-infra is used to provision the infrastructure (vpn tunnels, gke clusters, etc.)
safi-env-*-vpn-config prepared for future configuration of rules, policies, user groups etc.

GCP-Project:

safi-env-*-vpn google project which contains the GKE on which the VPN is deployed.

Cloudflare:

Cloudflare contains the network configuration for the tunnels and access rules which limit who has access to which resources.

Okta:

Identity provider, users and groups are configured there. They are used by Cloudflare to limit access to our resources.

HCV:

Tunnel secrets necessary for deployment and stored there. API_Tokens for Cloudflare and Okta terraform providers are also found here.

Installation steps:

The overall configuration of infrastructure can be found in https://github.com/SafiBank/SaFiMono/tree/main/devops/terraform/tf-env-vpn-infra .

Resource provisioning:

GKE needs to be provisioned inside GCP. Example configuration can be found in vpn_gke.tf
Cloudflare configuration has to be provisioned. The tunnel and routes have to be created, this configuration can be found in cloudflare_tunnel.tf
Cloudflared, which is the daemon that enables the tunnels has to be deployed into the GKE. The necessary files and configuration can be found in here
Okta has to be configured, to provide identity management for Cloudflare. Cloudflare needs to be setup to use Okta as indentity provider. Our configuration can be found in okta.tf
1. The necessary materials regarding this can be found at https://developers.cloudflare.com/cloudflare-one/identity/idp-integration/okta/ and https://registry.terraform.io/providers/cloudflare/cloudflare/latest/docs/resources/access_identity_provider

Manual steps necessary to be taken:

Private Routing(Split tunnels) is configured manually we use the include policy, so the private networks that are accessed trough Cloudflare were added there + the domains needed so Auth would work.

In okta.tf, add Okta groups in local variable that needs VPN access

locals {
  full_vpn_access_groups = [
    "DevOps"
  ]
  restricted_vpn_access_groups = [
    "Developers"
    "Meiro"
  ]
}

WARP clients are also authenticated, so add the Okta groups in “Policy for Device enrollment“ rules into the Safibank team is also configured manually, we enroll people who are in specific Okta Groups.

Firewall Rules (Gateway Network Policies) have to be configured manually. Cloudflare uses default allow all on it’s firewall.So we have deny all rules for each environment at the end of the ruleset.

Example of allow rule:

Example of deny rule:

Tunnels:

We have tunnels setup to each of our environments and towards CICD.

Tunnels shown in the dashboard of cloudflare:

Code to establish tunnels:

data "cloudflare_zone" "safi_domain" {
  name = local.cloudflare_domain
}

resource "random_password" "safi_domain" {
  length  = 32
  special = false
}

resource "cloudflare_argo_tunnel" "safi_vpn" {
  account_id = data.cloudflare_zone.safi_domain.account_id
  name       = format("%s-%s-vpn", local.prefix, var.env_name)
  secret     = random_password.safi_domain.result
}

resource "vault_generic_secret" "safi_vpn" {
  path = format("secret/%s/vpn/cloudflare", var.env_name)
  data_json = jsonencode(
    {
      "account_id"    = format("%s", data.cloudflare_zone.safi_domain.account_id),
      "tunnel_name"   = format("%s", cloudflare_argo_tunnel.safi_vpn.name),
      "tunnel_id"     = format("%s", cloudflare_argo_tunnel.safi_vpn.id),
      "tunnel_secret" = format("%s", cloudflare_argo_tunnel.safi_vpn.secret)
    }
  )
}

Each of those tunnels has routes, which point to networks in the Env/Cicd, which are configured in here for environments, and in here we have tunnels for CICD.

For each tunnel, we have routes to the GKE Cluster (Nodes and Master subnet). We also have routes towards GCP Services (Databases etc.) and towards private_default(VMs and other devices).

Example code of routes:

resource "cloudflare_tunnel_route" "gcp_services" {
  account_id = data.cloudflare_zone.safi_domain.account_id
  tunnel_id  = cloudflare_argo_tunnel.safi_vpn.id
  network    = local.curr_env.gcp_services
  comment    = "Route to GCP Services in the ${var.env_name} environment"

}

resource "cloudflare_tunnel_route" "gke_nodes" {
  for_each   = local.curr_projects
  account_id = data.cloudflare_zone.safi_domain.account_id
  tunnel_id  = cloudflare_argo_tunnel.safi_vpn.id
  network    = local.curr_env.projects[each.key].k8s.nodes
  comment    = "Route to GKE Nodes in the ${var.env_name} environment in ${each.key} project"
}

resource "cloudflare_tunnel_route" "gke_master" {
  for_each   = local.curr_projects
  account_id = data.cloudflare_zone.safi_domain.account_id
  tunnel_id  = cloudflare_argo_tunnel.safi_vpn.id
  network    = local.curr_env.projects[each.key].k8s.master
  comment    = "Route to GKE Master Subnet in the ${var.env_name} environment in ${each.key} project"
}

resource "cloudflare_tunnel_route" "private_default_nodes" {
  for_each   = local.private_default
  account_id = data.cloudflare_zone.safi_domain.account_id
  tunnel_id  = cloudflare_argo_tunnel.safi_vpn.id
  network    = local.curr_env.projects[each.key].private-default.nodes
  comment    = "Route to Private_Default Nodes in the ${var.env_name} environment in ${each.key} project"
}

resource "cloudflare_tunnel_route" "private_default_master" {
  for_each   = local.private_default
  account_id = data.cloudflare_zone.safi_domain.account_id
  tunnel_id  = cloudflare_argo_tunnel.safi_vpn.id
  network    = local.curr_env.projects[each.key].private-default.master
  comment    = "Route to GKE Master Subnet in the ${var.env_name} environment in ${each.key} project"
}

resource "cloudflare_tunnel_route" "confluent_kafka_cloud" {
  account_id = data.cloudflare_zone.safi_domain.account_id
  tunnel_id  = cloudflare_argo_tunnel.safi_vpn.id
  network    = local.curr_env.projects["applications"].confluent.kafka
  comment    = "Route to Confluent Kafka Cloud in the ${var.env_name} environment in apps project"
}

##### Should be corrected once done wiping dev env and deleted brave
# replace var.env_name to local.peer_env
locals {
  peer_env = "dev"
}
resource "cloudflare_tunnel_route" "meiro_private" {
  account_id = data.cloudflare_zone.safi_domain.account_id
  tunnel_id  = cloudflare_argo_tunnel.safi_vpn.id
  network    = local.safi_network.meiro[local.peer_env].subnets.private_subnets
  comment    = "Route to Meiro Private Subnet Resources in the ${var.env_name} environment in apps project"
}

resource "cloudflare_tunnel_route" "meiro_gcp_service" {
  account_id = data.cloudflare_zone.safi_domain.account_id
  tunnel_id  = cloudflare_argo_tunnel.safi_vpn.id
  network    = local.safi_network.meiro[local.peer_env].subnets.gcp_services
  comment    = "Route to Meiro GCP Service Subnet Resources in the ${var.env_name} environment in apps project"
}

Troubleshooting:

You can find the tunnel status on the zero trust dashboard.

Deployment status can be found in https://argocd.safibank.online/ .

Connection between Okta and Cloudflare can be tested in the dashboard under authentication . The test connect to Okta and returns information about the user that logged in.