SaFi Bank Space : Terraform Agents - PoC

We had to prove that we can use Terraform Agents effectively in our current environment, where significant points are :

we are using shared-vpc
we are having private gke cluster

With this in mind there are two problems which we are facing:

Terraform cannot access private resources (GKE) from TFC and we are notable to use any providers on given resource, we can only create it
(We can create GKE but we are not able to use Kubernetes, Helm Release, Kubectl providers)
ArgoCD is outside of out Shared VPC and if it wants to connect to GKE, effectively communication has to cross 3 VPCs(CICD-SharedVPC-GKE Master Node), which is not possible as VPC peering is not transient and we cant create peering explicitly from CICD to managed GKE Master Node VPC

Current solution is based on using terraform module which has 10+ null resources for:

downloading all the needed binaries (kubectl, gcloud, jq ..)
authenticate with cloud form TFC
trigger script which will
- deploy VM (no public connection to internet)
- create IAP tunnel
- execute installation of nginx via ssh over IAP
- create HTTP tunnel over this IAP tunnel and nginx to the GKE API
execute kubectl command over this tunnel, to deploy pre-rendered manifests (nginx + ArgoCD SA)
- this is solution for problem #2 - ArgoCD connection to GKE API via nginx proxy
retrieve required information like IP of nginx service or token from ArgoCD SA
destroys created VM

Next we store all information needed to the terraform state with a little “hack” due to how the module is operating and that we don't want to execute this process every time we run pipeline but we need this data for other resources. This is achieved by reading remote state by same terraform workspace and change values only when they are not empty (empty = nothing was executed and values should stay the same as run before)

Current solution (terraform local-exec + dynamic VM + IAP Tunnel)

This solution is essentially functional, but with its complexity is:

hard to read and understand (hand - over)
execution takes very long time
consist of many resources
is not in line with Terraform paradigm which should keep the state of infrastructure modeled via resources and code
there are some parts of solution which can fail due to unpredicted reasons which we cant influence and in this case the whole process has to be repeated again with manual triggering in code
this construct has to be deployed for every GKE we will install
it is not possible to manage artifacts in GKE, if they get deleted by accident, process has to be triggered manually

Proposed solution (Terraform Agents)

While we cant solve the problem #2, and we always need proxy for connection from CICD VPC, problem #1 can be simplified very much by using Terraform Agents:

Agents will be created by terraform itself as resource in SharedVPC
we dont have to manage VMs as we can use GCE with official HashiCorp docker image
with this solution and agents being placed in SharedVPC, we can have direct connection to all private resources and use providers for them
NGING Proxy and ArgoCD SA can be deployed as Kubernetes resources

Main points which we will get with this solution are:

stable idempotent process
less resources to maintain
increased code readability and understanding
we dont have to handle triggers and code reruning manually from code in case of errors
following Terraform paradigm
all components of the solution are supported
all deployed artifacts are managed by Terraform in standard way as resources

SaFi Bank Space : Terraform Agents - PoC

Current solution (terraform local-exec + dynamic VM + IAP Tunnel)

Proposed solution (Terraform Agents)

Attachments: