The current solution works and does not have any known security holes, if all services use iam-auth-lib correctly.

Even so, the security is fragile because it relies on the fact that the Cloudflare firewall setting is in line with the signature implementation (which require thoughtful attention when updating, e.g. when we updated from safi-sgn to safi-sign). A better approach would be (which is also in line with the zero trust philosophy) that the endpoints can check the authentication (and base their authorization decision on) something present instead of something missing (note: the latter is currently the case with developer access and service2service calls). To rephrase it: we want to have proof at the endpoint that some trusted component is enforcing access. This can be a trusted secure store and VIDA BE (signature - customer use case), can be a trusted auth provider, OKTA (JWT - back-office use case), can be identity based on VPN access (JWT token), can be Istio as the service mesh secure connection provider (mTLS and certificate), or any other component that we trust.

For reference, here is the current infrastructure:

So, the problem with the current solution is that we can’t distinguish between requests

from other services
coming through VPN
from API gateway (public requests).

Also, there is no possibility to allow fine grained access for developers, which will be an important feature in production when select developers need access. (Currently either all developers can access an endpoint or none can.)

Due to the fact that almost all services contain some internal only and some public endpoints, it is hard to enforce security policy on earlier points in the infrastructure.

For EPFS we’ve started to require for all requests, coming from public internet, to include signature headers, this check is oblivious to how each service works.

We propose two ways of how we can get this information to the services, so they can make proper authorization decisions. Both of them extends the current solution and allows to turn on the Istio based proof of request source:

This is not enabled currently (neither in `dev` nor in `epfs`)

Notes
When we do service to service call (like in the above diagram, service A call service B), we will use XFCC header to authenticate the request and it will be handled inside IAM lib.

Option #1: Separate Traefik ingress for trusted connections

Right know all traffic not originating from application services flows through a single ingress controller. Both requests from API gateway and other parts of infrastructure takes the same path. This means we can’t distinguish between public requests and internal requests to require appropriate security controls.

We can get the identity of individual pods in the same cluster from Istio. This opens up an easy solution to require trusted connections to take different path. We can do this by introducing additional Traefik Ingress controller with different Kubernetes service account, that will only be used for trusted connections (connections that don’t require header signature).

Access to this ingress controller must be correctly firewalled, because anything that can access this ingress controller will be treated as trusted (this includes both other applications running in the cluster and outside of it).

Additionally to fully solve our original problem of not being able to distinguish developers, we can add DNS override in Cloudflare VPN, to direct traffic coming from developers towards the new ingress controller which would allow us to use the same DNS names for both trusted and untrusted connections.

It’s important to note this option only allows a binary access control, either something is allowed to access the ingress controller or not, and everything with access to the ingress controller is treated in the same way. Either all developers can access an endpoint or none of them.

Option #2: Embrace Zero Trust and require identity all the way to the services

Zero trust ideology dictates that one shouldn’t ever trust and one should always verify. While mostly associated with corporate user access controls it makes sense even in applications. This is the reason why we have mTLS between services (provided by Istio). Problem with our current solutions is that we lose the identity information along the way and only know the last service/pod.

To remedy this we must inject additional verifiable identity information to the requests. For customer requests we have the request signatures, for service-to-service calls in the same cluster we have identity from Istio, but we don’t have any information from other sources of traffic. Calls between clusters can be authenticated by multi-cluster Istio deployment, but this still leaves requests from VPN. Cloudflare Zero Trust supports injecting identity in to the requests using Cf-Access-Jwt-Assertion header. This header contains all the information about the user including their groups. We can use this information in the services or anywhere along the requests path to limit what can users from VPN access.

What’s stopping us from implementing this solutions right now is Cloudflare being used only as layer 4 VPN, meaning it has not idea about application level traffic. We need it to be able to inject that header and this means it must be able to decrypt HTTPS connections.

To solve this we can start using Cloudflare as DNS provider for the safibank.online zone, because otherwise it can't terminate the HTTPS connections.

Benefit of this option is that we can filter traffic with higher granularity, for example only allowing access to services to development team that owns them instead of all developers or only allow backoffice users access to backoffice-manager and not all services.

SaFi Bank Space : Steps Towards service2service Authentication

This is not enabled currently (neither in `dev` nor in `epfs`)

Notes
When we do service to service call (like in the above diagram, service A call service B), we will use XFCC header to authenticate the request and it will be handled inside IAM lib.

Option #1: Separate Traefik ingress for trusted connections

Option #2: Embrace Zero Trust and require identity all the way to the services

Attachments: