SaFi Bank Space : Hashicorp Vault Secret Injection

Content Sections

Ways to consume vault secret value

There are multiple ways to consume secret from vault, but due to disadvantage of other methods, 2 solutions will only be topic here which are:

Vault Aware - Writing an application to connect directly to a Vault server using Vault REST APIs, needs to include vault client libraries. (Best and current solution we are using) https://www.vaultproject.io/api-docs/libraries
Vault Injection - No changes from the application side, annotation from the deployment will inject sidecar pod that will communicate with vault server. (Backup solution for easy configuration)

Main components for Vault Injection

Kubernetes Authentication - Need to enable kube authentication to vault
Service Account binded to Role and Policy- Create a vault role and policy, then integrate it with kube service account.

Procedure

For demo, install vault in dev mode with helm

helm upgrade --install \
    vault hashicorp/vault \
    --namespace vault \
    --create-namespace \
    --set "server.dev.enabled=true" \
    --set "server.dev.devRootToken=root" 
    
kubectl -n vault get po
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          4h33m
vault-agent-injector-5b5889ffb4-gtnbj   1/1     Running   0          4h33m

3 things to enable kube auth in vault

1. token_reviewer_jwt - tokent issued by service account in the namescpace, ssh into any pod and run cat /var/run/secrets/kubernetes.io/serviceaccount/token
2. kubernetes_host - ${KUBERNETES_PORT_443_TCP_ADDR} value
3. kubernetes_ca_cert ssh into any pod and run cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt to check the value

kubectl -n vault exec -it vault-0 -- sh 

vault login # password token "root"

vault auth enable kubernetes

vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
issuer="https://kubernetes.default.svc.cluster.local"

Create a sample secret

kubectl -n vault exec -it vault-0 -- sh 

vault secrets enable -path=secret/ -version=1 kv

#create helloworld secret
vault kv put secret/basic-secret/helloworld username=dbuser password=sUp3rS3cUr3P@ssw0rd

Create a Policy that will be used by the role

kubectl -n vault exec -it vault-0 -- sh 

cat <<EOF > /home/vault/app-policy.hcl
path "secret/basic-secret/*" {
  capabilities = ["read"]
}
EOF
vault policy write basic-secret-policy /home/vault/app-policy.hcl

Create Vault role that will used the Policy

kubectl -n vault exec -it vault-0 -- sh 

#This role is bounded by basic-secret-policy, basic-secret serviceaccount, and foo namespace.
vault write auth/kubernetes/role/basic-secret-role \
   bound_service_account_names=basic-secret \
   bound_service_account_namespaces=foo \
   policies=basic-secret-policy \
   ttl=1h

Deployment

Deploy Sample app that will consume the secret

Add deployment annotations to automatically inject sidecar pod and consume secret
https://www.vaultproject.io/docs/platform/k8s/injector/annotations

annotations:
  vault.hashicorp.com/agent-inject: "true"      # enable sidecar pod inject
  vault.hashicorp.com/tls-skip-verify: "true"   # skip ssl
  vault.hashicorp.com/agent-inject-secret-${SECRET_NAME}: "secret/${SECRET_NAME}" # refer the secret by replacing the varialbe ${SECRET_NAME}
  vault.hashicorp.com/role: "basic-secret-role" # vault role to use

NOTE: Kubernetes service account should match what is configured in the role properties bound_service_account_names and bound_service_account_namespaces.

Deployment sample

kubectl create ns foo

cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: basic-secret
  labels:
    app: basic-secret
spec:
  selector:
    matchLabels:
      app: basic-secret
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/tls-skip-verify: "true"
        vault.hashicorp.com/agent-inject-secret-helloworld: "secret/basic-secret/helloworld"
        vault.hashicorp.com/role: "basic-secret-role"
      labels:
        app: basic-secret
    spec:
      serviceAccountName: basic-secret
      containers:
      - name: app
        image: jweissig/app:0.0.1
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: basic-secret
  labels:
    app: basic-secret
EOF

kubectl -n foo apply -f deployment.yaml

Verification

Once pod is in ready state, you can check the values by running

kubectl -n example-app exec <pod-name> -- sh -c "cat /vault/secrets/helloworld"

Defaulted container "app" out of: app, vault-agent, vault-agent-init (init)
password: sUp3rS3cUr3P@ssw0rd
username: dbuser