Created by BharathKumar D, last modified by Pavol Antalík on Dec 21, 2022
Step 1: Mirror the 4.4.1 images from TM
Here is the pipeline to mirror the images form tm private docker registry to our gcp artifactory docker registry: GHA Pipeline
Step 2: HashiCorp Vault configuration
Service accounts and roles
As part of the Vault installation, secrets, policies and roles are pushed to HashiCorp Vault.
The Kubernetes auth backend in HashiCorp Vault should be configured to bind the Vault installer Kubernetes service account to a role with permissions to push secrets, policies and roles. This ensures the Crown Operator has permission to push these.
We have terraform code to do the above steps.
Step 3: HashiCorp Vault root-db-secrets
Install the root user password in a secret called root-db-secrets with the DB host name as the key and the password as the value. This should be underneath the used for storing secrets for the Vault instance
Step 4: Get the available components to install using vaultctl command
./vaultctl components -r vault-4.4.1.release.zip
List of components
webhook-operator:
defaultNamespace: webhook-operator
description: |
An operator that manages the creation and deletion of Mutating and Validating Webhook Configurations.
It also handles certificate rotations required for secure communication between kubernetes control plane and backend webhook services.
This should always be installed prior to any Vault installation,
as it manages critical webhook configurations required to mutate Kubernetes resources for Vault installation.
singleton: true
istio:
defaultNamespace: tm-system
description: |
A service mesh that can be deployed to Kubernetes. When operating distributed systems such as Vault,
this provides management of the communication between the various services, offering features such as traffic management,
improved reliability and performance, policy enforcement and telemetry.
This on-cluster version of Istio is provided for ease of use. Using your own Istio setup is also supported.
singleton: true
observability:
defaultNamespace: tm-monitoring
description: |
The Observability Stack is required alongside Vault. You can then use the Observability Stack
to view and observe the status of Vault using metrics, dashboards and alerts. This component will deploy prometheus-adapter RBAC
resources to "kube-system" namespace, regardless of given deployment namespace.
singleton: true
kafka:
defaultNamespace: tm-vault
description: |
Used for streaming data. This on-cluster version of Kafka for ease of use, but Thought Machine does not
provide production support for it and therefore it is not appropriate for production. Thought Machine highly recommends
using alternative Kafka clusters, either managed or self-hosted. This component is intended for development purposes only.
It requires a multi-AZ (Availability Zone) cluster. Upgrading this component from an older version could require manual intervention
to ensure that resources are spread evenly across all AZs.
kafka-init:
defaultNamespace: tm-vault
description: |
Component for bootstrapping kafka. Contains the Vault Kafka init job that
must be deployed to setup Kafka access permissions. If using this, make sure to install the
kafka-cleanup component after installing vault.
vault-core:
defaultNamespace: tm-vault
description: |
This component deploys the minimum packages required to run Vault Core.
replaces: vault
payments-hub:
defaultNamespace: tm-vault
description: |
This component deploys the required packages to enable the Vault Payment Hub, including the FPS and OnUs payment schemes.
replaces: vault
payments-hub-bacs:
defaultNamespace: tm-vault
description: |
This component deploys the additional packages to enable Vault Payment Hub BACS payment scheme.
replaces: vault
payments-hub-bottomline:
defaultNamespace: tm-vault
description: |
This component deploys the additional packages to enable Vault Payment Hub Bottomline payment gateway connectivity.
replaces: vault
payments-hub-form3:
defaultNamespace: tm-vault
description: |
This component deploys the additional packages to enable Vault Payments Hub Form3 payment gateway connectivity.
replaces: vault
xpl:
defaultNamespace: tm-vault
description: |
This component deploys the required packages to enable the Vault Experience Layer (XPL).
replaces: vault
xpl-payments-hub:
defaultNamespace: tm-vault
description: |
This component deploys the required packages to enable the Vault Payment Hub Experience Layer (XPL) Connector.
replaces: vault
migration:
defaultNamespace: tm-vault
description: |
This component deploys the services needed for migrations. This includes the data loader and the Posting API deployments
of Vault that allow the migration of historical postings. It is advised that this package is undeployed after the financial
migration is complete.
replaces: vault
migration-xpl:
defaultNamespace: tm-vault
description: |
For clients using the Experience Layer (XPL) this component will create transaction resources for certain postings
when using the postings migration topic.
replaces: vault
integration-vault-postings:
defaultNamespace: tm-vault
description: |
This component deploys the services needed for the Vault Payments Postings Connector.
Self-hosted clients do not need this as Vault Payments is currently SaaS only.
replaces: vault
kafka-cleanup:
defaultNamespace: tm-vault
description: |
Deploy IF kafka-init was deployed. This job should be run at the end of a Vault installation or upgrade
to delete Kafka access permissions that are no longer needed.
kafka-ca-rotation:
defaultNamespace: tm-vault
description: |
This component deploys a CronJob for the periodic rotation of Kafka CA.
replaces: vault
kafka-cert-rotation:
defaultNamespace: tm-vault
description: |
This component deploys a CronJob for the periodic rotation of Kafka certificates.
replaces: vault
cockroachdb-ca-rotation:
defaultNamespace: tm-vault
description: |
This component deploys a CronJob for the periodic rotation of Cockroach DB CA.
replaces: vault
cockroachdb-cert-rotation:
defaultNamespace: tm-vault
description: |
This component deploys a CronJob for the periodic rotation of Cockroach DB certificates.
replaces: vault
cert-rotation:
defaultNamespace: tm-vault
description: |
This component is for one-off certificate rotation. It is used by vaultctl for the
'rotate-certs' command and cannot be installed directly.
clusterstat:
defaultNamespace: tm-vault
description: |
Component to run clusterstat tests after vault component install.
saml-idp:
defaultNamespace: tm-vault
description: |
This component deploys the Vault dummy saml-idp service. This can be used during development and testing before a
third party Identity Provider (IdP) is used in production.
replaces: vault
Step 5: Install the operator 1.tmcomponent-operator and 2.crown-operator using vaultctl
create namespaces tm-system tm-monitoring tm-vault istio-system as below
kubectl --context <cluster_name> create ns tm-system
kubectl --context <cluster_name> create ns tm-monitoring
kubectl --context <cluster_name> create ns istio-system
kubectl --context <cluster_name> create ns webhook-operator
kubectl --context <cluster_name> create ns tm-vault
kubectl --context <cluster_name> label ns tm-vault --overwrite istio-annotation-tm-webhook-tm-vault=enabled
kubectl --context <cluster_name> label ns tm-vault --overwrite istio.io/rev=1.15.3
and run the vaultctl command as below - vaultctl binary is provided from TM-4.4.1 release files
vaultctl init-operators --context <cluster_name> -d "asia-southeast1-docker.pkg.dev/safi-repos/safi-tm/" --cloud-provider gcp --distribution gke
Example output
Creating cluster level object platform-critical (Kind: PriorityClass)
Object platform-critical (Kind: PriorityClass) already exists. Patching it
Creating cluster level object platform (Kind: PriorityClass)
Object platform (Kind: PriorityClass) already exists. Patching it
Creating cluster level object category-a (Kind: PriorityClass)
Object category-a (Kind: PriorityClass) already exists. Patching it
Creating cluster level object category-b (Kind: PriorityClass)
Object category-b (Kind: PriorityClass) already exists. Patching it
Creating cluster level object category-c (Kind: PriorityClass)
Object category-c (Kind: PriorityClass) already exists. Patching it
Creating namespaced object check-vault (Kind: ServiceAccount) to namespace tm-system
Creating namespaced object deploy-istio (Kind: ServiceAccount) to namespace tm-system
Creating cluster level object tmcomponent-operator-view (Kind: ClusterRole)
Creating cluster level object tmcomponent-operator-edit (Kind: ClusterRole)
Creating cluster level object tmcomponent-operator-admin (Kind: ClusterRole)
Creating namespaced object tm-component-operator (Kind: ServiceAccount) to namespace tm-system
Creating cluster level object crown-operator-view (Kind: ClusterRole)
Creating cluster level object crown-operator-edit (Kind: ClusterRole)
Creating cluster level object crown-operator-admin (Kind: ClusterRole)
Creating namespaced object vault-installer (Kind: ServiceAccount) to namespace tm-system
Creating cluster level object vault-istio (Kind: ClusterRole)
Creating cluster level object check-vault (Kind: ClusterRole)
Creating cluster level object tm-component-operator (Kind: ClusterRole)
Creating cluster level object crown-operator (Kind: ClusterRole)
Creating cluster level object check-vault (Kind: ClusterRoleBinding)
Creating cluster level object vault-istio (Kind: ClusterRoleBinding)
Creating cluster level object tm-component-operator (Kind: ClusterRoleBinding)
Creating cluster level object crown-operator (Kind: ClusterRoleBinding)
Creating cluster level object tmcomponents.tmachine.io (Kind: CustomResourceDefinition)
Creating cluster level object crowns.tmachine.io (Kind: CustomResourceDefinition)
Creating namespaced object tmcomponent-operator (Kind: Deployment) to namespace tm-system
Creating namespaced object crown-operator (Kind: Deployment) to namespace tm-system
tmcomponent-operator and crown-operator are ready!
We have automated the above steps using ArgoCD.. here is the ArgoCD vault-operator to install the 1.tmcomponent-operator and 2.crown-operator
Step 6: Generate the Values.yaml and update the values accordingly
./vaultctl example-values -r vault-4.4.1.release.zip kafka vault-core > 4.4.1-values.yaml
ls -rtlh 4.4.1-values.yaml
-rw-r--r-- 1 bharath_dasaraju bharath_dasaraju 80K Nov 2 06:56 4.4.1-values.yaml
The example values file i.e. 4.4.1-values.yaml file is as below.
Updated values file for 4.4.1-values.yaml file is as below.
Step 7: Install components
parameter --context referes to the kubernetes kube config context:
vaultctl install webhook-operator --context "SaFi Dev TM" -v 4.4.1-values.yaml -r vault-4.4.1.release.zip
# for testing the Kafka can be installed (NOT recommended for PROD, no support)
vaultctl install kafka --context "SaFi Dev TM" -v 4.4.1-values.yaml -r vault-4.4.1.release.zip
vaultctl install kafka-init --context "SaFi Dev TM" -v 4.4.1-values.yaml -r vault-4.4.1.release.zip
# for testing the dummy SAML component can be installed, Okta integration configured in Dev (using Terraform)
vaultctl install saml-idp --context "SaFi Dev TM" -v 4.4.1-values.yaml -r vault-4.4.1.release.zip
# main ThoughtMachine installation
vaultctl install vault-core --context "SaFi Dev TM" -v 4.4.1-values.yaml -r vault-4.4.1.release.zip
# observability package (Prometheus/Thanos/Grafana)
vaultctl install observability --context "SaFi Dev TM" -v 4.4.1-values.yaml -r vault-4.4.1.release.zip
uninstalling components example (some of the components add name of the namespace to the component name during install, so vault-core becomes vault-core-tm-vault:
vaultctl uninstall vault-core-tm-vault --context "SaFi Dev TM"
checking status of the installation:
vaultctl status --context "SaFi Dev TM"
example output:
Click here to expand...
Component: kafka-tm-vault
Release: vault-4.4.1
Status: ReadinessCheckComplete
Conditions:
TYPE STATUS LAST HEARTBEAT TIME LAST TRANSITION TIME REASON MESSAGE
Initialization True 2022-12-21T02:26:58Z 2022-12-15T09:41:15Z
Deployment True 2022-12-21T02:26:58Z 2022-12-15T14:51:47Z
ReadinessCheck True 2022-12-21T02:26:58Z 2022-12-15T09:53:35Z
GarbageCollection True 2022-12-21T02:26:58Z 2022-12-15T09:53:35Z
<=============================>
Component: observability
Release: vault-4.4.1
Status: ReadinessCheckComplete
Conditions:
TYPE STATUS LAST HEARTBEAT TIME LAST TRANSITION TIME REASON MESSAGE
Initialization True 2022-12-21T02:26:58Z 2022-12-20T17:02:59Z
Deployment True 2022-12-21T02:26:58Z 2022-12-20T17:03:22Z
ReadinessCheck True 2022-12-21T02:26:58Z 2022-12-20T17:07:49Z
GarbageCollection True 2022-12-21T02:26:59Z 2022-12-20T17:07:49Z
<=============================>
Component: vault-core-tm-vault
Release: vault-4.4.1
Status: ReadinessCheckComplete
Conditions:
TYPE STATUS LAST HEARTBEAT TIME LAST TRANSITION TIME REASON MESSAGE
Initialization True 2022-12-21T02:27:06Z 2022-12-20T11:55:31Z
Deployment True 2022-12-21T02:27:08Z 2022-12-20T11:59:01Z
ReadinessCheck True 2022-12-21T02:27:08Z 2022-12-20T11:59:26Z
GarbageCollection True 2022-12-21T02:27:08Z 2022-12-20T11:59:29Z
<=============================>
Component: webhook-operator
Release: vault-4.4.1
Status: ReadinessCheckComplete
Conditions:
TYPE STATUS LAST HEARTBEAT TIME LAST TRANSITION TIME REASON MESSAGE
Initialization True 2022-12-21T02:26:58Z 2022-12-14T10:29:07Z
Deployment True 2022-12-21T02:26:58Z 2022-12-14T10:29:10Z
ReadinessCheck True 2022-12-21T02:26:58Z 2022-12-14T10:29:17Z
GarbageCollection True 2022-12-21T02:26:58Z 2022-12-14T10:29:17Z
<=============================>
Controller Status:
tmcomponent-operator
Status: Ready
crown-operator
Status: Ready
TM is mainly configured manually, for now no Terraform or any other form of automation.
Example - Ops Dashboard roles: AdminRole present by default, if additional roles are needed, they have to be created manually (roles Developer, ViewOnly were created for testing Dev environment)
Step 9: Deploy smart contracts and workflows
Smart contracts and workflows can be deployed using CLU utility + scripting/pipelines.
https://github.com/SafiBank/SaFiMono/tree/main/tm-contracts/deploy
image-20221102-040909.png (image/png)