DEPLOYING PROMETHEUS STACK WITH HELM
First, we use the community-maintained Helm chart for deploying Prometheus Operator (kube-prometheus-stack) to Kubernetes.
By default, the Helm chart will also deploy and configure an instance of Prometheus.
Let's create a plain GKE cluster like below and add the helm repo
[main][~/external/learn-prometheus]$ k get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME gke-testthanos-default-pool-2d40448a-70l6 Ready <none> 112m v1.23.14-gke.1800 10.148.0.13 35.240.183.120 Container-Optimized OS from Google 5.10.147+ containerd://1.5.13 gke-testthanos-default-pool-2d40448a-qqcd Ready <none> 112m v1.23.14-gke.1800 10.148.0.14 35.187.236.0 Container-Optimized OS from Google 5.10.147+ containerd://1.5.13 gke-testthanos-default-pool-2d40448a-w004 Ready <none> 112m v1.23.14-gke.1800 10.148.0.12 35.240.206.4 Container-Optimized OS from Google 5.10.147+ containerd://1.5.13 [main][~/external/learn-prometheus]$ k get namespaces NAME STATUS AGE default Active 114m kube-node-lease Active 114m kube-public Active 114m kube-system Active 114m [main][~/external/learn-prometheus]$ [main][~/external/learn-prometheus]$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts "prometheus-community" has been added to your repositories [main][~/external/learn-prometheus]$
install the prometheus-operator-crds helm chart.
*[main][~/external/learn-prometheus]$ kubectl create namespace prometheus namespace/prometheus created *[main][~/external/learn-prometheus]$ helm upgrade --install prometheus-operator-crds prometheus-community/prometheus-operator-crds --namespace prometheus Release "prometheus-operator-crds" does not exist. Installing it now. NAME: prometheus-operator-crds LAST DEPLOYED: Mon Feb 13 09:49:05 2023 NAMESPACE: prometheus STATUS: deployed REVISION: 1 TEST SUITE: None *[main][~/external/learn-prometheus]$
The list of CRDs deployed in the GKE cluster now is
*[main][~/external/learn-prometheus]$ kubectl get customresourcedefinitions NAME CREATED AT alertmanagerconfigs.monitoring.coreos.com 2023-02-13T03:11:07Z alertmanagers.monitoring.coreos.com 2023-02-13T03:11:08Z backendconfigs.cloud.google.com 2023-02-13T03:01:26Z capacityrequests.internal.autoscaling.gke.io 2023-02-13T03:01:04Z frontendconfigs.networking.gke.io 2023-02-13T03:01:28Z managedcertificates.networking.gke.io 2023-02-13T03:01:19Z memberships.hub.gke.io 2023-02-13T03:05:10Z podmonitors.monitoring.coreos.com 2023-02-13T03:11:07Z probes.monitoring.coreos.com 2023-02-13T03:11:07Z prometheuses.monitoring.coreos.com 2023-02-13T03:11:08Z prometheusrules.monitoring.coreos.com 2023-02-13T03:11:07Z serviceattachments.networking.gke.io 2023-02-13T03:01:29Z servicemonitors.monitoring.coreos.com 2023-02-13T03:11:07Z servicenetworkendpointgroups.networking.gke.io 2023-02-13T03:01:28Z storagestates.migration.k8s.io 2023-02-13T03:01:30Z storageversionmigrations.migration.k8s.io 2023-02-13T03:01:30Z thanosrulers.monitoring.coreos.com 2023-02-13T03:11:07Z updateinfos.nodemanagement.gke.io 2023-02-13T03:01:33Z volumesnapshotclasses.snapshot.storage.k8s.io 2023-02-13T03:01:29Z volumesnapshotcontents.snapshot.storage.k8s.io 2023-02-13T03:01:29Z volumesnapshots.snapshot.storage.k8s.io 2023-02-13T03:01:30Z *[main][~/external/learn-prometheus]
Install the kube-prometheus-stack helm chart
*[main][~/external/learn-prometheus]$ helm upgrade --install kube-stack-prometheus prometheus-community/kube-prometheus-stack --namespace prometheus Release "kube-stack-prometheus" does not exist. Installing it now. NAME: kube-stack-prometheus LAST DEPLOYED: Mon Feb 13 11:38:18 2023 NAMESPACE: prometheus STATUS: deployed REVISION: 1 NOTES: kube-prometheus-stack has been installed. Check its status by running: kubectl --namespace prometheus get pods -l "release=kube-stack-prometheus" Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator. *[main][~/external/learn-prometheus]$ *[main][~/external/learn-prometheus]$ k get all -n prometheus NAME READY STATUS RESTARTS AGE pod/alertmanager-kube-stack-prometheus-kube-alertmanager-0 2/2 Running 1 (3m29s ago) 3m34s pod/kube-stack-prometheus-grafana-b8df47c57-p4q4f 3/3 Running 0 3m36s pod/kube-stack-prometheus-kube-operator-db4dc45c8-2qw6c 1/1 Running 0 3m36s pod/kube-stack-prometheus-kube-state-metrics-84dcfd67bc-24qcd 1/1 Running 0 3m36s pod/kube-stack-prometheus-prometheus-node-exporter-72d5c 1/1 Running 0 3m36s pod/kube-stack-prometheus-prometheus-node-exporter-csdsh 1/1 Running 0 3m36s pod/kube-stack-prometheus-prometheus-node-exporter-mlsdd 1/1 Running 0 3m36s pod/prometheus-kube-stack-prometheus-kube-prometheus-0 2/2 Running 0 3m33s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 3m35s service/kube-stack-prometheus-grafana ClusterIP 10.72.6.83 <none> 80/TCP 3m38s service/kube-stack-prometheus-kube-alertmanager ClusterIP 10.72.4.55 <none> 9093/TCP 3m38s service/kube-stack-prometheus-kube-operator ClusterIP 10.72.9.86 <none> 443/TCP 3m38s service/kube-stack-prometheus-kube-prometheus ClusterIP 10.72.13.175 <none> 9090/TCP 3m38s service/kube-stack-prometheus-kube-state-metrics ClusterIP 10.72.15.126 <none> 8080/TCP 3m38s service/kube-stack-prometheus-prometheus-node-exporter ClusterIP 10.72.8.215 <none> 9100/TCP 3m38s service/prometheus-operated ClusterIP None <none> 9090/TCP 3m34s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/kube-stack-prometheus-prometheus-node-exporter 3 3 3 3 3 <none> 3m37s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kube-stack-prometheus-grafana 1/1 1 1 3m37s deployment.apps/kube-stack-prometheus-kube-operator 1/1 1 1 3m37s deployment.apps/kube-stack-prometheus-kube-state-metrics 1/1 1 1 3m37s NAME DESIRED CURRENT READY AGE replicaset.apps/kube-stack-prometheus-grafana-b8df47c57 1 1 1 3m37s replicaset.apps/kube-stack-prometheus-kube-operator-db4dc45c8 1 1 1 3m37s replicaset.apps/kube-stack-prometheus-kube-state-metrics-84dcfd67bc 1 1 1 3m37s NAME READY AGE statefulset.apps/alertmanager-kube-stack-prometheus-kube-alertmanager 1/1 3m36s statefulset.apps/prometheus-kube-stack-prometheus-kube-prometheus 1/1 3m35s *[main][~/external/learn-prometheus]$
Port-forward prometheus service
*[main][~/external/learn-prometheus]$ kubectl port-forward --namespace prometheus svc/kube-stack-prometheus-kube-prometheus 9090:9090 Forwarding from 127.0.0.1:9090 -> 9090 Forwarding from [::1]:9090 -> 9090
Service Monitors
Prometheus uses a pull based model for collecting metrics from applications and services.
This means the applications and services must expose a HTTP(S) endpoint containing Prometheus formatted metrics. Prometheus will then, as per its configuration, periodically scrape metrics from these HTTP(S) endpoints.
The Prometheus Operator includes a Custom Resource Definition that allows the definition of the ServiceMonitor. ServiceMonitor is used to define an application you wish to scrape metrics from within Kubernetes. The k8s-controller will action the ServiceMonitors we define and automatically build the required Prometheus configuration
[~]$ k get servicemonitor kube-stack-prometheus-kube-prometheus -n prometheus -o yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
annotations:
meta.helm.sh/release-name: kube-stack-prometheus
meta.helm.sh/release-namespace: prometheus
creationTimestamp: "2023-02-13T03:39:24Z"
generation: 1
labels:
app: kube-prometheus-stack-prometheus
app.kubernetes.io/instance: kube-stack-prometheus
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/part-of: kube-prometheus-stack
app.kubernetes.io/version: 45.0.0
chart: kube-prometheus-stack-45.0.0
heritage: Helm
release: kube-stack-prometheus
name: kube-stack-prometheus-kube-prometheus
namespace: prometheus
resourceVersion: "18805"
uid: 7e905253-b9cc-4d5a-af1d-6f361ac3e242
spec:
endpoints:
- path: /metrics
port: http-web
namespaceSelector:
matchNames:
- prometheus
selector:
matchLabels:
app: kube-prometheus-stack-prometheus
release: kube-stack-prometheus
self-monitor: "true"
[~]$
[~]$ k get servicemonitor -n prometheus NAME AGE kube-stack-prometheus-grafana 18m kube-stack-prometheus-kube-alertmanager 18m kube-stack-prometheus-kube-apiserver 18m kube-stack-prometheus-kube-coredns 18m kube-stack-prometheus-kube-kube-controller-manager 18m kube-stack-prometheus-kube-kube-etcd 18m kube-stack-prometheus-kube-kube-proxy 18m kube-stack-prometheus-kube-kube-scheduler 18m kube-stack-prometheus-kube-kubelet 18m kube-stack-prometheus-kube-operator 18m kube-stack-prometheus-kube-prometheus 18m kube-stack-prometheus-kube-state-metrics 18m kube-stack-prometheus-prometheus-node-exporter 18m [~]$
port-forward grafana service
[~]$ kubectl get secret --namespace prometheus kube-stack-prometheus-grafana -o jsonpath='{.data.admin-user}' | base64 -d
admin
[~]$
[~]$ kubectl get secret --namespace prometheus kube-stack-prometheus-grafana -o jsonpath='{.data.admin-password}' | base64 -d
prom-operator
[~]$
[~]$ kubectl port-forward --namespace prometheus svc/kube-stack-prometheus-grafana 8080:80
Forwarding from 127.0.0.1:8080 -> 3000
Forwarding from [::1]:8080 -> 3000
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
Thanos
The Thanos Project turns Prometheus into a highly available metrics platform with unlimited metrics storage.
The three key features of Thanos, are as follows:
Global query view of all metrics from as many Prometheus instances as you require.
Long-term storage of metrics.
High availability of Prometheus.
High Availability with Thanos
At a high level, HA for Prometheus with Thanos works as detailed below:
First, a sidecar is deployed alongside the Prometheus container and interacts with Prometheus. A sidecar is an additional container within the Kubernetes Pod running alongside other containers.
Next, an additional service is deployed called Thanos Query and is configured to be aware of all instances of the Thanos Sidecar.
Thanos Query communicates with the Thanos Sidecar via gRPC and de-duplicates metrics across all instances of Prometheus when executing a query. Query exposes users to a Prometheus-esuqe user interface and also exposes the Prometheus API.
The diagram below shows this:
Implementation:
thanos-sidecar:
Step-1: First create gcp service account to access the gcs bucket
thanos-sidecar-secret.yaml
type: GCS
config:
bucket: thanospoc-safibank-test
service_account: |-
{
"type": "service_account",
"project_id": "safi-sandbox-firebasepoc",
"private_key_id": "ee4b4780ef932cddcf532a02746df6ae69b6f846",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQClPZD9JXv7yVJW\nrEA5j7IPkb586AcKvve56Q+34cVZymjfRWP078EhDznbKn3elQMiuUqEdZc7Ao6w\nRR5DUdVGbgI7SQQbygrJSOnPnlAe0XSJX4MApPUhUwx6nvr6pV8LqnGCvi9LX/U7\nZiGao/ljaOYqZAvVGOMxfQZRZtev1uOGWGBFMDzIHavjgIojK/kU91/Hm9d2KPbd\ngVhxJyiBCCz1UnGUJfqNgXfecByA+DSBF739zMnM3Ry2xTkheugPtpvL05V9dcyK\ny1xiRKiZp6usDUveb5MOPm3IRiejlHgDM2UnWd/bVsdotVJGu6aftXqLn++XYKZe\noSAtJ9gbAgMBAAECggEACzIrN0PJbUt8RTwhk+TdGzn0TrFH/N3omhOlCYAGJ6nT\nxkEE3005aDyIXoW4BpNRJZKpbmUrTIfFVEMypDwQt9dvVkYBvcJITdAfa35eDmCB\nxWcP0MTciRf/8fUAq2f2EyvGyiWJvdEwJAitATXYCXjLk5a4yIUJq8o9Wdmehp7s\nVWjy4hb0ATstFGeQfO7Zg5GUHiN5L2mlbMKf5pZrtxOzWZom5pBqKpzN7KLqk85C\n2R4/LZ3U+zwxIwinjLXsjssqhx9jU4ury+/zD1Wq5Yu0qmRAyo40pt1EA0iogaqd\nW45cUzOOMe2W7DI+B4HrK/Y3CLOlxqIAeCtLVEFqkQKBgQDc34AYlsF+ycX3GuN+\ncvpau9NJ6SvPqvZnhwvabeSUIgA/zZUnOYe+R1LBQsdQ5Et+KJH57lvVO2XoE3jd\nwdNOs7il7VB/gyV+7nufdpxR3C2mGwtm0/UlngZf14wKWmpLsrcDIpg5K/BC83nz\nYUm23hz6muRiN17gkqkFRdFRUQKBgQC/hRLk85g3e5RO94/g6UcgIC0gO8xl1FDs\nMLkxul7XC3/Q8kSRmS4l9ioR/RH3s81Vf7DmkooV6mHwkVK4+mRVYEYtxjAoV/B3\nmmxyyyh0iJNnp378t7+9tqxmzoeuhOnGfhqrx5hvvpqdJR25mJH68oxOicHWZyPA\nimeS33RXqwKBgQC5NaUpeUj582JQaji3cNykNvne8WpucjK671AuMpZI7RX+anqW\n8LP2B8LKEiAmobj4Kjb2ODzDWlKCv8lLUtvJFAVTBePJeQR5IquBNSmxrwqMTgNK\nj5PynkdEGO/yQoz9uveh5mF1F3YZ/iaVwk/YyvnL3xSBk2K4t+KJbFkwcQKBgCGV\nUC7dmD+CaAtA1fX2cgZsHf2c/saMaHRCRvc1WxQSbCNTmDp/xdvaQp5M5i2tVcA8\nV+TWpWgaZXotwXe2UIm4DBZSvncUlJIkfgfTIBDvMgnnERl3jyPROIH+jURJAkVQ\nL5UXtt9BaRNNQt0cp9x9U6Uh+t+Bzn8gxfC3RktpAoGBANLjFu8lHWRmvidUX7tc\njI54izAyzLEZ9X+cn5dVHCuGHUQ0hXEBRKutQt92/fs4m59gkGtj/I6Dl4TizqKC\n3EyadpzzDVPRDb4uIC0AelhdGmZraWSmfLFLFq9JF4JZfEazM1FTFGOuYSrRVd8V\nHUWrgkyl4cGS8VyOLNYHyDwn\n-----END PRIVATE KEY-----\n",
"client_email": "thanos-sidecar-gcs@safi-sandbox-firebasepoc.iam.gserviceaccount.com",
"client_id": "103490001705356959728",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/thanos-sidecar-gcs%40safi-sandbox-firebasepoc.iam.gserviceaccount.com"
}
*[main][~/external/learn-prometheus/thanos]$ kubectl create secret generic thanos-gcp-config --from-file=thanos.yaml=thanos-sidecar-secret.yaml secret/thanos-gcp-config created *[main][~/external/learn-prometheus/thanos]$
update prometheusSpec to update thanos Image as a sidecar container
---
prometheus:
prometheusSpec:
externalLabels:
cloud: gcp
cluster: prod
replicas: 3
thanos:
image: "quay.io/thanos/thanos:v0.30.2"
objectStorageConfig:
key: thanos.yaml
name: thanos-gcp-config
version: v0.30.2
thanosService:
enabled: true
Now deploy the helm chart another revision by updating the thanos image.
*[main][~/external/learn-prometheus/thanos]$ helm upgrade --install kube-stack-prometheus prometheus-community/kube-prometheus-stack --namespace prometheus --values thanos-prometheus-values.yaml Release "kube-stack-prometheus" has been upgraded. Happy Helming! NAME: kube-stack-prometheus LAST DEPLOYED: Mon Feb 13 15:30:10 2023 NAMESPACE: prometheus STATUS: deployed REVISION: 2 NOTES: kube-prometheus-stack has been installed. Check its status by running: kubectl --namespace prometheus get pods -l "release=kube-stack-prometheus" Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator. *[main][~/external/learn-prometheus/thanos]$
[~]$ k get pods NAME READY STATUS RESTARTS AGE alertmanager-kube-stack-prometheus-kube-alertmanager-0 2/2 Running 1 (3h54m ago) 3h54m kube-stack-prometheus-grafana-b8df47c57-p4q4f 3/3 Running 0 3h54m kube-stack-prometheus-kube-operator-db4dc45c8-2qw6c 1/1 Running 0 3h54m kube-stack-prometheus-kube-state-metrics-84dcfd67bc-24qcd 1/1 Running 0 3h54m kube-stack-prometheus-prometheus-node-exporter-72d5c 1/1 Running 0 3h54m kube-stack-prometheus-prometheus-node-exporter-csdsh 1/1 Running 0 3h54m kube-stack-prometheus-prometheus-node-exporter-mlsdd 1/1 Running 0 3h54m prometheus-kube-stack-prometheus-kube-prometheus-0 3/3 Running 0 64s prometheus-kube-stack-prometheus-kube-prometheus-1 3/3 Running 0 81s prometheus-kube-stack-prometheus-kube-prometheus-2 3/3 Running 0 81s [~]$
[~]$ k logs prometheus-kube-stack-prometheus-kube-prometheus-1 -c thanos-sidecar
level=info ts=2023-02-13T07:32:42.264774269Z caller=options.go:26 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2023-02-13T07:32:42.265692868Z caller=factory.go:52 msg="loading bucket configuration"
level=info ts=2023-02-13T07:32:42.266647234Z caller=sidecar.go:362 msg="starting sidecar"
level=info ts=2023-02-13T07:32:42.266874749Z caller=reloader.go:199 component=reloader msg="nothing to be watched"
level=info ts=2023-02-13T07:32:42.267000159Z caller=intrumentation.go:56 msg="changing probe status" status=ready
level=info ts=2023-02-13T07:32:42.267154438Z caller=intrumentation.go:75 msg="changing probe status" status=healthy
level=info ts=2023-02-13T07:32:42.26717809Z caller=http.go:73 service=http/server component=sidecar msg="listening for requests and metrics" address=:10902
level=info ts=2023-02-13T07:32:42.267306496Z caller=grpc.go:131 service=gRPC/server component=sidecar msg="listening for serving gRPC" address=:10901
level=info ts=2023-02-13T07:32:42.267374121Z caller=tls_config.go:232 service=http/server component=sidecar msg="Listening on" address=[::]:10902
level=info ts=2023-02-13T07:32:42.26743407Z caller=tls_config.go:235 service=http/server component=sidecar msg="TLS is disabled." http2=false address=[::]:10902
level=info ts=2023-02-13T07:32:42.271102531Z caller=sidecar.go:179 msg="successfully loaded prometheus version"
level=info ts=2023-02-13T07:32:42.289551724Z caller=sidecar.go:201 msg="successfully loaded prometheus external labels" external_labels="{cloud=\"gcp\", cluster=\"test\", prometheus=\"prometheus/kube-stack-prometheus-kube-prometheus\", prometheus_replica=\"prometheus-kube-stack-prometheus-kube-prometheus-1\"}"
level=warn ts=2023-02-13T07:32:44.267292444Z caller=shipper.go:239 msg="reading meta file failed, will override it" err="failed to read /prometheus/thanos.shipper.json: open /prometheus/thanos.shipper.json: no such file or directory"
[~]$
Now install the Thanos Cluster using Helm Chart
sample values file as below
objstoreConfig: |-
type: GCS
config:
bucket: thanospoc-safibank-test
service_account: |-
{
"type": "service_account",
"project_id": "safi-sandbox-firebasepoc",
"private_key_id": "ee4b4780ef932cddcf532a02746df6ae69b6f846",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQClPZD9JXv7yVJW\nrEA5j7IPkb586AcKvve56Q+34cVZymjfRWP078EhDznbKn3elQMiuUqEdZc7Ao6w\nRR5DUdVGbgI7SQQbygrJSOnPnlAe0XSJX4MApPUhUwx6nvr6pV8LqnGCvi9LX/U7\nZiGao/ljaOYqZAvVGOMxfQZRZtev1uOGWGBFMDzIHavjgIojK/kU91/Hm9d2KPbd\ngVhxJyiBCCz1UnGUJfqNgXfecByA+DSBF739zMnM3Ry2xTkheugPtpvL05V9dcyK\ny1xiRKiZp6usDUveb5MOPm3IRiejlHgDM2UnWd/bVsdotVJGu6aftXqLn++XYKZe\noSAtJ9gbAgMBAAECggEACzIrN0PJbUt8RTwhk+TdGzn0TrFH/N3omhOlCYAGJ6nT\nxkEE3005aDyIXoW4BpNRJZKpbmUrTIfFVEMypDwQt9dvVkYBvcJITdAfa35eDmCB\nxWcP0MTciRf/8fUAq2f2EyvGyiWJvdEwJAitATXYCXjLk5a4yIUJq8o9Wdmehp7s\nVWjy4hb0ATstFGeQfO7Zg5GUHiN5L2mlbMKf5pZrtxOzWZom5pBqKpzN7KLqk85C\n2R4/LZ3U+zwxIwinjLXsjssqhx9jU4ury+/zD1Wq5Yu0qmRAyo40pt1EA0iogaqd\nW45cUzOOMe2W7DI+B4HrK/Y3CLOlxqIAeCtLVEFqkQKBgQDc34AYlsF+ycX3GuN+\ncvpau9NJ6SvPqvZnhwvabeSUIgA/zZUnOYe+R1LBQsdQ5Et+KJH57lvVO2XoE3jd\nwdNOs7il7VB/gyV+7nufdpxR3C2mGwtm0/UlngZf14wKWmpLsrcDIpg5K/BC83nz\nYUm23hz6muRiN17gkqkFRdFRUQKBgQC/hRLk85g3e5RO94/g6UcgIC0gO8xl1FDs\nMLkxul7XC3/Q8kSRmS4l9ioR/RH3s81Vf7DmkooV6mHwkVK4+mRVYEYtxjAoV/B3\nmmxyyyh0iJNnp378t7+9tqxmzoeuhOnGfhqrx5hvvpqdJR25mJH68oxOicHWZyPA\nimeS33RXqwKBgQC5NaUpeUj582JQaji3cNykNvne8WpucjK671AuMpZI7RX+anqW\n8LP2B8LKEiAmobj4Kjb2ODzDWlKCv8lLUtvJFAVTBePJeQR5IquBNSmxrwqMTgNK\nj5PynkdEGO/yQoz9uveh5mF1F3YZ/iaVwk/YyvnL3xSBk2K4t+KJbFkwcQKBgCGV\nUC7dmD+CaAtA1fX2cgZsHf2c/saMaHRCRvc1WxQSbCNTmDp/xdvaQp5M5i2tVcA8\nV+TWpWgaZXotwXe2UIm4DBZSvncUlJIkfgfTIBDvMgnnERl3jyPROIH+jURJAkVQ\nL5UXtt9BaRNNQt0cp9x9U6Uh+t+Bzn8gxfC3RktpAoGBANLjFu8lHWRmvidUX7tc\njI54izAyzLEZ9X+cn5dVHCuGHUQ0hXEBRKutQt92/fs4m59gkGtj/I6Dl4TizqKC\n3EyadpzzDVPRDb4uIC0AelhdGmZraWSmfLFLFq9JF4JZfEazM1FTFGOuYSrRVd8V\nHUWrgkyl4cGS8VyOLNYHyDwn\n-----END PRIVATE KEY-----\n",
"client_email": "thanos-sidecar-gcs@safi-sandbox-firebasepoc.iam.gserviceaccount.com",
"client_id": "103490001705356959728",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/thanos-sidecar-gcs%40safi-sandbox-firebasepoc.iam.gserviceaccount.com"
}
query:
enabled: true
dnsDiscovery:
sidecarsService: kube-stack-prometheus-kube-thanos-discovery
sidecarsNamespace: prometheus
stores:
- "dnssrv+_grpc._tcp.kube-stack-prometheus-kube-thanos-discovery.prometheus.svc.cluster.local"
queryFrontend:
enabled: true
bucketweb:
enabled: true
refresh: "10m"
compactor:
enabled: true
persistence:
size: "10Gi"
existingClaim: ""
storegateway:
enabled: true
persistence:
size: "10Gi"
existingClaim: ""
ruler:
enabled: false
receive:
enabled: false
metrics:
enabled: true
serviceMonitor:
enabled: true
We use bitnami helm-chart to deploy the thanos-cluster like below.
*[main][~/external/learn-prometheus/thanos]$ helm upgrade --install thanos bitnami/thanos -f bitnami-helm-values.yaml
Release "thanos" has been upgraded. Happy Helming!
NAME: thanos
LAST DEPLOYED: Mon Feb 13 16:23:17 2023
NAMESPACE: prometheus
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
CHART NAME: thanos
CHART VERSION: 12.0.4
APP VERSION: 0.30.2** Please be patient while the chart is being deployed **
Thanos chart was deployed enabling the following components:
- Thanos Query
- Thanos Bucket Web
- Thanos Compactor
- Thanos Store Gateway
Thanos Query can be accessed through following DNS name from within your cluster:
thanos-query.prometheus.svc.cluster.local (port 9090)
To access Thanos Query from outside the cluster execute the following commands:
1. Get the Thanos Query URL by running these commands:
export SERVICE_PORT=$(kubectl get --namespace prometheus -o jsonpath="{.spec.ports[0].port}" services thanos-query)
kubectl port-forward --namespace prometheus svc/thanos-query ${SERVICE_PORT}:${SERVICE_PORT} &
echo "http://127.0.0.1:${SERVICE_PORT}"
2. Open a browser and access Thanos Query using the obtained URL.
*[main][~/external/learn-prometheus/thanos]$ export SERVICE_PORT=$(kubectl get --namespace prometheus -o jsonpath="{.spec.ports[0].port}" services thanos-query)
*[main][~/external/learn-prometheus/thanos]$
*[main][~/external/learn-prometheus/thanos]$ kubectl port-forward --namespace prometheus svc/thanos-query 9091:${SERVICE_PORT}
Forwarding from 127.0.0.1:9091 -> 10902
Forwarding from [::1]:9091 -> 10902
Handling connection for 9091
Handling connection for 9091
Handling connection for 9091
Handling connection for 9091
[~]$ k get po NAME READY STATUS RESTARTS AGE alertmanager-kube-stack-prometheus-kube-alertmanager-0 2/2 Running 1 (4h55m ago) 4h55m kube-stack-prometheus-grafana-b8df47c57-p4q4f 3/3 Running 0 4h55m kube-stack-prometheus-kube-operator-db4dc45c8-2qw6c 1/1 Running 0 4h55m kube-stack-prometheus-kube-state-metrics-84dcfd67bc-24qcd 1/1 Running 0 4h55m kube-stack-prometheus-prometheus-node-exporter-72d5c 1/1 Running 0 4h55m kube-stack-prometheus-prometheus-node-exporter-csdsh 1/1 Running 0 4h55m kube-stack-prometheus-prometheus-node-exporter-mlsdd 1/1 Running 0 4h55m prometheus-kube-stack-prometheus-kube-prometheus-0 3/3 Running 0 62m prometheus-kube-stack-prometheus-kube-prometheus-1 3/3 Running 0 62m prometheus-kube-stack-prometheus-kube-prometheus-2 3/3 Running 0 62m thanos-bucketweb-7dfd44498c-7zrr4 1/1 Running 0 11m thanos-compactor-794754bb76-78vrm 1/1 Running 0 11m thanos-query-68c8cbf5c7-qdzl9 1/1 Running 0 11m thanos-query-frontend-657b5488c8-xsdwg 1/1 Running 0 37m thanos-storegateway-0 1/1 Running 0 10m [~]$ k get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 4h55m kube-stack-prometheus-grafana ClusterIP 10.72.6.83 <none> 80/TCP 4h55m kube-stack-prometheus-kube-alertmanager ClusterIP 10.72.4.55 <none> 9093/TCP 4h55m kube-stack-prometheus-kube-operator ClusterIP 10.72.9.86 <none> 443/TCP 4h55m kube-stack-prometheus-kube-prometheus ClusterIP 10.72.13.175 <none> 9090/TCP 4h55m kube-stack-prometheus-kube-state-metrics ClusterIP 10.72.15.126 <none> 8080/TCP 4h55m kube-stack-prometheus-kube-thanos-discovery ClusterIP None <none> 10901/TCP,10902/TCP 62m kube-stack-prometheus-prometheus-node-exporter ClusterIP 10.72.8.215 <none> 9100/TCP 4h55m prometheus-operated ClusterIP None <none> 9090/TCP,10901/TCP 4h55m thanos-bucketweb ClusterIP 10.72.15.70 <none> 8080/TCP 37m thanos-compactor ClusterIP 10.72.9.229 <none> 9090/TCP 37m thanos-query ClusterIP 10.72.4.160 <none> 9090/TCP 37m thanos-query-frontend ClusterIP 10.72.2.13 <none> 9090/TCP 37m thanos-query-grpc ClusterIP 10.72.3.216 <none> 10901/TCP 37m thanos-storegateway ClusterIP 10.72.8.0 <none> 9090/TCP,10901/TCP 37m [~]$
Attachments:
image-20230213-034918.png (image/png)
image-20230213-035400.png (image/png)
image-20230213-035536.png (image/png)
image-20230213-042021.png (image/png)
image-20230213-082914.png (image/png)
image-20230213-091432.png (image/png)