SaFi Bank Space : Tyk Pump with Prometheus

What is Tyk Pump?

Traffic analytics are captured by the Gateway nodes and then temporarily stored in Redis. The Tyk Pump is responsible for moving those analytics into a persistent data store, such as MongoDB, where the traffic can be analyzed.

The Tyk Pump is an open source analytics purger that moves the data generated by your Tyk nodes to any back-end. It is primarily used to display your analytics data in the Tyk Dashboard.

What is Prometheus?

Prometheus is an open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.

Tyk expose the following counters:

• tyk_http_status{code, api}

• tyk_http_status_per_path{code, api, path, method}

• tyk_http_status_per_key{code, key}

• tyk_http_status_per_oauth_client{code, client_id}

And the following Histogram for latencies:

• tyk_latency{type, api}

Custom Prometheus metrics

From Pump 1.6+ it's possible to add custom prometheus metrics using the custom_metrics configuration. For example:

"prometheus": {
  "type": "prometheus",
	"meta": {
		"listen_address": "localhost:9090",
		"path": "/metrics",
		"custom_metrics":[
      {
        "name":"tyk_custom_http_status_per_api_name",
        "description":"This is a custom counter",
        "metric_type":"counter",
        "labels":["response_code","api_name"]
      }
    ]
	}
},

This will create a metric for HTTP status code and API name.

There are 2 types of metric_type: counter and histogram.

If you are using histogram, its always going to use the request_time to observe, and you can also set the configuration option buckets where you can define the buckets into which observations are counted. buckets type is an array of float64 and its default value is [1, 2, 5, 7, 10, 15, 20, 25, 30, 40, 50, 60, 70, 80, 90, 100, 200, 300, 400, 500, 1000, 2000, 5000, 10000, 30000, 60000].

The labels configuration determines the label name and value extracted from the analytic record. The available values are: ["host","method", "path", "response_code", "api_key", "time_stamp", "api_version", "api_name", "api_id", "org_id", "oauth_id","request_time", "ip_address"]

JSON / Conf File

{
    ...
    "pumps": {
      "prometheus": {
        "type": "prometheus",
        "meta": {
          "listen_address": "localhost:9090",
          "path": "/metrics"
        }
      }    
    }
}

Env Variables

TYK_PMP_PUMPS_PROMETHEUS_TYPE=prometheus
TYK_PMP_PUMPS_PROMETHEUS_META_ADDR=localhost:9090
TYK_PMP_PUMPS_PROMETHEUS_META_PATH=/metrics
TYK_PMP_PUMPS_PROMETHEUS_META_CUSTOMMETRICS=[]

How is Tyk Pump currently deployed in our Cloud Infrastructure?

In the SafiMono git repo, pump is declared in the devops/argocd/environments/dev/tyk/tykapigw/values.yaml file.

  pump:
    enabled: true
    replicaCount: 1
    image:
        repository: tykio/tyk-pump-docker-pub
        tag: v1.4.0
        pullPolicy: Always
    extraEnvs: 
      - name: TYK_PMP_PUMPS_PROMETHEUS_TYPE
        value: "prometheus"
      - name: TYK_PMP_PUMPS_PROMETHEUS_META_ADDR
        value: ":9090"
      - name: TYK_PMP_PUMPS_PROMETHEUS_META_PATH
        value: "/metrics"

A separate service and servicemonitor objects are declared here deployed via ArgoCD using a kustomize overlay here.

Assuming servicemonitor and service are configured correctly, the target should reflect in Thanos.

https://thanos.dev.safibank.online/targets#pool-serviceMonitor/tyk/pump-tyk-pro/0

How to test and check if API calls via tyk is reflecting in Prometheus?

Since we use thanos-query, we can make a simple get call on one of the apis and the metrics should reflect like so.

Sample get via tyk gateway

Prometheus Metrics scraped from Tyk Pump Analytics