Monitoring apps on kubernetes from Prometheus running outside cluster

Prometheus has in a way become a de-facto monitoring solution today. It is free, opensource, has huge community support and almost every opensource app or framework out there has support for Prometheus.

When we want to monitor applications hosted on Kubernetes using Prometheus, we have to run Prometheus inside the cluster in order to pull metrics from pods, for obvious reasons of no reachability directly to pod bypassing ingress from outside Kubernetes cluster.

What if you have Prometheus running outside Kubernetes cluster? We have way of federating metrics using Prometheus federation which allows pulling selected metrics from other Prometheus servers. Still you have to run one Prometheus inside cluster. It is also good idea to have monitor running in separate environment than the environment it is monitoring.

To solve this problem, we built k8s-prometheus-proxy. It’s an application that runs inside kubernetes cluster and supports two modes of pulling metrics from application pods running in kubernetes cluster.

Metrics from all pods
The proxy allows pulling metrics from all running pods of a particular application given namespace, pod name prefix and metrics uri.

It does following steps :

1. Based on configured token, make kubernetes api call to master to get all pods for given namespace
2. Filter running pods based on pod prefix
3. For all filtered pods, makes http call on metrics URI to pods ips and port specified in incoming request
4. Aggregate response from all pods, add pop ip and pod name as additional labels to each metric
5. Return the aggregated response

Kubernetes service discovery
If the kubernetes masters are accessible from outside the cluster, (which is the case when you want to invoke kubernetes apis) this proxy also supports using Prometheus’s in-built Kubernetes service discovery to pull metrics from application pods. It uses Prometheus’s very powerful feature of relabeling to achieve this.

In the prometheus job configuration that uses kubernetes service discovery we do following relable configs:
1. we change the __address__ label to the proxy hostname so that prometheus makes the metrics call to the proxy instead of pod ip.
2. we add __target_pod as the lable so that pod ip gets passed as http get parameter to proxy

You can find more details in the github repository k8s-prometheus-proxy.