Nginx ingress proxies
Observability, Tetrate

Observability 101: What you see is what you get

One of the most repeated pieces of advice for anyone getting started with microservices is to make sure you can see everything that’s going on inside your services. Leverage the power of observability. However, observability is a loaded term – so it’s valuable to understand what that terms mean, and what’s involved.

Read More
SkyWalking service topology
Apache SkyWalking, Observability, Open Source

Observability at scale: SkyWalking it is!

SkyWalking, a top-level Apache project, is the open source APM and observability analysis platform that is solving the problems of 21st-century systems that are increasingly large, distributed, and heterogenous. It’s built for the struggles system admins face today: To identify and locate needles in a haystack of interdependent services, to get apples-to-apples metrics across polyglot apps, and to get a complete and meaningful view of performance. 

SkyWalking is a holistic platform that can observe microservices on or off a mesh, and can provide consistent monitoring with a lightweight payload.

Let’s take a look at how SkyWalking evolved to address the problem of observability at scale, and grew from a pure tracing system to a feature-rich observability platform that is now used to analyze deployments that collect tens of billions of traces per day. 

Read More
Prometheus Service Monitor
Istio, Observability, Open Source, Service Mesh, Tetrate

How to configure Prometheus-Operator & scrape metrics from Istio 1.6

How do you get Istio metrics in Prometheus

The way that observability metrics are created, exchanged, and scraped has changed for Istio versions Istio 1.4 and up.

Here is how I configure Prometheus-Operator resources to scrape metrics from Istio 1.6 and install the latest Grafana Dashboards.

Install Prometheus and create a ServiceMonitor for Prometheus-operator

Prometheus-Operator is far more dynamic than the default Prometheus install. It adds some CRD to dynamically and transparently re-configure your Prometheus cluster.

A ServiceMonitor is a resource describing which pods to scrape based on a Service.

In Istio 1.6+ we have two types of things to monitor: Istio control-plane resources and Istio-proxy data-plane.

For that we create 2 different ServiceMonitor resources:

Configuring Prometheus operator to monitor Istio control plane

kind: ServiceMonitor
  name: prometheus-oper-istio-controlplane
    release: prometheus
  jobLabel: istio
      - {key: istio, operator: In, values: [mixer,pilot,galley,citadel,sidecar-injector]}
    any: true
  - port: http-monitoring
    interval: 15s
  - port: http-policy-monitoring
    interval: 15s

If you know a bit of Prometheus, this is pretty easy to read:

  • look for any service with label istio equals to mixer , pilot
  • scrape port named http-monitoring and http-policy-monitoring every 15s

The only thing to be careful about are the labels at the beginning: they are selectors that MUST match the Prometheus install serviceMonitorSelector. If you fail to do so, Prometheus will not consider this resource.
You can check how yours is configured by looking at the prometheus resource:

kubectl get prometheus  -o yaml | grep -A4 serviceMonitorSelector    
        release: prometheus

In my case, it is release: prometheus

As you can see from my example, this Prom Operator was installed using Helm. I know… sorry…

Istio and Envoy Whitepaper

Configuring Prometheus operator to monitor Istio data plane

The Data-Plane resource is quite the same but is targeting all the Istio-Proxy containers and adds some relabeling:

kind: ServiceMonitor
  name: prometheus-oper-istio-dataplane
    monitoring: istio-dataplane
    release: prometheus
      - {key: istio-prometheus-ignore, operator: DoesNotExist}
    any: true
  jobLabel: envoy-stats
  - path: /stats/prometheus
    targetPort: http-envoy-prom
    interval: 15s
    - sourceLabels: [__meta_kubernetes_pod_container_port_name]
      action: keep
      regex: '.*-envoy-prom'
    - action: labelmap
      regex: "__meta_kubernetes_pod_label_(.+)"
    - sourceLabels: [__meta_kubernetes_namespace]
      action: replace
      targetLabel: namespace
    - sourceLabels: [__meta_kubernetes_pod_name]
      action: replace
      targetLabel: pod_name

Again, pure Prom config. Just make sure you have the right label so the Operator will take care of the resource.

Add a label istio-prometheus-ignore=”true” to your deployments in case you don’t want Prometheus to scrape the proxy’s metrics.

Service discovery result

After few seconds for the whole thing to settle, you can connect to your Prom frontend, using Port-Forward on port 9090 or using the Istio Ingress-Gateway that you configured with SSL cert using SDS (check my older posts).

Prometheus Service Discovery

How to enable Istio dashboard in Grafana

Now that you have Istio Telemetry V2 into your Prometheus cluster, you maybe want to see the graphs with Grafana.

Glad you’ve read this far. I know this blog is missing some pictures and colors… but who cares, we are engineers right?

Istio Dashboards for Grafana are stored in many places. You can find the latest in the Istio Github repo, but the best solution for you is to grab the one that matches your Istio install from the Istio install zip (or tar) where you grabbed istioctl !

From Istio docs, get it with:

curl -L | sh -

This will create a folder with all the Istio stuffs. Note that Addons (Grafana, Kiali, Prometheus..) will NOT be managed by istioctl quite soon. You can find all the deployment scripts in this folder.

Dashboards are also located in this folder (istio-1.6.7 as the time of this writing) at manifests/charts/istio-telemetry/grafana/dashboards/

For them to be used by Grafana (the one installed by Prom Operator), you need to copy them inside a secret. Here’s the script I use for that (do a cd istio-<your-version>before using it):


# go into the dashboards folder
pushd manifests/charts/istio-telemetry/grafana/dashboards

# create the basic command to create the configmap
ISTIO_DASHBOARD_SECRET="kubectl -n monitoring create cm prometheus-oper-istio-dashboards "

# append each file to the secret
for i in *.json ; do
  echo $i

# run the secret creation command

# label the configmap so it is used by Grafana
kubectl label -n monitoring --overwrite cm prometheus-oper-istio-dashboards grafana_dashboard=1


Restart the Grafana pod and you should see the Dashboards in Grafana:

Istio Telemetry from Prometheus in Grafana dashboard

Sebastien ThomasSebastien Thomas is a Tetrate engineer and infrastructure MacGyver with special expertise in Istio setup and support. Tetrate offers Istio with support for enterprise as well as a service mesh platform powered by Istio and Envoy to create the best fabric for securing, observing, and managing microservices.
Sign up to hear more of the latest on Istio and getting the most out of your mesh.
Read More
Envoy's Postgres filter - Network observability
Envoy proxy & GetEnvoy, Observability, Open Source

How to use Envoy’s Postgres filter for network observability

Starting with release 1.15.0 Envoy proxy supports decoding of Postgres messages for statistics purposes. This feature allows for an aggregated view of the types of Postgres transactions happening in the network. That aggregated view instantly provides a breakdown of types of Postgres operations and the number and severity of errors. Presented in a time series format allows for a clear overview of how the error rate of composition of queries changed over time.

Read More
Apache SkyWalking, Observability, Open Source

What’s new in SkyWalking? MeterSystem and mesh monitoring in 8.0

Apache SkyWalking, the observability platform and open source application performance monitor (APM) project, today announced the general availability of its 8.0 release. Known for its powerful metrics, tracing and service mesh capabilities, SkyWalking extends its functionality with the new release by addressing user demand for metrics integration with other metrics collection systems, including Prometheus.

Read More