Kubernetes Custom Columns Output Awk

I learned in my Reforge product strategy program about measuring user satisfaction. Using the TARS framework, targeted, adopted, retained and satisfied you can better understand how your product features are resonating or not. With a platform options can be limited and low level components might signal more the prevalence of problems solved versus enjoyment of a feature.

Sidecar Features

In dev.lab you receive some sidecars by default. LinkerD as your service mesh sidecar and Vault as your secret sidecar. In order for our users to connect to a relational database (MSSQL) you have to enable a flag in a helm values.yaml file. The same applies for an eventing sidecar for publishing and consuming messages over a message queue. Out of 29 applications in production 13% use a database and 10% use some form of eventing. The only alternatives are to use the older VM platform and set everything up by hand or use the provided tools.

Kubernetes Custom Columns Output

In order to keep up to date on the metrics I need to leverage a combination of kubectl and awk.

kubectl get pods --all-namespaces -o="custom-columns=NAME:.metadata.name,CONTAINERS:.spec.containers[*].name,LABELS:.metadata.labels"

Which will give you an export of data like below.

NAME	CONTAINERS	LABELS
someproductionworkload1-5d67656bf5-6dhwn	someproductionworkload1,linkerd-proxy,vault-agent	map[app.kubernetes.io/instance:someproductionworkload1 app.kubernetes.io/name:someproductionworkload1 app.i.com/name:someproductionworkload1 app.i.com/version:0.1.1 linkerd.io/control-plane-ns:linkerd linkerd.io/proxy-deployment:someproductionworkload1 linkerd.io/workload-ns:someproductionworkload1 pod-template-hash:5d67656bf5]
someproductionworkload1-5d67656bf5-cl5bx	someproductionworkload1,linkerd-proxy,vault-agent	map[app.kubernetes.io/instance:someproductionworkload1 app.kubernetes.io/name:someproductionworkload1 app.i.com/name:someproductionworkload1 app.i.com/version:0.1.1 linkerd.io/control-plane-ns:linkerd linkerd.io/proxy-deployment:someproductionworkload1 linkerd.io/workload-ns:someproductionworkload1 pod-template-hash:5d67656bf5]
someproductionworkload1-5d67656bf5-n26lr	someproductionworkload1,linkerd-proxy,vault-agent	map[app.kubernetes.io/instance:someproductionworkload1 app.kubernetes.io/name:someproductionworkload1 app.i.com/name:someproductionworkload1 app.i.com/version:0.1.1 linkerd.io/control-plane-ns:linkerd linkerd.io/proxy-deployment:someproductionworkload1 linkerd.io/workload-ns:someproductionworkload1 pod-template-hash:5d67656bf5]
someproductionworkload2-66fc85bb9f-lvsr7	someproductionworkload2,linkerd-proxy,vault-agent	map[app.kubernetes.io/instance:someproductionworkload2 app.kubernetes.io/name:someproductionworkload2 app.i.com/name:someproductionworkload2 app.i.com/version:0.16.1 linkerd.io/control-plane-ns:linkerd linkerd.io/proxy-deployment:someproductionworkload2 linkerd.io/workload-ns:someproductionworkload2 pod-template-hash:66fc85bb9f]
someproductionworkload2-66fc85bb9f-xhmt5	someproductionworkload2,linkerd-proxy,vault-agent	map[app.kubernetes.io/instance:someproductionworkload2 app.kubernetes.io/name:someproductionworkload2 app.i.com/name:someproductionworkload2 app.i.com/version:0.16.1 linkerd.io/control-plane-ns:linkerd linkerd.io/proxy-deployment:someproductionworkload2 linkerd.io/workload-ns:someproductionworkload2 pod-template-hash:66fc85bb9f]
someproductionworkload2-66fc85bb9f-zpsvp	someproductionworkload2,linkerd-proxy,vault-agent	map[app.kubernetes.io/instance:someproductionworkload2 app.kubernetes.io/name:someproductionworkload2 app.i.com/name:someproductionworkload2 app.i.com/version:0.16.1 linkerd.io/control-plane-ns:linkerd linkerd.io/proxy-deployment:someproductionworkload2 linkerd.io/workload-ns:someproductionworkload2 pod-template-hash:66fc85bb9f]

Using Awk to clean up the data

One of the reasons I am exporting labels and not selecting a specific one is because there are lots of workloads in the cluster that are infrasture and not user created applications. The labels are also specific to the application and not something general enough I can select across all namespaces. I get around this by selecting lines that feature the app.i.com label. Another issue with the data is that there are multiple copies of a pod per app and I only need to know about the one app. I keep an associative array to track if I processed that app or not.

Easiest to save the below as a file and run it ala

kubectl get pods --all-namespaces -o="custom-columns=NAME:.metadata.name,CONTAINERS:.spec.containers[*].name,LABELS:.metadata.labels" | awk -f script.awk

{
    if ($5 ~ /app.i.com/){
        sub("app.i.com/name:","",$5);
        if ($5 in projects){
        next;
        }

        split($2,a,",");
        projects[$5];

        for(k in a) {

            if (a[k] != "vault-agent" && a[k] != "linkerd-proxy" && a[k] != $5){
                printf "%s %s\n", $5, a[k];
            }
        }
    }
}

I split the second column of container images by comma and skip the default containers and log if a database (kerberos sidecar) or event consumer is used.

The output should end up being…

app1 kerberos-sidecar
app2 event-consumer

Application Metrics

I keep hand curated product metrics for dev.lab. You can see the results of the metrics I track on my notion page.