I am writing a series of blog posts about troubleshooting Kubernetes. One of the reasons why Kubernetes is so complex is because troubleshooting requires many levels of information gathering. It’s like trying to find the other end of a string in a tangled string ball.

Today, I got this alert in my Slack channel and I have no idea what this means.

Annotations message: There are 2 different versions of Kubernetes components running.

Let’s dig into it.

I am using the Helm Chart Prometheus Operator. This gives me a very good all-in-one Prometheus/Grafana/Alerting solution.

This alert is one of the default alerts that comes in the package.

The alert sends me to this run book:

- runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeversionmismatch

However, this doesn’t really tell me anything besides the name of the alert.

Let’s look at the alert it self and the query in the Prometheus UI:

alert: KubeVersionMismatch expr: count(count by(gitVersion) (kubernetes_build_info{job!="kube-dns"})) > 1

I still only seem to know that there are some different versions of Kubernetes running but I am not sure exactly what it is talking about.

Here is the query:

count(count
  by(gitVersion) (kubernetes_build_info{job!="kube-dns"})) > 1

Let’s run this query ourselves and get more details of what it returns

query

Ah right, it returns a number. We can eliminate most of the query to see what the underlying data is. Change the query to:

kubernetes_build_info{job!="kube-dns"}

This would give you:

query

Yup, per the query the data does show that there are different gitVersion being returned.

However, what does this mean?

Looking at the data some more you will notice this key/value

job="apiserver"

This is information on the job apiserver. This is the Kubernetes API. It is saying currently there are 2 different versions of Kubernetes API server running in this cluster. This might or might not be a concern. If you are upgrading your Kubernetes cluster, then this is not a concern, because Kubernetes can work in a multi version state. You probably shouldn’t leave it at this state but it will be fine. If you are not upgrading your server and no changes are planned, then you will definitely want to investigate this.

As it turns out, I am running on GKE and it is automatically upgrading the k8s master for me.

Need personalized help?

ManagedKube provides DevOps consulting services that help you leverage the power of Docker/Kubernetes in building highly resilient, secure, and scalable fully automated CI/CD workflows.

Schedule a free 15 minute consultation today by e-mailing us: consulting@managedkube.com

More troubleshooting blog posts

Contact me if you have any questions about this or want to chat, happy to start a dialog or help out: blogs@managedkube.com {::nomarkdown}

Learn more about integrating Kubernetes apps

ManagedKube Website

{:/nomarkdown}

GKE | Prometheus | Operator | alert | KubeVersionMismatch