Helm v3 Debugging with cert-manager
Helm v2 to v3 migration
Must read doc: https://helm.sh/blog/helm-3-released/
Download
Helm v3 binary
https://github.com/helm/helm/releases
The helm-2to3 tool
https://github.com/helm/helm-2to3#install
This will help you migrate your local configs over
Converting an existing chart from v2 to v3
This page documents the changes from v2 to v3: https://helm.sh/docs/topics/v2_v3_migration/
This is based on the ManagedKube/kubernetes-ops repository
Located: https://github.com/ManagedKube/kubernetes-ops/tree/master/kubernetes/helm/cert-manager/cert-manager
All of the values files are in there and the workdir is based the root of this repository
Launching cert-manager
Workdir: ./kubernetes/helm/cert-manager/cert-manager
Apply cert-manager CRDs before installing (or it will fail)
kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.11/deploy/manifests/00-crds.yaml
Install cert-manager
kubectl create ns cert-manager
helm upgrade cert-manager --install --namespace cert-manager -f values.yaml ./ --dry-run
helm list --namespace cert-manager
Install cert-manager-cluster-issuer
Workdir: ./kubernetes/helm/cert-manager/cert-manager-cluster-issuer
helm upgrade cert-manager-cluster-issuer --install --namespace cert-manager -f values.yaml -f environments/gcp-dev/values.yaml ./ --dry-run
Launching cert-manager-cluster-issuer
Workdir: ./kubernetes/helm/cert-manager/cert-manager-cluster-issuer
Template:
helm template cert-manager-cluster-issuer --namespace cert-manager -f values.yaml -f ./environments/gcp-dev/values.yaml ./
helm upgrade cert-manager-cluster-issuer --install --namespace cert-manager -f values.yaml -f ./environments/gcp-dev/values.yaml ./
Deleting
Seems like there are a few steps to delete.
Doc: https://docs.cert-manager.io/en/master/tasks/uninstall/kubernetes.html
Check for CRDs:
kubectl get Issuers,ClusterIssuers,Certificates,CertificateRequests,Orders,Challenges --all-namespaces
Delete CRDs:
kubectl delete -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.11/deploy/manifests/00-crds.yaml
kubectl delete APIService v1beta1.webhook.cert-manager.io
Delete helm chart:
helm --namespace cert-manager delete cert-manager
Debugging
Creating the certificate:
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: my-domain
namespace: my-domain
spec:
secretName: my-domain-tls
issuerRef:
name: issuer-dns01
dnsNames:
- my-domain.example.com
Checkout out what the status of ths certificate is.
kubectl -n my-domain get certificate
NAME READY SECRET AGE
my-domain True my-domain-tls 4m20s
Then we can describe it:
kubectl -n my-domain describe certificate my-domain
Name: my-domain
Namespace: my-domain
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"cert-manager.io/v1alpha2","kind":"Certificate","metadata":{"annotations":{},"name":"my-domain","namespace":"my-domain"},"spec...
API Version: cert-manager.io/v1alpha2
Kind: Certificate
Metadata:
Creation Timestamp: 2019-11-16T03:45:54Z
Generation: 1
Resource Version: 32857679
Self Link: /apis/cert-manager.io/v1alpha2/namespaces/my-domain/certificates/my-domain
UID: 9738f3f8-0823-11ea-a9f7-42010a660008
Spec:
Dns Names:
my-domain.example.com
Issuer Ref:
Kind: ClusterIssuer
Name: issuer-dns01
Secret Name: my-domain-tls
Status:
Conditions:
Last Transition Time: 2019-11-16T03:48:11Z
Message: Certificate is up to date and has not expired
Reason: Ready
Status: True
Type: Ready
Not After: 2020-02-14T02:48:10Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal GeneratedKey 2m48s cert-manager Generated a new private key
Normal Requested 2m48s cert-manager Created new CertificateRequest resource "my-domain-3817824489"
From the Events it tells us that it created a new CertificateRequest named my-domain-3817824489
We can describe that CRD to get the status of the request:
kubectl -n my-domain describe CertificateRequest my-domain-3817824489
Name: my-domain-3817824489
Namespace: my-domain
Labels: <none>
Annotations: cert-manager.io/certificate-name: my-domain
cert-manager.io/private-key-secret-name: my-domain-tls
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"cert-manager.io/v1alpha2","kind":"Certificate","metadata":{"annotations":{},"name":"my-domain","namespace":"my-domain"},"spec...
API Version: cert-manager.io/v1alpha2
Kind: CertificateRequest
Metadata:
Creation Timestamp: 2019-11-16T03:53:41Z
Generation: 1
Owner References:
API Version: cert-manager.io/v1alpha2
Block Owner Deletion: true
Controller: true
Kind: Certificate
Name: my-domain
UID: ad59c00f-0824-11ea-a9f7-42010a660008
Resource Version: 32859582
Self Link: /apis/cert-manager.io/v1alpha2/namespaces/my-domain/certificaterequests/my-domain-3817824489
UID: ad7a55b6-0824-11ea-a9f7-42010a660008
Spec:
Csr: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVR.....
Issuer Ref:
Name: issuer-dns01
Status:
Conditions:
Last Transition Time: 2019-11-16T03:53:41Z
Message: Referenced "Issuer" not found: issuer.cert-manager.io "issuer-dns01" not found
Reason: Pending
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IssuerNotFound 35s (x5 over 35s) cert-manager Referenced "Issuer" not found: issuer.cert-manager.io "issuer-dns01" not found
There seems to be an error: Referenced "Issuer" not found: issuer.cert-manager.io "issuer-dns01" not found
It did not find the Issuer named issuer-dns01
This puzzled me for a bit and had to read some of the cert-manager’s documentation. The problem is that an Issuer is
local to the namespace and we created a ClusterIssuer. A ClusterIssuer can serve the entire cluster with one issuer.
I made a mistake on the Certificate yaml
I was missing the kind: kind: ClusterIssuer
The Certificate definition should be:
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: my-domain
namespace: my-domain
spec:
secretName: my-domain-tls
issuerRef:
kind: ClusterIssuer
name: issuer-dns01
dnsNames:
- my-domain.example.com
Lets apply this again:
kubectl -n my-domain apply -f ~/Downloads/certificate.yaml
certificate.cert-manager.io/my-domain unchanged
Describing the cert:
kubectl -n my-domain describe certificate my-domain
Name: my-domain
Namespace: my-domain
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"cert-manager.io/v1alpha2","kind":"Certificate","metadata":{"annotations":{},"name":"my-domain","namespace":"my-domain"},"spec...
API Version: cert-manager.io/v1alpha2
Kind: Certificate
Metadata:
Creation Timestamp: 2019-11-16T03:59:32Z
Generation: 1
Resource Version: 32861603
Self Link: /apis/cert-manager.io/v1alpha2/namespaces/my-domain/certificates/my-domain
UID: 7f0228fa-0825-11ea-a9f7-42010a660008
Spec:
Dns Names:
my-domain.example.com
Issuer Ref:
Name: issuer-dns01
Secret Name: my-domain-tls
Status:
Conditions:
Last Transition Time: 2019-11-16T03:59:32Z
Message: Waiting for CertificateRequest "my-domain-3817824489" to complete
Reason: InProgress
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Requested 2s cert-manager Created new CertificateRequest resource "my-domain-3817824489"
Describing the CertificateRequest
kubectl -n my-domain describe CertificateRequest my-domain-1647441326
Name: my-domain-1647441326
Namespace: my-domain
Labels: <none>
Annotations: cert-manager.io/certificate-name: my-domain
cert-manager.io/private-key-secret-name: my-domain-tls
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"cert-manager.io/v1alpha2","kind":"Certificate","metadata":{"annotations":{},"name":"my-domain","namespace":"my-domain"},"spec...
API Version: cert-manager.io/v1alpha2
Kind: CertificateRequest
Metadata:
Creation Timestamp: 2019-11-16T04:00:43Z
Generation: 1
Owner References:
API Version: cert-manager.io/v1alpha2
Block Owner Deletion: true
Controller: true
Kind: Certificate
Name: my-domain
UID: 7f0228fa-0825-11ea-a9f7-42010a660008
Resource Version: 32862040
Self Link: /apis/cert-manager.io/v1alpha2/namespaces/my-domain/certificaterequests/my-domain-1647441326
UID: a93e9fea-0825-11ea-a9f7-42010a660008
Spec:
Csr: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSR
Issuer Ref:
Kind: ClusterIssuer
Name: issuer-dns01
Status:
Certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZiVENDQ
Conditions:
Last Transition Time: 2019-11-16T04:00:45Z
Message: Certificate fetched from issuer successfully
Reason: Issued
Status: True
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal OrderCreated 17s cert-manager Created Order resource my-domain/my-domain-1647441326-1246269154
Normal CertificateIssued 15s cert-manager Certificate fetched from issuer successfully
We can also look at the cert-manager’s logs to see what is happening:
kubectl -n cert-manager logs -f cert-manager-67d8bc785c-4w7rf
I1116 03:46:58.236681 1 controller.go:129] cert-manager/controller/challenges "level"=0 "msg"="syncing item" "key"="my-domain/my-domain-1647441326-1246269154-1548188278"
I1116 03:46:58.236752 1 metrics.go:385] cert-manager/metrics "level"=3 "msg"="incrementing controller sync call count" "controllerName"="challenges"
I1116 03:46:58.236979 1 dns.go:121] cert-manager/controller/challenges/Check "level"=0 "msg"="checking DNS propagation" "dnsName"="my-domain.example.com" "domain"="my-domain.example.com" "resource_kind"="Challenge" "resource_name"="my-domain-1647441326-1246269154-1548188278" "resource_namespace"="my-domain" "type"="dns-01" "nameservers"=["10.104.0.10:53"]
I1116 03:46:58.241334 1 wait.go:277] Searching fqdn "_acme-challenge.my-domain.example.com." using seed nameservers [10.104.0.10:53]
I1116 03:46:58.241361 1 wait.go:308] Returning cached zone record "example.com." for fqdn "_acme-challenge.my-domain.example.com."
I1116 03:46:58.244250 1 wait.go:295] Returning authoritative nameservers [ns-cloud-d1.googledomains.com., ns-cloud-d2.googledomains.com., ns-cloud-d3.googledomains.com., ns-cloud-d4.googledomains.com.]
I1116 03:46:58.255396 1 wait.go:123] Looking up TXT records for "_acme-challenge.my-domain.example.com."
I1116 03:46:58.356011 1 wait.go:123] Looking up TXT records for "_acme-challenge.my-domain.example.com."
E1116 03:46:58.356112 1 sync.go:184] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"my-domain.example.com\" not yet propagated" "dnsName"="my-domain.example.com" "resource_kind"="Challenge" "resource_name"="my-domain-1647441326-1246269154-1548188278" "resource_namespace"="my-domain" "type"="dns-01"
I1116 03:46:58.356195 1 controller.go:135] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="my-domain/my-domain-1647441326-1246269154-1548188278"
I1116 03:47:08.356438 1 controller.go:129] cert-manager/controller/challenges "level"=0 "msg"="syncing item" "key"="my-domain/my-domain-1647441326-1246269154-1548188278"
I1116 03:47:08.358285 1 metrics.go:385] cert-manager/metrics "level"=3 "msg"="incrementing controller sync call count" "controllerName"="challenges"
I1116 03:47:08.358756 1 dns.go:121] cert-manager/controller/challenges/Check "level"=0 "msg"="checking DNS propagation" "dnsName"="my-domain.example.com" "domain"="my-domain.example.com" "resource_kind"="Challenge" "resource_name"="my-domain-1647441326-1246269154-1548188278" "resource_namespace"="my-domain" "type"="dns-01" "nameservers"=["10.104.0.10:53"]
I1116 03:47:08.365683 1 wait.go:277] Searching fqdn "_acme-challenge.my-domain.example.com." using seed nameservers [10.104.0.10:53]
I1116 03:47:08.365716 1 wait.go:308] Returning cached zone record "example.com." for fqdn "_acme-challenge.my-domain.example.com."
I1116 03:47:08.370115 1 wait.go:295] Returning authoritative nameservers [ns-cloud-d1.googledomains.com., ns-cloud-d2.googledomains.com., ns-cloud-d3.googledomains.com., ns-cloud-d4.googledomains.com.]
I1116 03:47:08.381055 1 wait.go:123] Looking up TXT records for "_acme-challenge.my-domain.example.com."
I1116 03:47:08.482071 1 wait.go:123] Looking up TXT records for "_acme-challenge.my-domain.example.com."
There is a lot of different things going on in here.
Cert-manager has access to create DNS records and it looks like it created one and searching for it:
I1116 03:46:58.241334 1 wait.go:277] Searching fqdn "_acme-challenge.my-domain.example.com." using seed nameservers
It is querying it and it looks like it currently can’t resolve it and declaring that the DNS has not proprogated yet:
I1116 03:46:58.356011 1 wait.go:123] Looking up TXT records for "_acme-challenge.my-domain.example.com."
E1116 03:46:58.356112 1 sync.go:184] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"my-domain.example.com\" not yet propagated" "dnsName"="my-domain.example.com" "resource_kind"="Challenge"
This makes sense, since adding a DNS entry sometime takes a while to proprogate.
Subsequently it can resolve the DNS and the certificate request is approved by Let’s Encrypt.
I1116 04:00:45.052096 1 sync.go:63] cert-manager/controller/orders "level"=3 "msg"="updated Order resource status successfully" "resource_kind"="Order" "resource_name"="my-domain-1647441326-1246269154" "resource_namespace"="my-domain"
I1116 04:00:45.135225 1 sync.go:63] cert-manager/controller/orders "level"=3 "msg"="updated Order resource status successfully" "resource_kind"="Order" "resource_name"="my-domain-1647441326-1246269154" "resource_namespace"="my-domain"
I1116 04:00:45.619888 1 event.go:255] Event(v1.ObjectReference{Kind:"Order", Namespace:"my-domain", Name:"my-domain-1647441326-1246269154", UID:"a9415afa-0825-11ea-a9f7-42010a660008", APIVersion:"acme.cert-manager.io/v1alpha2", ResourceVersion:"32862033", FieldPath:""}): type: 'Normal' reason: 'Complete' Order completed successfully
I1116 04:00:45.631311 1 sync.go:63] cert-manager/controller/orders "level"=3 "msg"="updated Order resource status successfully" "resource_kind"="Order" "resource_name"="my-domain-1647441326-1246269154" "resource_namespace"="my-domain"
I1116 04:00:45.635073 1 sync.go:159] cert-manager/controller/certificaterequests-issuer-acme/updateStatus "level"=3 "msg"="updating resource due to change in status" "resource_kind"="CertificateRequest" "resource_name"="my-domain-1647441326" "resource_namespace"="my-domain" "diff"=["\"{\\\"conditions\\\":[{\\\"type\\\":\\\"Ready\\\",\\\"status\\\":\\\"False\\\",\\\"lastTransitionTime\\\":\\\"2019-11-16T04:00:43Z\\\",\\\"reason\\\":\\\"Pending\\\",\\\"message\\\":\\\"Waiting on certificate issuance from order my-domain/my-domain-1647441326-1246269154: \\\\\\\"ready\\\\\\\"\\\"}]}\" != \"{\\\"conditions\\\":[{\\\"type\\\":\\\"Ready\\\",\\\"status\\\":\\\"True\\\",\\\"lastTransitionTime\\\":\\\"2019-11-16T04:00:45Z\\\",\\\"reason\\\":\\\"Issued\\\",\\\"message\\\":\\\"Certificate fetched from issuer successfully\\\"}],\\\"certificate\\\":\\\"LS0tLS1CRUdJTiBDRVJUSUZJ\\\"}\""]
I1116 04:00:45.636467 1 event.go:255] Event(v1.ObjectReference{Kind:"CertificateRequest", Namespace:"my-domain", Name:"my-domain-1647441326", UID:"a93e9fea-0825-11ea-a9f7-42010a660008", APIVersion:"cert-manager.io/v1alpha2", ResourceVersion:"32862032", FieldPath:""}): type: 'Normal' reason: 'CertificateIssued' Certificate fetched from issuer successfully
I1116 04:00:45.668671 1 event.go:255] Event(v1.ObjectReference{Kind:"Certificate", Namespace:"my-domain", Name:"my-domain", UID:"7f0228fa-0825-11ea-a9f7-42010a660008", APIVersion:"cert-manager.io/v1alpha2", ResourceVersion:"32862017", FieldPath:""}): type: 'Normal' reason: 'Issued' Certificate issued successfully
The cert-manager retrieves the certificate and places it in the secret.
Migration from Helm v2 to v3
Helm provides a handy tool named https://github.com/helm/helm-2to3#migrate-helm-v2-releases for helping users migrate currently deployed helm releases from Helm v2 to v3.
My current helm version:
helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
My current Helm v2 releases:
helm list
NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
cert-manager 1 Sat Nov 16 07:18:06 2019 DEPLOYED cert-manager-v0.11.0 v0.11.0 cert-manager
~/Downloads/helm-v3.0.0-linux-amd64/linux-amd64/helm 2to3 convert --delete-v2-releases cert-manager --dry-run
2019/11/16 07:24:07 NOTE: This is in dry-run mode, the following actions will not be executed.
2019/11/16 07:24:07 Run without --dry-run to take the actions described below:
2019/11/16 07:24:07
2019/11/16 07:24:07 Release "cert-manager" will be converted from Helm v2 to Helm v3.
2019/11/16 07:24:07 [Helm 3] Release "cert-manager" will be created.
2019/11/16 07:24:07 [Helm 3] ReleaseVersion "cert-manager.v1" will be created.
2019/11/16 07:24:07 [Helm 2] Release "cert-manager" will be deleted.
2019/11/16 07:24:07 [Helm 2] ReleaseVersion "cert-manager.v1" will be deleted.
The dry run output looks reasonable. Lets run it for reals this time with the --dry-run flag:
~/Downloads/helm-v3.0.0-linux-amd64/linux-amd64/helm 2to3 convert --delete-v2-releases cert-manager
2019/11/16 07:24:50 Release "cert-manager" will be converted from Helm v2 to Helm v3.
2019/11/16 07:24:50 [Helm 3] Release "cert-manager" will be created.
2019/11/16 07:24:50 [Helm 3] ReleaseVersion "cert-manager.v1" will be created.
2019/11/16 07:24:50 [Helm 3] ReleaseVersion "cert-manager.v1" created.
2019/11/16 07:24:50 [Helm 3] Release "cert-manager" created.
2019/11/16 07:24:50 [Helm 2] Release "cert-manager" will be deleted.
2019/11/16 07:24:50 [Helm 2] ReleaseVersion "cert-manager.v1" will be deleted.
2019/11/16 07:24:50 [Helm 2] ReleaseVersion "cert-manager.v1" deleted.
2019/11/16 07:24:50 [Helm 2] Release "cert-manager" deleted.
2019/11/16 07:24:50 Release "cert-manager" was converted successfully from Helm v2 to Helm v3.
Lets list it:
~/Downloads/helm-v3.0.0-linux-amd64/linux-amd64/helm list --all-namespaces
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
cert-manager cert-manager 1 2019-11-16 15:18:06.162681463 +0000 UTC deployed cert-manager-v0.11.0 v0.11.0
Looks like it is in the Helm v3 now.
kubectl -n cert-manager get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cert-manager-756d9f56d6-7f7dm 1/1 Running 0 7m57s 10.103.5.56 gke-gcp-dev-generic-1-16723766-xcs5 <none> <none>
cert-manager-cainjector-74bb68d67c-tlwgn 1/1 Running 0 7m57s 10.103.5.57 gke-gcp-dev-generic-1-16723766-xcs5 <none> <none>
The cert-manager pods are all still there with no changes
Running a deploy with Helm v3 to test it out:
~/Downloads/helm-v3.0.0-linux-amd64/linux-amd64/helm upgrade cert-manager --install --namespace cert-manager -f values.yaml ./
load.go:112: Warning: Dependencies are handled in Chart.yaml since apiVersion "v2". We recommend migrating dependencies to Chart.yaml.
Error: UPGRADE FAILED: cannot patch "cert-manager-cainjector" with kind Deployment: Deployment.apps "cert-manager-cainjector" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"cainjector", "app.kubernetes.io/instance":"cert-manager", "app.kubernetes.io/managed-by":"Helm", "app.kubernetes.io/name":"cainjector"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable && cannot patch "cert-manager" with kind Deployment: Deployment.apps "cert-manager" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"cert-manager", "app.kubernetes.io/instance":"cert-manager", "app.kubernetes.io/managed-by":"Helm", "app.kubernetes.io/name":"cert-manager"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
Oops…it fails…ugg
Well…it looks like it is trying to change an immutable field. Not really sure what we can do about this or if we really want to spend time figuring out this problem. In this specific case, it is probably just easier to delete the release and install it again. For this deployment, it wouldnt really matter. If this was an nginx-ingress then that would matter because deleting it would mean downtime.
I also confirmed that installing the cert-manager with Helm v3 and upgrading it works as expected. So I guess
with all migrations it is not perfect.
Contact me if you have any questions about this or want to chat, happy to start a dialog or help out: blogs@managedkube.com {::nomarkdown}
Learn more about integrating Kubernetes apps
{:/nomarkdown}
Helm | v3 | Debugging | with | cert-manager