Problem & Context
One of the k8s clusters were upgraded from version 1.19 to 1.22 by a teammate and it’s discovered later that somehelm releases can not be upgraded because:
the manifest contains removed kubernetes api(s) for this kubernetes version
Even the resource definition is updated with the correct apiVersion the same issue persists: The manifest of the current revision of the release is still needed during helm upgrade
Some context:
helm@v2is used for this cluster and there is atillercomponent- the affected releases are serving traffic and ideally, we don’t want to take them down unnecessarily
Attempted Approaches
1st attempt is to see if helm supports a way to force upgrade without diff ing against the current release. helm upgrade --force end up running into the same issue.
With the quick solution out of the picture, 3 practical solutions remain:
- locate and modify the manifest with the correct
apiVersion - deploy the same chart under a different name, and migrate traffic to the new deployment (e.g. point DNS to the new ingress)
- take down the service (
helm delete --purge) and re-release with the correctapiVersion
Balancing effort and risk, decided to go with option #1. Learned quite a lot about helm and sharing the learning here.
How-to
tiller keeps track of helm releases using ConfigMaps, you can see those via kubectl get configmaps -n <tiller_namespace> -l OWNER=TILLER
Each revision of each release maps to a unique ConfigMap . The release manifest/configuration is encoded and stored under jsonPath={.data.release}
On a high level, the approach can be described as:
- locate the right revision and find the matching
ConfigMap - decode the manifest, modify and encode in the same way
kubectl edit/patchtheConfigMapwith altered manifest- proceed with
helmrelease
The 2nd step took most of the time but here’s the official reference from helm . v2.17.0 is referenced here. In case you’re running a different minor version of helm, switching to the correct branch.
Commands
1. Prerequisites:
export HELM_PROTOBUF_HOME=${HOME}/src
mkdir -p ${HELM_PROTOBUF_HOME}HELM_VERSION=2.17.0git clone https://github.com/protocolbuffers/protobuf.git ${HELM_PROTOBUF_HOME}/protobuf
export PROTOBUF_SCHEMA=${HELM_PROTOBUF_HOME}/protobuf/srcgit clone https://github.com/helm/helm.git ${HELM_PROTOBUF_HOME}/helmpushd ${HELM_PROTOBUF_HOME}/helm
git checkout tags/v${HELM_VERSION} -n v${HELM_VERSION}
popd
export HELM_PROTOBUF_SCHEMA=${HELM_PROTOBUF_HOME}/helm/_proto
Note that both repos (protoc ,helm) need to point to the right version that matches the version helm you used on your k8s cluster.
2. To locate the right revision, run:
kubectl get configmap -n <tiller_namespace> -l OWNER=TILLER,NAME=<release_name> to locate the right revision. The revision name follow the format of <release_name>.<revision_id> where <revision_id> is a positive integer.
3. To decode the release manifest:
kubectl get configmaps/<release_name>.<revision_id> -n <tiller_namespace> -o=jsonpath='{.data.release}' | base64 -d | gunzip | protoc --proto_path ${HELM_PROTOBUF_SCHEMA} --proto_path ${PROTOBUF_SCHEMA} --decode hapi.release.Release ${HELM_PROTOBUF_SCHEMA}/hapi/**/* > manifest_decoded.yaml
4. You can make the modification inside manifest_decoded.yaml . A good example would be to update the apiVersion of resources.
5. Encode the modified manifest:
cat manifest_decoded.yaml | protoc --proto_path ${HELM_PROTOBUF_SCHEMA} --proto_path ${PROTOBUF_SCHEMA} --encode hapi.release.Release ${HELM_PROTOBUF_SCHEMA}/hapi/**/* | gzip | base64 --wrap 0
6. Update the ConfigMap , you can either use kubectl edit ,kubectl patch . If you have no problem understanding the steps so far, you’ll be able to figure out the last step without detailed command.
Summary
This is a good opportunity to gain insights into how helm/tiller manages release manifests. I’m glad this was figured out after some research.
Although helm@v2 is definitely not the best option for a new project, I’m sure there are many places where it is still widely used. I hope this is helpful!
