Kubernetes Blunder Version 1.22

Disclaimer: I’m not a dev ops guy… I’m definitely not a kubernetes guy. This is based on my experience blundering through some k8s upgrades until I could migrate away from this infrastructure.

Anyways, on to the story, which I hope can help someone else (or at least provide a laugh for a real dev ops engineer).

The control plane upgrade for k8s version 1.22 went smoothly! We’re typically only upgrading to avoid the AWS end-of-life calendar (opens in a new tab). When the control plane goes well, I typically get excited, thinking the entire version upgrade will be a breeze. Since we’re a bootstrapped company and my responsibilities primarily lie in the product and application realm, an easy upgrade (for me roughly 20–30 hours), means more time to spend on the other areas of my job.

Unfortunately, one of our add-ons kubernetes-external-secrets failed to upgrade. The add-on was deprecated in favor of ESO or External Secrets Operator, so my first thought is that the deprecation has finally caught up with us and we need to migrate. I had tried my hand at the migration once before and ended up finding a way to keep the deprecated version running due to complications with the upgrade that I couldn’t sort out in a reasonable amount of time.

The Bug: every time I run bazel run on my deployment, I see this error:

WARNING: This chart is deprecated
Error: UPGRADE FAILED: current release manifest contains removed kubernetes api(s) for this kubernetes version and it is therefore unable to build the kubernetes objects for performing the diff. error from kubernetes: [unable to recognize "": no matches for kind "ClusterRole" in version "rbac.authorization.k8s.io/v1beta1", unable to recognize "": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"]

What Happened?

After a ton of searching, I found this article (opens in a new tab) that included the fix to my problem.

Recommendation: The best practice is to upgrade releases using deprecated API versions to supported API versions, prior to upgrading to a kubernetes cluster that removes those API versions.

If you don’t update a release as suggested previously, you will have an error similar to the following when trying to upgrade a release in a Kubernetes version where its API version(s) is/are removed:

Helm fails in this scenario because it attempts to create a diff patch between the current deployed release (which contains the Kubernetes APIs that are removed in this Kubernetes version) against the chart you are passing with the updated/supported API versions. The underlying reason for failure is that when Kubernetes removes an API version, the Kubernetes Go client library can no longer parse the deprecated objects and Helm therefore fails when calling the library. Helm unfortunately is unable to recover from this situation and is no longer able to manage such a release. See Updating API Versions of a Release Manifest (opens in a new tab) for more details on how to recover from this scenario.

Okay! We have some details, so let’s take a look at the recovery, as suggested. I opted for manual steps over using the “mapkubeapis” helm plugin (opens in a new tab).

Steps to Recover

Depending on your configuration you will follow the steps for the Secret or ConfigMap backend.
Get the name of the Secret or Configmap associated with the latest deployed release:

This is a good start. I have no idea if I have a Secret or Configmap backend… let’s run both!

kubectl get secret -l owner=helm,status=deployed,name=kubernetes-external-secrets — namespace kubernetes-external-secrets | awk ‘{print $1}’ | grep -v NAME

returned:

sh.helm.release.v1.kubernetes-external-secrets.v27

So then, I tried the other option:

kubectl get configmap -l owner=helm,status=deployed,name=kubernetes-external-secrets - namespace kubernetes-external-secrets | awk '{print $1}' | grep -v NAME

which returned:

error: name cannot be provided when a selector is specified

... so I’m going to say we have a Secret backend! Cool, good to know!

Step 2: Get latest deployed release details:
Secrets backend: kubectl get secret <release_secret_name> -n <release_namespace> -o yaml > release.yaml

ls confirms I now have a release.yaml locally. Fun!

Step 3: Backup the release in case you need to restore if something goes wrong: cp release.yaml release.bak
In case of emergency, restore: kubectl apply -f release.bak -n <release_namespace>

Definitely not what I want to see, but a good idea… ran cp and confirmed via ls

Step 4: Decode the release object:
Secrets backend:cat release.yaml | grep -oP '(?<=release: ).*' | base64 -d | base64 -d | gzip -d > release.data.decoded

response:

grep: invalid option - P

Uh oh! Okay, let’s google some more. It turns out that the grep installed by default as part of coreutils on macs is not “gnu grep,” and the version installed doesn’t have a -P flag (aka- -perl regexp or Interpret PATTERN as a Perl regular expression.)

Cool, so solution should be to install a version of grep that does have the -p flag and is “gnu grep”:

brew install grep

returns:

All commands have been installed with the prefix "g".
If you need to use these commands with their normal names, you
can add a "gnubin" directory to your PATH from your bashrc like:
 PATH="/usr/local/opt/grep/libexec/gnubin:$PATH"

Okay, I can handle that!, 2 g’s!

cat release.yaml | ggrep -oP '(?<=release: ).*' | base64 -d | base64 -d | gzip -d > release.data.decoded

sudo nano release.data.decoded confirms it worked! On a roll here!

Step 5: Change API versions of the manifests. Can use any tool (e.g. editor) to make the changes. This is in the manifest field of your decoded release object (release.data.decoded)

Sure, control+w in nano to search for the deprecated release, manually change v1beta1 to v1

Step 6: Encode the release object:Secrets backend: cat release.data.decoded | gzip | base64 | base64

This printed a huge blob of encoded text in my terminal. We’ll call it a win!

Step 7: Replace data.release property value in the deployed release file (release.yaml) with the new encoded release object
Step 8: Apply file to namespace: kubectl apply -f release.yaml -n <release_namespace>

This returns:

Warning: resource secrets/sh.helm.release.v1.kubernetes-external-secrets.v27 is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create - save-config or kubectl apply. The missing annotation will be patched automatically.
secret/sh.helm.release.v1.kubernetes-external-secrets.v27 configured

Step 9: Perform a helm upgrade with a version of the chart with supported Kubernetes API versions

For this, I just ran a bazel run deploy, which in our environment handles the upgrade

Woohoo! Lens confirms KES deployment is back up and available! The saga ends and I was able to confirm that the secrets deployment was working appropriately for our cluster.

Step 10: Add a description in the upgrade, something along the lines to not perform a rollback to a Helm version prior to this current version

I opted to skip this important step because no one else touches this and we’ll never do a helm rollback. Hopefully 🤞 we won’t be using kubernetes for many more upgrades!

Key Takeaway

Next time, I will certainly follow the advice above and upgrade add-ons using deprecated APIs before upgrading the control plane!