Sam Gabrail – Platform Engineer

Secure Kubernetes Secrets with Akeyless

Did you know that Kubernetes secrets are not actually that secret? I certainly didn’t know that when I started with Kubernetes.

In this video, find out why that’s the case and how with Akeyless, you can securely store your secrets centrally and then easily retrieve them for your microservices that are running in Kubernetes.

Let’s get started. There are many types of Kubernetes secrets.

You can see there’s the opaque, There’s a service account, Docker config, basic authentication, SSH authentication, TLS, and bootstrap token data. If you don’t specify anything by default, it’s considered an opaque secret.

There’s a service account token as well that you can use. However, in Kubernetes version one twenty two and later, the recommended approach is to actually obtain a short lived automatically rotated service account token by using the token request API.

And here’s an example. So you can use a YAML file and put the secrets that way. Just make sure that you don’t push this into Git because this is all Base64 encoded, which can be easily decoded. There’s the Docker config secrets, which allows us to store credentials to access a container image registry that is private.

Then a basic authentication secret, which is very similar to opaque.

However, it is built for username and password, so you must have these two in the data field.

Then there’s the SSH authentication secret as well. So you can store a private key as you can see here.

There’s also the TLS secret where you can store a certificate and a key.

And finally, a bootstrap token secret, and this is for tokens used during the node bootstrap process. It stores token used to sign well known config maps.

It usually created in the cube system namespace.

So these are the main types of secrets in Kubernetes, and there’s a lot in this documentation you can look at in terms of working with these secrets.

However, really quickly here in our repo, I show you the main ways of creating secrets. The first one is Kubectl and using it like this on the command line. And you can use from literal flag to provide the username and password, for example. So we’re giving him the secret here from literal. So we’re not Base64 encoding it or anything, and we’re supplying it on the command line. And then you can verify the secret’s create is successful using Kubectl get secret, and then the name of that secret, which is my secret in this case.

You can also use a manifest file, which is that YAML file I mentioned before. So you give it the kind secret and name, the type, and then the data here, in this case, username and password.

And both are Base64 encoded.

The encoding here is for admin and secret like that for for username and password. And once again, you can apply it like any kind of resource or manifest that you apply in Kubernetes with the Kubectl apply command as you can see.

You can also use customize, and you can build a customized dot YAML file like this. We’re putting the username password and then applying Kubectl apply dash k for customize, and the dot is the context where we are. So in the directory where we are when you have this customization dot YAML file. Now let’s see how this has become a misconception that Kubernetes secrets are actually secrets and that they’re encrypted at rest. So what I wanna show you is that that’s not true, and it’s just a Base64 encoding that we can easily decode.

So in my terminal, I can run Kubectl get secrets, and I have a bunch of secrets. I’m interested in this opaque secret.

So I can go ahead and run hoop CTL get and secret.

And then this secret here and give it an output JSON path equal dot data.

And you can quickly see my password and username, and these are Base64 encoded.

So I can easily decode any of these by running that same command. And this time, I wanna decode the username and pipe that to a Base64 dash dash decode, and I get my admin as my username. And if I wanna do this with password, I get my password. Okay?

So that’s fine. And someone might say, but you have to have Kubectl permissions to be able to access this. So what I’m gonna show you now is I’m actually going to grab the data directly from the database.

So the database could be etcd or in my case here, since I’m running K3s, it is SQLite.

So I am on the actual master node right here.

And what I’m gonna do is I’m gonna run this SQLite command to access the database.

Now I’m dropped into a prompt or a shell in the database, I’m gonna run this select command inside my database, and this will give me a hex encoding here that I’m simply gonna just copy.

And I’m gonna go back to my terminal here and clear this.

And I have a Python script here, decode hex py that would we can give it the hex data, and it prints out the, decoded string that we have here. And you can see that we’ve got our data here, password and username, just exactly what we saw before.

And, again, I can easily decode any of this. So I can run this command, for example, which will decode my username that I already have here, and I get admin as well. So straight out of the database, I was able to grab the username and password. And as you saw, it’s just encoded that it can easily decode it.

So anybody who gets access to the data that sits on the database can effectively view the secret. And that is why it’s important to use Akeyless because it encrypts all your secrets at rest. Now we’re going to see how to use Akeyless with Kubernetes secrets. So we’re gonna have this demo where we’re going to use the Akeyless agent injector.

So some prerequisites is that we need a Kubernetes cluster version one nineteen and above, Helm and Kubectl installed, an Akeyless account, and your Kubernetes authentication method already configured.

I just wanna show you how I’ve configured my Kubernetes authentication. So under users authentication methods, I have this my Kubernetes auth method already configured.

And I can see here the configuration for Kubernetes with my gateway using native Kubernetes, my cluster endpoint, cluster CA, a barrier token.

And I also have associated roles here. So if we click here, it takes us to access roles. And under access roles, I have a very simple rule that allows me to access anything under the path Kubernetes, anything under Kubernetes that can read and list. And, really, that’s where we’re going to spend our time in terms of retrieving secrets from the Kubernetes folder here, and I only have one secret, my Kubernetes secret, that, I’m gonna be working with. The gateway also is configured if you look at Kubernetes auth. Same thing. Here’s my Kubernetes auth method right here with the same view that we saw in the actual AWS console.

Now if you need to learn a little bit more about how to configure Kubernetes authentication method inside of Akeyless, you can go to the documentation here, which is what I followed, dedicated Kubernetes service accounts. And just follow the instructions here to go ahead and build the configuration that I just showed you. Finally, for the demo we’re walking through right now, I followed the documentation that is here, made a few modifications, but for the most part, you can also follow it right here. And both these two links, I will have inside the blog post and also in the description of this video.

Step one is to install the Akeyless Secrets injector.

First of all, we want to run the Helm repo add and Helm repo update commands. So we can go ahead and run those.

I already added these from before, so they should already exist.

And then what we need to do is run the Helm upgrade install command here and reference the values dot YAML file, which I’ll show you in just a minute. So run this.

And, really, what we wanna focus on in the values dot YAML file are the environment variables here.

So I can see Akeyless URL. This is the default SaaS URL for Akeyless.

The access ID that I’m using for my access type of Kubernetes. So for my authentication method of Kubernetes, I need the access ID.

Also, the API gateway URL right here, which I’m using actually Ngrok. And if you notice, I am running Ngrok here. And what it’s doing, it’s forwarding the public URL to a local host port eighty eighty where I’m actually running my gateway.

And then finally, the Akeyless Kubernetes auth configuration name, that’s the name of my authentication method in Kubernetes.

Now if I run Kubectl get all, I should be able to see all my pods are running. Here are the two injector pods that I have.

Alright.

Now there are multiple modes of retrieval for secrets in Akeyless that we can use within our Kubernetes environment.

These two modes are environment variables and file injection.

Now one drawback of environment variables mode is that you need to restart the pod every time the secret changes in Akeyless.

However, with file injection, we can run a sidecar that continuously retrieves any updates to the secrets in Akeyless, whether we’re making these updates manually or if we’re using rotated or dynamic secrets.

So now step two is to retrieve secrets into an environment variable, and we’re going to use this app underscore env dot YAML file. And as you can see, it’s just a deployment.

And what I wanna draw your attention to is really the annotation here. So the annotation Akeyless enabled is true. This will trigger the mutating webhook that we have, which is the Akeyless injector to go ahead and retrieve secrets for us. And if you notice, here, we’ve got a pod or a container in this pod.

It’s just running Alpine with a few commands. And here we have echo my secret. As you can see, it’s now an environment variable, my secret, and we’re retrieving the value from Kubernetes, my Kubernetes secret. And this is a great point to show you what our secrets look like inside of Akeyless.

So in Akeyless itself in the UI, I have a secret under that path, Kubernetes my Kubernetes secret. So let’s take a closer look at that.

So I’ve got a static secret under Kubernetes my secret right here, and the value of that is my password.

Alright? So we are gonna retrieve this.

Let’s go ahead and Kubectl apply this file, but first, let’s change our namespace to this app.

And let’s go ahead and run kube CTL apply dash f app environment dot YAML.

And that should go ahead and deploy this for us. I see the pod already here.

So let’s clear, and let’s look at the logs for this pod.

And you can see we’ve retrieved our password, my password. As you can see here from running the spot, we’re just echoing the environment variable here. So that is successful. We’re able to get our secret using environment variables.

Alright. Let’s continue to move on here to the next step, and we’re gonna retrieve secrets into a file now. So let’s apply this command here, Kubectl apply app file dot YAML.

And real quickly as this applies, it’s also a deployment.

And now we have an additional annotation that is saying to inject file, and we’re grabbing our secret from the same path Kubernetes and my Kubernetes secret.

And what we’re doing now is we’re actually printing out the secrets and then the actual secret under this file.

And as you notice here, if we don’t specify the location of where we want the secret to be inside of the file system of the container, it’s gonna default to Akeyless secrets.

So let’s now look at the logs for this one. So if we run Kubectl get all, I see my second pod here, and that’s we get the logs for that one, and that also prints out my password right here.

We could take this a step further and kube CTL exec into this pod and just get a shell into this alpine pod or container and run cat secrets or cat Akeyless secrets, Kubernetes, my secret. And we see that my password shows up in the right place where we expect it to inside of our container.

And now let’s exit out of that and move on to our final piece in the demo, which is retrieving secrets into a file continuously with a sidecar.

So far, so good. However, some applications we want to use maybe dynamic secrets or rotated secrets.

We wanna find a way for us to retrieve the secret on a continuous basis.

So let’s go ahead and apply this file here.

And as the deployment gets applied, let’s take a look at this deployment. And really what I wanna focus on are the annotations.

So we’ve seen that first two annotations. However, the second one, I’m actually changing the location where I want the secret to show up inside the file system of the container. So now it’s under secrets, secrets version dot JSON.

And here, I’m showing that I want the sidecar enabled, and I want the sidecar refresh interval to be five seconds. So every five seconds, I’m going to retrieve the secret. Maybe this is a little aggressive. Thirty minutes is the default, but five seconds just for a demo so we can see changes happening.

And then sidecar versions to retrieve, we’re gonna retrieve only two versions of our secret.

Alright, same thing. We’re running an Alpine container and we have a while loop here, an infinite while loop that is going out and looking at our secrets version dot JSON file. And we’re printing that and going to sleep f every fifteen seconds. So let’s go ahead and now take a look at some of our logs.

So let’s see our container or our pod is up and running. So notice now that I have two containers actually running inside of this pod. So let’s take a look at the logs, and it’s actually defaulting to Akeyless sidecar.

Alright. There are three containers. There’s the keyless in it, which completed, and then there’s the Alpine, which is the actual application, and then there’s the keyless sidecar itself right here. So we don’t see any outputs right now when we change the secret, we’ll see some output here for the Akeyless sidecar. So for now, let’s make sure we specify the Alpine container by running the dash c command on Alpine.

And we can see here that I’ve got version four and version three. Now what we’ll do is we’ll leave the logs running, and we’ll go back to our Akeyless console.

And let’s make a change for my password. Let’s change that and make sure we keep the previous version. Let’s call this my password two. Submit that, and let’s go back.

If we wait a little bit, this should automatically change for us to reflect the change we’ve done. And remember, we’re only grabbing the last two. So here we go. The version five now is showing my password too, and version four is still my password.

So if I exit from here and clear and try to run the same command, but this time getting the logs from the Akeyless sidecar, you can see this info log here saying that we successfully wrote a secret to this particular file, which is great. Now let’s actually get into the pod and go into the file just to take another look. So if you go in here and exec into this container and make sure we are using shell.

And then if we cat secrets, you can see under secrets, there’s time stamp and secrets version. Under secrets version is where the actual secrets are getting written to.

Alright?

And that’s pretty much it. It’s really helpful because, again, these secrets can change. The only thing that you need to keep in mind as a an application developer is there has to be some sort of logic in the application to check this file in case of an error. So there has to be a retry mechanism.

For example, if you have a web application that is trying to retrieve secrets to talk to a database and that database is using dynamic database secrets inside of Akeyless.

If on failure to connect to the database, then that web server needs to be able to recheck or reread the password that’s inside of the file here and try again. And that way, we solve the whole issue of having long lived credentials in your environment.

In this video, we explored the shortcomings of Kubernetes secrets and why they were not as secure as initially believed.

We examined various types of secrets in Kubernetes such as opaque, service account, Docker config, SSH authentication, and more.

We demonstrated ways to create and manage these secrets using Kubectl and YAML files. We highlighted that Kubernetes secrets were merely Base64 encoded and could be easily decoded, posing a security risk. So we introduced the Akeyless agent injector to retrieve secrets to address this. We covered retrieving secrets as environment variables or files and using a sidecar container to update secrets continuously.

You can now have all your secret in one place properly encrypted at rest in Akeyless and retrieve them from different environments such as Kubernetes in our example.

Thanks for watching, and I will see you in another video.