Secrets Sprawl In Practice

Gui Martins, DevSecOps Specialist, ObjectSharp

Gui has been working with Kubernetes and micro services and serverless in the cloud for a while. He also has a Youtube channel where he creates various technical tutorials for developers.

Hello, everyone. Hi. Gui Martins, for all of you guys can’t say Guilherme as well if you want to try to practice pronouncing that. As you can see, I’m a DevSecOps Specialist, whatever that means, right? That’s another buzzword, I guess. All that actually means is that I’ve been all over the place in 10 plus years, wearing lots of hats. Right?

Developer, operations, networking. So, lots of things. That translates to a bunch of things that I’ve done in the past. So, you can see pretty much all the technologies that all of you are familiar with. And I’ve been on the side of implementing those. And part of being a DevSecOps Specialist is implementing process, process, process. Getting everything right so everybody is ready to go. Which by the way, thank you for being here. And as you guys already here, there’s a lot of space for us to grow in that space, especially implementing security on top of all that. And thank you for everybody at home as well.

And I dabble a little bit into speaking on YouTube a little bit about that, FancyGUI, if you guys… Fancy, if you guys want to check it out later. And why should you listen to me? Right? Of course, I’ve done all that. But I’ve also done that into a bunch of financial institutions. I used to work for Microsoft doing consulting. And I do work with lots of big financial institutions. And I’ve seen all their secrets.

And as all of you can imagine, being CTOs and also engineers in security, you have seen how it can be difficult to apply all of that we were talking about here today into your company. It is kind of complicated. Right? Everybody feels there is going to be a lot of friction. Everybody feels it’s going to be complicated. And some people think that it got it right, which is often true. But I’m going to show you something, as you saw from the title of my presentation, that there’s some sort of sprawl that just happens. And cue Megalodon music, like the horror movie music. And this is how it starts.

So, I come into the company and I asked them this, “How does your architecture look like?” What it looks like, it’s simple. Right? And if you ask anybody in your own company, you’re going to have multiple products. How many products do you have? And how your applications communicate with each other? And how your users communicate? Is sounds simple, doesn’t it? But if you’re talking about 1000 plus company, 1000 employees plus company, you’re going to see that there’s going to be a lot of things that you’re going to be missing here to fill in the gaps.

And more often than not, I come across a diagram like this. Does anybody here have a diagram like this from a product? Probably, right? It starts off like this. You’re going to have an application and user communicating with it in a database or cache to make sure everything’s fast, a queue service to queue everything up. But it’s missing something, isn’t it? Well, it’s probably because it’s quite simplistic. I’m not talking about networking here. I’m not talking about anything else. But it’s also missing something quite important.

The credentials, the certificates, the keys, everything that makes your system communicate and glue in everything together. And it’s often overlooked. It’s not in the forefront of our mindset of how we create or architectures, how we create our applications, or products. And there are more services involved.

When you start building that up, and I’m talking about greenfield, but not only, big companies, they start to realize they have to process that data. They have to have operations engineers connected. They have to have their DBAs connected. How did he do that? They’re going to have secrets. They’re going to have certificates. They’re going to have to go to those environments somehow. I guess you guys can see the trend here, right?

And also, where does it run? For now, most companies are going to be into 1 or 2 cloud providers, or even just 1. But I truly believe, and I’ve seen it in all the companies that I work with that cross cloud is the answer for getting the best value offer for whatever you’re going to be running on the cloud. You need to be prepared for that. And if you’re thinking about all those secrets, and all those been providers, well, where do I put my secrets? Right?

Well, I’m going to just put in the Azure Key Vault for Azure. I’m going to put in AWS Secrets Manager for AWS and GCP Secret Manager. Simple enough, right? Well, how do you share those secrets between each other? It’s complicated, right? And this is just blown up too many environments to your disaster recovery environment, to your business units and subsidiaries and regions. And that is just 1 product. Imagine how many of those are you going to have to manage later on.

So, this is the actual sprawl that happens. And because we didn’t think about those secrets from the get go, this is what happens. And now, as Chase, everybody here is actually talking about, you’re going to lose track of who has access to what, of where your secrets are, and the versions that you have.

And I’m not even talking about this software delivery. I’m not talking about the lifecycle of your applications, not talking about user management, and scanning or your infrastructure, how you delivering that, how you manage the secrets of that infrastructure. And then you’re asking, “Hey, I know you’re scaring me here. What is the answer to all of that?”

Well, what’s the secret sauce? Right? Why me going to those clients and trying to implement their architectures and successfully so I can make it right? Is it because it just chose the right tool for the job? Well, not really. It does help that you have the right tool. But what I know he works is that if you bring something that I call the secret management framework, you’re going to be successful. It’s simple. And it’s just the mindset that you have to remember when you’re implementing those tools, that you’re going to have to have it right from the forefront. So, that is privacy control and availability.

So, hear me out. Privacy, ask yourselves, is it just you that has access to your secrets? Is it really? Are you going to have just 1 account that has access to all the secrets from 1 environment? Are you going to be able to segregate your team? Because they have to have multiple vaults everywhere. Are your environments truly isolated? Or do you have only access to the things that you’re supposed to have? These are the things that I have to think about in privacy before you use a tool to create a solution. A tool’s not going to solve for that. You are going to have to solution for that.

Control. Well, now that I know who has access to what, how do I control that that keeps flowing? How do I know that I’m not going to just give access to someone someday to a super root account that is going to have access to everything else? You have to audit that. You have to have the centralized auditing, so you don’t lose track of that. Because if you have to go to Splunk, Datadog, and login analytics to try to find the data of who has access to what, when, and where, well, you’re going to be lost. So, having an identity management system, something that is centralized, whatever that might be, is the solution for you to be centralized. And then you don’t have to have 1000 plus engineers just looking at logs that are going to be spread it all apart.

And this is my favorite, availability. People often forget that the secret management is a mission critical infrastructure. How many of you here has slept in the night that you have to rotate a secret in production? It’s complicated, isn’t it? If your service that has your secret is not rotated often enough and it’s not really up to date, or it’s not available to do a rotation, guess what? Your secret is not going to rotate. And chances are your production is going to have downtime. And most people don’t think about that. They just take it for granted. But the communication is a key part of how your infrastructure is going to work. And those secrets are just part of that.

So, you have to have backups of your secrets, of your secret management framework, of your solution. You’re going to have to have it running multiple regions, because if God forbid, a disaster happens, you’re not going to lose access to all of your secrets that are necessary to go back to where you were into all those regions. And you’re going to make sure that you’re going to be rotating those secrets and all of those regions automatically and the best way that you could do it. And that’s when you’re going to ask yourself, “How many 9s do I have in my secret management solution?” And that’s quite important.

So, with all that in mind, if you guys just take all of those 3 topics from the… let me just go back there. Go on. If you just take privacy, control, and availability, and apply that whichever tool you choose, which so it happens that Akeyless quite helps to solve all that, you are going to be successful in whatever you do. That’s it. Thank you, guys.