Secrets Management “Done Right” Improves Your DevOps KPIs

An IT Ecosystem cloud technologies infrastructure is constantly changing and must scale to fit the needs of your organization. Cloud technologies require secrets management to manage an organization’s tokens, keys, and passwords. DevOps Engineers are tasked to manage the ever-increasing amount of secrets to automate deployments and build infrastructure in the cloud. There are several ways to manage your organization’s secrets, some better than others.

Initially, organizations building their Cloud capabilities may fall into the trap of doing what allows them the greatest velocity at first. Which is, clear-text secrets managed using an Excel document or passed using a communication channel (Email, SCM, etc.). Secrets are then hard coded into automation using a credential/password manager that are difficult to manage and update as the capabilities are scaled. Beyond the operational burden of manually managing secrets, a security risk is introduced if tokens are not kept track of in environments.

The next option when cloud capabilities are involved is turning to built-in secret management tools through various cloud providers. While these provide a solution mainly for the services of the cloud provider, they can’t scale with a growing number of tools and platforms. For example, you use a solution like AWS DevOps (CodePipeline, CodeDeploy, CodeBuild), but also have development teams using Azure DevOps (Kubernetes, Azure Pipelines, Jenkins). Having credentials stored in AWS, creates a complex situation to use the same credential management elsewhere. 

Implementing a solution like this is known as a “walled garden” or “closed platform” approach, where the provider offers a solution only within the parameters of its own architecture (for example, AWS Secrets Manager offers functionality only for the AWS universe.) At a point in your DevOps roadmap, externalizing secrets away from a cloud provider will be a huge operational burden on your team. A DevOps team is dealing with dozens of tools daily (Ansible, Docker, Puppet, Chef… to name a very few) not to mention many of these credentials have different levels of permissions. Like all other pieces of a DevOps Pipeline, secrets need to be automated as well.

Vault-as-a-Service: The Path Toward Fully Automated CI/CD Pipelines

The ideal situation for a DevOps team is a scalable and centralized secret management, vault-as-a-service solution. Dependence on cloud providers’ ‘closed garden’ secret management systems or manually updating secrets that are baked into automation code is an enemy of successful, secure DevOps. Using vault-as-service to provide on-demand, temporary secrets is the path towards fully automated CI/CD pipelines. Anything from password and tokens up to TLS Certificates can be provided on-demand into a pipeline run. 

There are several KPI benefits for managing credentials the right way: 

Impact on DevOps KPIs

Deployment Frequency

The ability to use on-demand temporary secrets instead of secrets baked into your automation configuration files and deployment systems allows a DevOps team to avoid the burden of manually updating credentials in pipeline code. There is no time spent cleaning up credentials after a deployment, and more time is spent on building automation and new integrations. Pointing to different environments on the fly, with different permissions allows the pipeline to be ephemeral and dynamic.

Failed Deployment Rate

Failing during deployments is often a setback that costs DevOps teams several hours in their day, and unhappy end-users. Hardcoded, out of date credentials or keys failing right after an hour-long code scan from products such as SonarQube, Synopsys Tools (PrimeTime, Coverity) or Fortify is a harsh reality of manually updating credentials in scripts. A credential error is often extremely difficult to disseminate from a normal tool outage. Skip worrying about potentially out of date credentials or certificates, and set your focus on other important tasks during deployment windows. 

Lead Time

Without automated secrets management, deploying on a large scale is difficult. Shorten time to market by providing temporary secrets across all your automation. Secrets that originate from a single source that don’t need to be manually updated or cleaned by the DevOps team will allow a developer’s code to go to production faster. The SDLC (Software Development Lifecycle) of your organization can’t afford to have inefficiencies or bottlenecks caused by injecting manual processes such as updating credentials in your automation flow.  

MTTR (Mean Time to Repair)

Automatically managing TLS certificate lifecycles and SSH Keys, instead of relying on things like an Excel spreadsheet will give your organization a much shorter time to recovery. In the case of an outage, with on-demand temporary secrets a new connection is established with an environment on the fly to deploy hotfixes. Building resilient systems is of huge importance, and temporary on-demand secrets provide huge benefits when recovering or rolling back quickly. Create pipelines quickly and rollback as quickly as you deploy.

Secrets Management done right through a vault-as-a-service allows a DevOps team to improve across several common KPI Metrics. Using on-demand temporary secrets will allow your DevOps team to automate the automation. Operationalizing a tedious manual task, such as updating credentials will free up time to focus on other deployment tasks and improve your organizations development efforts.