This is a quick introduction to AWS’s product called Secrets Manager. While working at a Stride client, I was involved in migrating secrets to this product, so I had the fun of reading all the aws secrets manager docs and learning about it.
We’re going to talk about why a secrets manager is useful. Then we’re going to compare this secrets manager to other solutions. We’ll talk through things to keep in mind when it comes to the secrets product, as well secrets in general.
Let’s quickly review what a secret is. Secrets are things we especially need to hide, like database passwords and API keys. These are things that are not stored in your codebase because of their sensitivity.
Why use a secrets manager?
So why use a secrets manager? A very small-scale way to hide information would be to just store your secret in a file on your server. This approach is often used in small Rails projects—for example, in a `.m` file. But this is not going to scale. If you have thousands of secrets, hundreds of microservices, or hundreds of developers who need to change and update these secrets, it’s just not maintainable with one file.
You get some extra bonuses when you use a secrets manager. For instance, you can:
- Auto-rotate secrets, just as you might do with your iTunes password. You might want to change that secret on a regular basis, in case a bad actor gets hold of it. If it only has that value for a short amount of time, they can’t do as much damage.
- Define and access editing permissions. Maybe certain groups of developers are in charge of a certain microservice, and they can change those secrets but not the whole ecosystem.
- Implement infrastructure as code. If you’ve ever heard of Terraform, that’s a way to write in code all of your resources and then use that configuration to launch your apps. So you can put your secrets manager into Terraform, or you can put it into Kubernetes configs. When I used this technique at that Stride client, the main piece I worked on was changing those configs to pull in the secrets from AWS Secrets Manager. That was very easy to do, because it was all infrastructure as code. It was just unplugging one thing and plugging another in. So if you decide to change your solution, you can do that very easily, as long as you have already implemented a secrets manager.
Secret management tools
Let’s compare and contrast a couple of players in secrets management. Of course we have AWS Secrets Manager. Then there is another widely used product from HashiCorp called Vault. And AWS also has a less featured product called Parameter Store. Google has a product called Secret Manager that’s pretty similar to the AWS Secrets Manager, but it’s still in its infancy and is still being built. If you host on Google Cloud, you can check it out.
- AWS Secrets Manager is a natively hosted product on AWS, so it comes with Identity and Access Management (IAM) role provisioning for access. It includes auto-rotation and a built-in UI. Pricing is per secret and per API call, and items can be tagged for clarity in billing. The size of a secret is limited to 10KB.
- AWS Systems Manager Parameter Store comes with IAM role provisioning and is included free with other services. However, there is no auto-rotation, and the size limit is 4KB.
- HashiCorp Vault is an open-source product that offers enterprise versions. Access is controlled per path and has a “dynamic secret” feature to create the secret only when used. The price is the cost of maintaining infrastructure to hold the Vault secrets (VPN) and other operational costs. There is no limit to the size of secrets.
Managing secret management costs
My Stride client had started out with its secrets on Vault, but they saw that the operational costs were not lining up. As more and more microservices were built, this meant that more and more people needed provisioning to be set up in Vault. That's when we decided to move to AWS Secrets Manager, because it came with features like auto-rotation that made it more valuable for the cost.
Just as with anything else in AWS, you can save money by really reading the documents and finding better ways to use their products. For example, you can create fewer secrets using the SecretString, and setting these per environment. So instead of having 10 secrets for staging and 10 secrets for production, you can have one secret for each but use a SecretString, which is basically JSON. Something to keep in mind with this method is that you do have to send the full SecretString each time you change it, if you’re using the command-line interface (CLI). And you have to explicitly set keys if using auto-rotation; AWS isn’t going to know the keys for your particular secrets. So there is more up-front work, but in the end, it will save you money.
Things to keep in mind when using AWS Secrets Manager
- Access policy: As with anything, you’ll want to consider access carefully and set a policy early.
- Naming conventions: If you have a huge organization, you’re going to want to think about naming conventions. The AWS UI is a little cumbersome to search through, so you’ll want a key that indicates your service at the beginning, as this will make it a lot easier to filter. For example, you might use the pattern environment/service/secret so that you will see items sorted accordingly in the Secrets Manager Console.
- Sharing secrets across devices: There was an issue that came up at my Stride client where one service was using a secret, and the other was setting the secret, so it was a password just between the two services. And then the name got changed while secrets were being reevaluated, and the change wasn’t communicated downstream to the people using the service. So you want to be sure if there are shared secrets that you have a shared policy, such as “set it and forget it.”
- Keeping dev and production secrets in sync as you add new ones: You can solve this with a continuous integration (CI) pipeline. At my Stride client, there was a check such that if you added a secret to one file, all the files in the Kubernetes configurations were changed and set, as well. That’s a fail-safe for fault-free deployment.
If this introduction has piqued your interest, you can read some of these awesome documents on the AWS Secrets Manager page.
Melissa Wahnish
