Ephemeral Database Credentials
How To Manage Credentials?
There are many ways to manage credentials. Some better than others. A simple way is to keep them in a config file. Do you commit this alongside your code? Maybe storing them in a database is safer? How about encrypting them for extra security? But where do you store the credentials for the database that stores the credentials? And where do you store that encryption key? It’s turtles all the way down!
This is what HashiCorp Vault was designed for.
HashiCorp Vault
We have been using HashiCorp for about 18 months. It provides a secure, encrypted place to store our credentials and any other general secrets. Access is granted via integration with either our SSO (Okta), Kubernetes, AWS or numerous other ways. Policies allow us to provide granular access across all of our application and engineers.
If you’re wondering where the encryption keys are stored, they are constructed after the Vault server starts up by multiple engineers providing their own Shamir’s Secret Sharing based keys. The encryption keys never leave the process they run in and never touch the disk.
Vault makes great use of token and TTLs. Tokens can create tokens, orphaned tokens, wrapped tokens and one-time usage tokens. Ultimately, all tokens expire after some TTL or can be manually revoked. TTLs themselves are based on a hierarchy of settings which can get quite complex.
Audibility provides insights into who is gaining access to our systems.
The number of integrations Vault provides is impressive and means there’s always another level to take your Vault usage to.
Persistent Credentials
Even with something like Vault, which has improved the security of our most secret of secrets, a secret is only truly secret when nobody knows it. As soon we read a secret out of Vault it becomes less secure. Credentials may coexist with a long-lived application or used by an engineer who needs access to the production database. They may be shared over Slack or left on a jump host. Over time these credentials spread, making them less secure and harder to rotate.
Role-based credentials with differing levels of access can help, but it also means you have more credentials to manage. Ultimately, you will have some high-level credentials that everyone will want when the poop hits the fan. The cat will come out of the bag.
Ephemeral Credentials
What if credentials simply expired after a certain amount of time? That way they could never get too old and it would limit the distance them could spread through general usage.
If it was easy for anyone to request these expiring credentials, then there would be less incentive to pass these credentials around.
Vault Database Secrets Engine
With the Vault Database Secrets Engine, we’re able to take our usage of Vault to the next level. It integrates with our databases to create ephemeral database credentials based role and policies.
Vault will provision new credentials within your database using SQL statements that you provide. Once they expire (or are revoked) it will run a second set of user-defined SQL commands to remove those credentials.
Provisioning credentials to do a full spectrum of things in your database requires either the root database user or high-level database privileges. If your goal is to reduce long-lived credentials to a minimum, then handing over the root credentials to Vault makes sense. Vault even offers to rotate the root credentials for you!
You need to consider how much you trust Vault and what you will do it you lose the root password to your database. Since we’re on AWS RDS, it’s a case of going to the web console and resetting this password.
Secrets Engine Setup
Setting up the secrets engine is fairly straight-forward and can be done with just three Vault CLI commands as shown in these Vault docs…
We use Terraform to do this for the 22 databases we manage this way.
Expiring Credentials
“Pop quiz, hotshot. There’s a password in your app. Once the app goes to 50 requests per second, the password is critical. In an hour I’m gonna delete the password. What do you do? What do you do?” — Your Vault server
So you’ve setup Vault and integrated it with your applications, but now the credentials you gave your application will expire. How do you handle this?
Timely Renewals
Your application will need new credentials before the old ones expire. There are a few ways you can manage this.
HashiCorp provides some client libraries you can embed in your application’s code to request credentials from Vault. You can set a timer in your code to request new credentials before the old ones expire.
Supporting Polyglot Applications
At Bench we have services written in several languages, such as Scala/Java, Golang, Node.js and Python. Even the ones that are more common (Scala) differ in complexity and the libraries they use. Rolling out ephemeral credentials across all our services would have been a fair bit of code churn if we were to use Vault’s client libraries. Also, we were not completely sure if this would turn out to be a horrible experiment and need to be rolled back. Therefore, we decided to implement this in a consistent way outside of the applications.
Persistent database credentials were already being read from Vault prior to application startup and passed to the application. So it was a matter of requesting ephemeral, not persistent, credentials from Vault during this process. Rollback would just be a switch back to the persistent credentials, assuming we hadn’t deleted them.
External provisioning of credentials meant we needed to restart the application to create and inject new credentials before the application’s current credentials were due to expire. Since all our applications now run in Kubernetes, and we chose a consistent TTL for all application ephemeral credentials, we could indiscriminately look for pods nearing that TTL in age and gracefully terminate them. Kubernetes would immediately start new pods to replace them. And since this is Kubernetes, we can run this simple script as a Kubernetes CronJob at a reasonable cadence.
Support Engineers
Engineers can also self-request access to any production or non-production database for an appropriate amount of time (defined as a TTL) using a simple script we wrote to interact with Vault. For instance, we give a shorter TTL for “write” credentials than we do for “read” and for “admin” credentials it’s shorter still. In our configuration, “write” does not allow schema mutations, whereas “admin” does.
Here’s an example of what that script interface looks like.
db-creds read my-db
db-creds write my-db
db-creds admin my-db
Credentials are printed to the terminal, along with expiry information and a lease ID (used for revocation). The engineer generally copy-and-pastes the credentials into their database GUI.
Security vs Convenience
One challenge of supporting engineers is getting the right balance between security and convenience. The longer credentials live, the less secure they become. But engineers dislike requesting credentials too often. By making the process as efficient as possible, we’ve landed on a good compromise, but this is something we plan to review in the future.
Conclusion
Using ephemeral credentials has given us one less thing to worry about… old credentials that may have been shared or compromised. While there is still a window of time in which these credentials can be compromised, the likely distance they will travel within that window is much less. Vault audit log tells us who, or what was the initial requestor of the credentials. Once identified, compromised credentials are revoked without affecting any other application instances or database users.
—
Written by Phil Whelan, former Bench Principal Engineer.
If you are interested in learning more about Bench Accounting or a career with our Engineering team, apply here: https://bench.co/careers/
Originally published at https://medium.com on May 7, 2019.
