At first glance, storing Kubernetes secrets seems simple enough. Kubernetes stores them automatically in etcd, a key-value store. And, Kubernetes can be configured to encrypt these secrets by default to help keep them secure.
That’s all good and well if you’re content with keeping your secrets inside Kubernetes. But, what if you want to keep external copies of your secrets, rather than just relying on Kubernetes and etcd? What if, in particular, you believe in GitOps, and want to be able to use Git to manage everything in your Kubernetes deployment?
That’s where things get trickier. Kubernetes itself provides no native way to store secrets in a Git repository. But, fortunately, this can be done easily enough – and quite securely – by using third-party tools to bridge the gap between the Kubernetes’ secrets framework and your Git repository.
In this article, we walk through different solutions for one approach to storing Kubernetes secrets using a Git repository.
Why store Kubernetes secrets in Git?
Why might you want to store Kubernetes secrets in a Git repository in the first place? There are several potential reasons:
- To ensure that your secrets are backed up and will remain intact even if your Kubernetes environment is wiped out for some reason. Rebuilding the environment will be easier if you can import pre-existing secrets, rather than having to rebuild them all from scratch.
- To provide an independent secrets registry where your secrets will remain secure even if your Kubernetes environment is hacked.
- To have a revision history of changes to your secrets, which you can use for auditing purposes if the need arises.
- In some cases, to make it possible to modify secrets without having to work from within the Kubernetes environment.
You can achieve most of these goals using an approach other than Git, of course. Git isn’t the only way to store credentials outside of Kubernetes. (Plenty of third-party key management tools like AWS KMS and HashiCorp Vault support Kubernetes secrets.)
Git, however, has the advantages of robust built-in version-control functionality, compatibility with GitOps workflows and being open-source (which reduces risks of lock-in). Therefore, for some teams, storing Kubernetes secrets in a plain-old Git repository may be preferable to relying on a conventional key management tool.
The challenges of Kubernetes secrets in Git
Storing Kubernetes secrets in Git presents some challenges. The most obvious is that, since secrets contain sensitive information (like passwords and SSH keys), putting them in a Git repository creates some security risks. Even if the repository is private, it’s not exactly a best practice to put passwords and similar data in it, unless you encrypt them.
A second challenge is getting the secrets out of Git, if and when you need to do so. If you’re using Git simply as a backup solution for your secrets, this may not matter. But, if you want Kubernetes to be able to pull secrets directly from your Git repository, you need a secure way for it to interact with Git for that purpose.
Solutions for Git-based secrets management with Kubernetes
Fortunately, several tools exist to help you address the challenges described above and store secrets in Git in a way that’s secure and accessible to Kubernetes.
SealedSecrets
One such tool is SealedSecrets, which is developed by Bitnami. It works by allowing you to encrypt ordinary Kubernetes secrets by turning them into what the tool calls a SealedSecret. Because the data inside SealedSecrets is encrypted, it’s safe to store SealedSecrets inside a regular Git repository.
To decrypt secrets, SealedSecrets runs a controller within the Kubernetes cluster. Once decrypted, the data from a SealedSecret can be used just like any other Kubernetes secret.
This architecture makes it easy to both secure secrets and to import them into a cluster from Git. Perhaps the biggest drawback is that you have to rely on the SealedSecrets controller to decrypt the data inside a SealedSecret, which could create challenges in the event that your cluster crashes and the controller is no longer available to decrypt secrets stored in Git.
Kamus
Kamus works in a similar fashion to SealedSecrets. It lets you encrypt a Kubernetes secret so it can be stored in a Git repository. Then, the encrypted secret can be imported into your cluster, decrypted and used as a normal secret.
The major difference between Kamus and SealedSecrets is that Kamus relies on an external tool for actually encrypting the secrets. This tool can be a standard AES algorithm, or a cloud-based key management service. (Currently, Kamus supports AWS KMS, Azure KeyVault and Google Cloud KMS.)
This approach could be preferable for teams that want some choice in exactly how their secrets are encrypted. At the same time, however, it adds another moving piece to the architecture, making configuration and management of the tool just a bit more complicated.
SOPS
A third approach is to use SOPS, a general-purpose secrets management tool from Mozilla, which encrypts secrets inside a Git repository, then decrypts them prior to deployment of a workload into Kubernetes.
SOPS is not designed specifically for managing Kubernetes secrets (although it certainly supports them), so this approach requires more custom setup than the purpose-built tools described above. But SOPS is a widely used, well-supported open-source tool. And, SOPS can be used for storing other types of secrets, not just Kubernetes secrets. For that reason, a SOPS-based approach might work well for organizations that need to manage lots of different types of secrets using a single management tool and Git.
Securely Keeping Kubernetes Secrets in a Culture of GitOps
For many teams – especially the GitOps-minded – keeping Kubernetes secrets solely inside Kubernetes isn’t ideal. Storing them in Git provides reliability and manageability advantages. There are various tools available to help export Kubernetes secrets into Git repositories securely, then import them back to Kubernetes when necessary. These tools make integrating secrets into the rest of your Git-based workflow easier than it may first appear.
