Use Helm Secrets To Save Password in Code

5 min readOct 22, 2021

The password by software development is a critical information, and it should be good protected. Where should save the secrets, is mostly separate space than the source code. But during the development process, make all developer can use and manage the secrets like the source will be make the live easier.

Rolle

Developer who create or setup the initial password of Redis server. He should use the build server public GPG key to encrypte the secret. If His GPG key is allowed in SOPS config .sops.yaml, he can also decrypt the secret locally.

Jenkins CI/CD server, in the build process the k8s secret will be decoded by helm secrets and deploy to K8S

Let’s Go

Here is our example of a Redis Deployment on K8S. We use the Helm Charts with plugin Helm Secrets to deploy the secrets on K8S. The Redis password will be encrypt and saved in code repository with history.

Tools

Let us install the tools we need, before we do the deployment with Helm on K8s, we have to install it on our Mac

GPGTools

Same like helm I installed gpg also via homebrew

brew install gnupg

For other OS, you can find on gnupg.org

SOPS: Secrets OPerationS

sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.
— from https://github.com/mozilla/sops —

To install it on mac is the same in homebrew

❯ brew install sops

helm

For MacOS I used here homebrew to install Helm

❯ brew install helm # install
❯ helm version # check you install version
*version.BuildInfo{Version:"v3.7.1", GitCommit:"1d11fcb5d3f3bf00dbe6fe31b8412839a96b3dc4", 
GitTreeState:"clean", GoVersion:"go1.17.2"}*

For other OS you can find doc direct in helm documentation online.

Helm secrets plugin

To install helm secrets plugin use the helm plugin manager is recommand

helm plugin install <https://github.com/jkroepke/helm-secrets> --version v3.9.1

You can also find other install method on the project page.

Make Helm secret ready

You should have the public key from jenkins build server. If not, you can get it from your Jenkins admin, or even create a GPG Key pair for your self.

To generate or import an GPG Key you can find a lot of documentation online. Here I just show you a way to create a key pair for yourself.

❯ gpg --gen-key
gpg (GnuPG) 2.3.2; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.Note: Use "gpg --full-generate-key" for a full featured key generation dialog.GnuPG needs to construct a user ID to identify your key.Real name: banana
Email address: banana@mail.com
You selected this USER-ID:
    "banana <banana@mail.com>"Change (N)ame, (E)mail, or (O)kay/(Q)uit? o
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: key EFCEA5BFD7A0BEF5 marked as ultimately trusted
gpg: revocation certificate stored as '/Users/xxxxx/.gnupg/openpgp-revocs.d/BADA29B0BE78160F45B0AC03EFCEA5BFD7A0BEF5.rev'
public and secret key created and signed.pub   ed25519 2021-10-22 [SC] [expires: 2023-10-22]
      BADA29B0BE78160F45B0AC03EFCEA5BFD7A0BEF5
uid                      banana <banana@mail.com>
sub   cv25519 2021-10-22 [E] [expires: 2023-10-22]

This key hash value BADA29B0BE78160F45B0AC03EFCEA5BFD7A0BEF5 will be used in the next step.

Create Helm Chart

We create a helm chart for our simple secret

helm create redis-secret

After that we got a helm chart with simple template structure

mychart
|-- Chart.yaml
|-- charts
|-- templates
|   |-- NOTES.txt
|   |-- _helpers.tpl
|   |-- deployment.yaml
|   |-- ingress.yaml
|   `-- service.yaml
`-- values.yaml

But we just keep the folder structure, the most file will be deleted.

❯ rm templates/**/*.yaml❯ tree
.
├── Chart.yaml
├── charts
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   └── tests
└── values.yaml

Coding Redis Secrets

Now let use create a secret manifest file in templates folder via kubectl

❯ kubectl create secret generic redis -o yaml --dry-run=client \\
--from-literal password=123456 \\
> templates/redis-secret.yaml

The result in redis-secret.yaml looks like:

# redis-secret.yaml
apiVersion: v1
data:
  password: MTIzNDU2
kind: Secret
metadata:
  creationTimestamp: null
  name: redis-secret

Before we can encrypt this password, we need SOPS config file .sops.yaml

❯ touch .sops.yaml

This file contains the gpg key hash, what we created/imported previous step.

creation_rules:
  # List of GPG keys (fingerprints) allowed to (de|en)crypt secrets
  # managed with SOPS. For details see: docs/secrets.md
  #
  # * `banana <banana@mail.com>`  key-id: BADA29B0BE78160F45B0AC03EFCEA5BFD7A0BEF5
  - encrypted_regex: '^(data)$'
    pgp: >-
      BADA29B0BE78160F45B0AC03EFCEA5BFD7A0BEF5

Now we can use the helm encrypt our redis-secret.yaml.

❯ helm secrets enc templates/redis-sec.yaml

Now the redis-secret.yaml looks like:

apiVersion: v1
data:
    password: ENC[AES256_GCM,data:2JzZDr7IDFY=,iv:UkhNIb5ahnJw0/4Pj1JkPniZ57Zzaz10+rDrwMYvuzE=,tag:BjQ9XXFGyGgtZ9K7rrROcg==,type:str]
kind: Secret
metadata:
    creationTimestamp: null
    name: redis-secret
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2021-10-22T15:29:17Z"
    mac: ENC[AES256_GCM,data:wg+Z1MI0l89IdTKdx4GIqgIDUrGw8IhcIXzOoqoTztyqd7ugXvSHVmeIbaaOA2SzUEPQd3023uK/KYksDyWhrmcAbxcEQfvP2S6rXNv6MHfMw727llLm9QvIWh5ch5CN0Z5jqi1lWVFqwcaGViYWN/Th/uZec+Nwz8JDoYAftPY=,iv:OyuzJbS3FgORK53jon2l1gUpD9XDN5OdtQvKRUCmVTs=,tag:sMur1xTTWkCfgfJ66XG4eg==,type:str]
    pgp:
        - created_at: "2021-10-22T15:29:17Z"
          enc: |
            -----BEGIN PGP MESSAGE-----            hF4DYTnVj1/p00gSAQdARx36kFjN4wCZsZgpBt6+AVIF76IMtlHAYfhX7SG17Tsw
            5nf7xrlOH5mkH6X1/v6OLTEmWPjeGp2J/cNFDzDpWS62D44KINUu9399g1dVOKr8
            1GgBCQIVtIjdJHG0gsMLCmB1oreLFeG8ulzWvhGf2rbr1SmBgVbFLdw4tQ/mLZh4
            CrhXPyR5nxq1UU7/VEjC5h7ye6AxDS7ktuTd/tk/6FzDXPvx5t4QSlR3kSZi0C80
            gmENh1mIOPdNgQ==
            =AT9+
            -----END PGP MESSAGE-----
          fp: BADA29B0BE78160F45B0AC03EFCEA5BFD7A0BEF5
    encrypted_regex: ^(data)$
    version: 3.7.1

Now this file could be added and committed into the code repository and shared with teams.

❯ helm secrets upgrade --install redis-cache .

SOPS Issue by decrypt

❯ sops -d redis-secret.enc.yaml > redis-secret.new.yaml
Failed to get the data key required to decrypt the SOPS file.Group 0: FAILED
  5CF08262C7B4855BDA927D73469D7126DCD86331: FAILED
    - | could not decrypt data key with PGP key:
      | golang.org/x/crypto/openpgp error: Could not load secring:
      | open /Users/qiang.li/.gnupg/secring.gpg: no such file or
      | directory; GPG binary error: exit status 2Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

Solution:

change the GPG_TTY into the .zshrc. If you have also installed p10k theme, make sure, your environment GPG_TTY is before the p10k configs in your .zshrc.

GPG_TTY=$(tty)
export GPG_TTY