HelmRelease Custom Resource

Each release of a chart is declared by a HelmRelease resource. The schema for these resources is given in the custom resource definition. They look like this:

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: rabbit
  namespace: default
spec:
  releaseName: rabbitmq
  targetNamespace: mq
  timeout: 300
  resetValues: false
  forceUpgrade: false
  chart:
    repository: https://kubernetes-charts.storage.googleapis.com/
    name: rabbitmq
    version: 3.3.6
  values:
    replicas: 1

The releaseName will be given to Helm as the release name. If not supplied, it will be generated by affixing the HR namespace and the target namespace to the resource name. In the above example, if releaseName were not given, it would be generated as default-mq-rabbitmq. Because of the way Helm works, release names must be unique in the cluster.

If you don’t supply the targetNamespace, the release will be installed in the same namespace as the HelmRelease object.

The chart section gives a pointer to the chart; in this case, to a chart in a Helm repo. Since the helm operator is running in your cluster, and doesn’t have access to local configuration, the repository is given as a URL rather than an alias (the URL in the example is what’s usually aliased as stable). The name and version specify the chart to release.

The timeout sets the timeout value for the helm install or upgrade. If you don’t supply it, it is set to 300.

The resetValues, if set to true, will reset values on helm upgrade.

The forceUpgrade, if set to true, will force Helm upgrade through delete/recreate

The values section is where you provide the value overrides for the chart. This is as you would put in a values.yaml file, but inlined into the structure of the resource. See below for examples.

Why use URLs to refer to repositories, rather than names? ^

A HelmRelease must be able to stand on its own. If we used names in the spec, which were resolved to URLs elsewhere (e.g., in a repositories.yaml supplied to the operator), it would be possible to change the meaning of a HelmRelease without altering it. This is undesirable because it makes it hard to specify exactly what you want, in the one place; or to read exactly what is being specified, in the one place. In other words, it’s better to be explicit.

Using a chart from a Git repo instead of a Helm repo

You can refer to a chart from a git repo, rather than a chart repo, with a chart: section like this:

spec:
  chart:
    git: [email protected]:fluxcd/flux-get-started
    ref: master
    path: charts/ghost

In this case, the git repo will be cloned, and the chart will be released from the ref given. If not supplied, the operator uses the value specified by the --git-default-ref flag (which defaults to master). Commits to the git repo may result in releases, if they update the chart at the path given.

Note that you will usually need to provide an SSH key to grant access to the git repository. The example deployment shows how to mount a secret at the expected location of the key (/etc/fluxd/ssh/). If you need more than one SSH key, you’ll need to also mount an adapted ssh_config; this is also demonstrated in the example deployment.

Notifying Helm Operator about Git changes

The Helm Operator fetches the upstream of mirrored Git repositories with a 5 minute interval. In some scenarios (think CI/CD), you may not want to wait for this interval to occur.

To help you with this the Helm Operator serves a HTTP API endpoint to instruct it to immediately refresh all Git mirrors.

$ kubectl -n flux port-forward deployment/flux-helm-operator 3030:3030 &
$ curl -XPOST http://localhost:3030/api/v1/sync-git
OK

Note: the HTTP API has no built-in authentication, this means you either need to port forward before making the request or put something in front of it to serve as a gatekeeper.

Supplying values to the chart

You can supply values to be used with the chart when installing it, in two ways.

.spec.values

This is a YAML map as you’d put in a file and supply to Helm with -f values.yaml, but inlined into the HelmRelease manifest. For example,

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
# metadata: ...
spec:
  # chart: ...
  values:
    foo: value1
    bar:
    baz: value2
    oof:
    - item1
    - item2

.spec.valuesFrom

This is a list of secrets, config maps (in the same namespace as the HelmRelease) or external sources (URLs) from which to take values.

The values are merged in the order given, with later values overwriting earlier. These values always have a lower priority than those passed via the .spec.values parameter.

This is useful if you want to have defaults such as the region, clustername, environment, a local docker registry URL, etc., or if you simply want to have values not checked into git as plaintext.

Config maps

spec:
  # chart: ...
  valuesFrom:
  - configMapKeyRef:
      # Name of the config map, must be in the same namespace as the
      # HelmRelease
      name: default-values  # mandatory
      # Key in the config map to get the values from
      key: values.yaml      # optional; defaults to values.yaml
      # If set to true successful retrieval of the values file is no
      # longer mandatory
      optional: false       # optional; defaults to false

Secrets

spec:
  # chart: ...
  valuesFrom:
  - secretKeyRef:
      # Name of the secret, must be in the same namespace as the
      # HelmRelease
      name: default-values # mandatory
      # Key in the secret to get thre values from
      key: values.yaml     # optional; defaults to values.yaml
      # If set to true successful retrieval of the values file is no
      # longer mandatory
      optional: true       # optional; defaults to false

External sources

spec:
  # chart: ...
  valuesFrom:
  - externalSourceRef:
      # URL of the values.yaml
      url: https://example.com/static/raw/values.yaml # mandatory
      # If set to true successful retrieval of the values file is no
      # longer mandatory
      optional: true                                       # optional; defaults to false

Chart files

spec:
  # chart: ...
  valuesFrom:
  - chartFileRef:
      # path within the helm chart (from git repo) where environment-prod.yaml is located
      path: overrides/environment-prod.yaml # mandatory
      # If set to true successful retrieval of the values file is no
      # longer mandatory
      optional: true                                       # optional; defaults to false

Rollbacks

From time to time a release made by the Helm operator may fail, it is possible to automate the rollback of a failed release by setting .spec.rollback.enable to true on the HelmRelease resource.

Note: a successful rollback of a Helm chart containing a StatefulSet resource is known to be tricky, and one of the main reasons automated rollbacks are not enabled by default for all HelmReleases. Verify a manual rollback of your Helm chart does not cause any problems before enabling it.

When enabled, the Helm operator will detect a faulty upgrade and perform a rollback, it will not attempt a new upgrade unless it detects a change in values and/or the chart.

Configuration

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
# metadata: ...
spec:
  # Listed values are the defaults.
  rollback:
    # If set, will perform rollbacks for this release.
    enable: false
    # If set, will force resource update through delete/recreate if
    # needed.
    force: false
    # Prevent hooks from running during rollback.
    disableHooks: false
    # Time in seconds to wait for any individual Kubernetes operation.
    timeout: 300
    # If set, will wait until all Pods, PVCs, Services, and minimum
    # number of Pods of a Deployment are in a ready state before
    # marking the release as successful. It will wait for as long
    # as the set timeout.
    wait: false

Reinstalling a Helm release

If a Helm release upgrade fails due to incompatible changes like modifying an immutable field (e.g. headless svc to ClusterIP)you can reinstall it using the following command:

$ kubectl delete hr/my-release

When the Helm Operator receives a delete event from Kubernetes API it will call Tiller and purge the Helm release. On the next Flux sync, the Helm Release object will be created and the Helm Operator will install it.

Authentication

At present, per-resource authentication is not implemented. The HelmRelease definition includes a field chartPullSecret for attaching a repositories.yaml file, but this is ignored for now.

Instead, you need to provide the operator with credentials and keys (see the following Authentication for Helm repos section for how to do this).

Authentication for Helm repos

As a workaround, you can mount a repositories.yaml file with authentication already configured, into the operator container.

Note: When using a custom repositories.yaml the default that ships with the operator is overwritten. This means that for any repository you want to make use of you should manually add an entry to your repositories.yaml file.

To prepare a file, add the repo locally as you would normally:

helm repo add <URL> --username <username> --password <password>

You need to doctor this file a little, since it will likely contain absolute paths that will be wrong when mounted inside the container. Copy the file and replace all the cache entries with just the filename.

cp ~/.helm/repository/repositories.yaml .
sed -i -e 's/^\( *cache: \).*\/\(.*\.yaml\)/\1\2/g' repositories.yaml

Now you can create a secret in the same namespace as you’re running the Helm operator, from the repositories file:

kubectl create secret generic flux-helm-repositories --from-file=./repositories.yaml

Lastly, mount that secret into the container. This can be done by setting helmOperator.configureRepositories.enable to true for the flux Helm release, or as shown in the commented-out sections of the example deployment.

Azure ACR repositories

For Azure ACR repositories, the entry in repositories.yaml created by running az acr helm repo add is unsufficient for the Helm operator. Instead you will need to create a service principal and use the plain text id and password this gives you. For example:

- caFile: ""
  cache: <repository>-index.yaml
  certFile: ""
  keyFile: ""
  name: <repository>
  url: https://<repository>.azurecr.io/helm/v1/repo
  username: <service principal id>
  password: <service principal password>

Authentication for Git repos

In general, it’s necessary to have an SSH key to clone a git repo. This is sometimes (e.g., on GitHub) called a “deploy key”. To use a chart from git, the Helm Operator needs a key with read-only access.

To provide an SSH key, put the key in a secret under the entry identity, and mount it into the operator container as shown in the example deployment. The default ssh_config expects an identity file at /etc/fluxd/ssh/identity, which is where it’ll be if you just uncomment the blocks from the example.

If you’re using more than one repository, you may need to provide more than one SSH key. In that case, you can create a secret with an entry for each key, and mount that as well as an ssh_config file mentioning each key as an IdentityFile.