Also see the issues labeled with
FAQ, which often
Flux is taking a long time to apply manifests when it syncs¶
If you notice that Flux takes tens of seconds or minutes to get through each sync, while you can apply the same manifests very quickly by hand, you may be running into this issue: fluxcd/flux#1422.
Briefly, the problem is that mounting a volume into
kubectl's caching, which makes it much much
slower. You may have used such a volume mount to override
$HOME/.kube/config, possibly unknowingly -- the Helm chart did this
for you, prior to
The remedy is to mount the override to some other place in the
filesystem, and use the environment entry
KUBECONFIG to point
kubectl at it. This is what the Helm chart now does, so fixing it
may be as easy as reapplying the chart if that's what you're using.
This is also documented in the FAQ.
fluxctl returns a 500 Internal Server Error¶
This usually indicates there's a bug in the Flux daemon somewhere -- in which case please tell us about it!
Flux answers everything with
git repo is not configured¶
This means Flux can't read from and write to the git repo. Check that
... you've supplied a git repo URL. If it's of the form
https://github.com/user/repothen you will need to use the SSH-style URL,
... the deploy key has read/write access to the repo. In GitHub, deploy keys are installed in the settings for a repository. To get the deploy key Flux is using, use
... that the host where your git repo lives is in
~/.ssh/known_hostsin the fluxd container. We prime the container image with host keys for
vs-ssh.visualstudio.com, but if you're using your own git server, you'll need to add its host key. See "Using a private Git host".
I'm using GCR/GKE and I keep seeing "Quota exceeded" in logs¶
GCP (in general) has quite conservative API rate limiting, and Flux's default settings can bump API usage over the limits. See fluxcd/flux#1016 for advice.
Flux doesn't seem to be able to use my imagePullSecrets¶
If you're using
kubectl v1.13.x to create them, then it may be due
to this problem. In
short, there was a breaking change to how
kubectl creates secrets,
that found its way into the Kubernetes 1.13.0 release. It has been
corrected in kubectl
so using that version or newer to create secrets should fix the
Why are my images not showing up in the list of images?¶
Sometimes, instead of seeing the various images and their tags, the
fluxctl list-images shows nothing. There's a number of
reasons this can happen:
- Flux just hasn't fetched the image metadata yet. This may be the case if you've only just started using a particular image in a workload.
- Flux can't get suitable credentials for the image repository. At
present, it looks at
imagePullSecrets attached to workloads, service accounts, platform-provided credentials on GCP, AWS or Azure, and a Docker config file if you mount one into the
fluxdcontainer (see the command-line usage).
- When using images in ECR, from EC2, the
NodeInstanceRolefor the worker node running
fluxdmust have permissions to query the ECR registry (or registries) in question.
.iam.allowContainerRegistry=true) both make sure this is the case.
- When using images from ACR in AKS, the HostPath
/etc/kubernetes/azure.jsonshould be mounted into the Flux Pod. Set
registry.acr.enabled=Truein the helm chart or alter the Deployment:If you encounter permission errors, you can alternatively create a secret
spec: containers: image: docker.io/fluxcd/flux ... volumeMounts: - name: acr-credentials mountPath: /etc/kubernetes/azure.json readOnly: true volumes: - name: acr-credentials hostPath: path: /etc/kubernetes/azure.json type: ""
acr-credentialsbased on the
azure.jsonfile and set
- Flux excludes images with no suitable manifest (linux amd64) in manifestlist
- Flux doesn't yet understand image refs that use digests instead of tags; see fluxcd/flux#885.
If none of these explanations seem to apply, please file an issue.
Why do my image tags appear out of order?¶
You may notice that the ordering given to image tags does not always correspond with the order in which you pushed the images. That's because Flux sorts them by the image creation time; and, if you have retagged an older image, the creation time won't correspond to when you pushed the image. (Why does Flux look at the image creation time? In general there is no way for Flux to retrieve the time at which a tag was pushed from an image registry.)
This can happen if you explicitly tag an image that already exists. Because of the way Docker shares image layers, it can also happen implicitly if you happen to build an image that is identical to an existing image.
If this appears to be a problem for you, one way to ensure each image build has its own creation time is to label it with a build time; e.g., using OpenContainers pre-defined annotations.
What is the "sync tag"; or, why do I see a
flux-sync tag in my git repo?¶
Flux keeps track of the last commit that it's applied to the cluster,
by pushing a tag (controlled by the command-line flags
--git-label) to the git repository. This gives
it a persistent high water mark, so even if it is restarted from
scratch, it will be able to tell where it got to.
Technically, it only needs this to be able to determine which image
releases (including automated upgrades) it has applied, and that only
matters if it has been asked to report those with the
flag. Future versions of Flux may be more sparing in use of the sync
Flux fails with an error log similar to couldn't get resource list for example.com/version: the server is currently unable to handle the request¶
This means your Kubernetes cluster fails to respond to list queries for resources in example.com/version.
If the error is transient, Flux will work once the error recedes.
However, the error won't normally go away since most of the time it's caused by a misconfiguration of your cluster.
For instance, you can run into this problem:
* When a
Kubernetes Webhook server
is removed without removing its Webhook definition.
* When a custom resource definition (CRD) is not available due to
We recommend trying to address the root cause by fixing your cluster configuration. In the examples above, you would need to remove the Webhook definition or add the CRD.
However, fixing your cluster configuration may not always be possible. The
problem is common enough that Flux provides a flag called
--k8s-unsafe-exclude-resource. The name says it all, you should only use it
if you know what you are doing.
--k8s-unsafe-exclude-resource will tell Flux to avoid querying the cluster
for those resources. This in turn means that Flux won't take into account those
excluded cluster resources when syncing. This can cause excluded resources:
* to be unexpectedly overwritten by their corresponding definition in
Git during a sync (even if they are annotated with
flux.weave.works/ignore: "true" on the cluster-side).
* not to be garbage-collected.
The rule of thumb is that you can use
resources not matching any manifests in your Git repository.