GitOps : an overview of practices and tools
This article is an introduction to GitOps principles. We will take Kubernetes as an example.
By Arthur Outhenin-Chalandre
Targeted audience: Kubernetes Administrator
Infrastructure As Code & Gitops
The concept is similar to the infrastructure as code with clearly defined constraints ensuring good practices.
The infrastructure as code, and therefore Gitops, is largely based on a declarative specification of an infrastructure. This is achieved by using tools like Ansible, Chef/Puppet, Terraform, Salt and of course Kubernetes.
It means that manual operations via
ssh or even
kubectl is the new
ssh !) are prohibited. Any changes must be described in a declarative manner with either a provisioning or a deployment tool. This point is described in the 10th point of the 12factor.
Since we don’t make any manual changes, we can easily replicate our deployments. We can talk about “immutable deployment”.
You may have guessed it, GitOps is based on git, but we can draw a parallel with any other decent versioning tools and we might not use Git at all to get a similar result.
The central aspect of Gitops is based on the fact that we will consider that our Git repository is our only source of truth, this prohibits the manual operations described above. And it also encourages us to use only git to perform our usual deployments via a “Continuous Deployment” fashion.
The concept of convergence is also an important aspect of GitOps. Since our git repository is our source of absolute truth, we will seek to converge the state of our deployments with the state described in git. This includes active supervision of our infrastructure and reporting of discrepancies.
Git good practices
All our environments must be described in git in a declarative way. This brings the advantages of a classic VCS (Version Control System), including the possibility of having rollbacks and a clear history of our modifications.
To this can be added the first point of the 12factor prohibiting the multiplication of branches or repositories describing multiple environments of the same deployment. For instance, we can’t maintain a “pre-production” branch with the specificities of your pre-production environment.
Instead, good practices recommend describing all environments in your main branch. So you will have to update all your environments at the same time and you will no longer have to keep the specificities of your environment apart.
This does not prevent you from creating other branches for your new features. You can of course apply workflows such as Git Flow, Github Flow, GitLab Flow or any other alternative that are more suitable for your usage and your team.
On Kubernetes, you can use “Kustomize” to do this. It is native to “kubectl” and offer an overlay system. In short, you can describe a “base” and apply a set of “patches” to it. This will help to switch from a common base to a specific environment with a minimum set of patches.
Different strategies of Continuous Deployment
For the “Continuous Deployment” step, we can consider two different strategies:
- the more traditional “push” strategy which involves using your forge to “push” modification you have just made in git.
- the “pull” strategy which let the infrastructure manage itself and will on its own “pull” the git repository and apply it itself. This allows to separate the CD part from the forge, which implies fewer vendors lock-in. However, some solutions require the platform on which our git repository must be hosted (i.e., GitHub).
The different tools on Kubernetes
GitLab CI allows you to use “CD” as part of the strategy that we previously called “push”. It is an interesting tool especially if it is already your current forge. GitLab offers integration with Kubernetes clusters. You will simply need to add your Kubernetes cluster credentials to the GitLab project. You should note that GitLab provides a pre-made CI/CD environment that may be sufficient for you, you can find more information about that here.
Jenkins, just like GitLab CI is an interesting tool especially if you are already using it as a forge. It is also part of the strategy that we called “push”.
JenkinsX is also based on the “push” strategy but offer native integration into Kubernetes. It offers quickstart environments with “preview environments”, but also many very convenient features like dev pods, which is pod in your Kubernetes cluster synchronized with your code in your local filesystem using “Skaffold” and “ksync”. This allows to considerably reduce differences between dev and production.
Despite these extremely useful features, JenkinsX is still too much linked to Github which is a problem for us at Objectif Libre since we are GitLab users. However, they are actively working on Github’s emancipation.
Flux is the Weave’s CD tool (which is under the governance of the CNCF as a “Sandbox Project”).
It is a Kubernetes operator that synchronizes the Kubernetes manifests contained in a Git repository and deploys them. It is, therefore, part of the strategy that we called “pull”.
Flux can be extended and we can add numerous templating tools for manifest like kustomize or jsonnet. Flux can also pull and watch docker registry. In case of changes on a registry it can commit on the git repository to update the tag of the docker image used in a manifest. It can also alert (e.g., via Slack) when there is a discrepancy between the manifests stored on Git and the resources currently deployed on the Kubernetes cluster.
Flux supports only one Git repository per instance. There is an operator to deploy several instances of Flux, but it is no longer actively maintained. Fluxcloud, a tool developed by the community, offers the possibility to send Flux events on slack.
Since it is advised to separate Kubernetes manifests from the application code, all your manifests should be collected in a single Git repository, If this is not suitable for your use case, a solution like Argo CD would most likely be a better choice.
Argo CD is, like Flux, a Kubernetes operator synchronizing Kubernetes manifests stored on a Git repository. It is also part of the strategy that we called “pull”. Weave is also extensible and supports Helm, Kustomize and Jsonnet.
Argo CD also has a web UI and a CLI utility on which you can create and track Argo CD resources. Argo CD supports several “Applications” which are actually a grouping of manifests stored in a git repository. It is, threfore, not subject to the same limitations as Flux. It also supports alerts with “Jobs” created by Argo CD during certain events. For example, there is an event in case of a synchronization failure
SyncFail) and one in case of a synchronization success (
Argo CD stores application deployments via CRDs, It is therefore possible to integrate Argo manifests in the GitOps process.
Argo Flux is a very recent project announced shortly before the KubeCon US 2019 (November 2019). It aims to be the merging of Argo CD and Flux. The development and regular releases on Flux and Argo CD are still ongoing.
Weave, Intuit and AWS have jointly announced the project and will participate in its development.
The project it still at a very early stage and has, to this day, no release. You can find the project’s progress on the Gitops Engine repository.
The tool we have chosen for our GitOps deployments on Kubernetes is Argo CD because it can be deployed and configured itself in a GitOps process, i.e. in a declarative way with CRDs. It is also rather simple to use while offering essential features such as SSO via OIDC.
However, we will keep in our radar the evolution of tools such as Argo Flux and Jenkins.