By Stephen Watts November 09, 2023
Simply put, GitOps applies DevOps practices like version control and CI/CD to infrastructure management. The name GitOps, coined by Weavewerks, is from Git, an open-source version control system that allows teams to manage infrastructure using the same tools and orchestration processes they use for software development.
Speed and scalability drive the development of modern software applications, and mature DevOps organizations can deploy code to production hundreds of times a day. DevOps teams are able to do this through automation enabled by best practices such as version control, code review and CI/CD pipelines. But while the software development lifecycle (SDLC) has been largely automated, infrastructure management still relies on mostly manual processes and specialized teams. This makes it challenging to provision and configure cloud resources with the speed and scale continuous deployments require.
GitOps helps by automating the process of provisioning infrastructure. Rather than manually configuring an environment every time a new component is needed, GitOps teams use configuration files stored as code to automate the creation of environments using machine-readable definition files. These configuration files generate the same infrastructure each time they’re deployed, ensuring consistency across IT environments.
In this blog post we’ll look at how GitOps works, why you should use it and how you can apply it. We’ll also look at how to get started with GitOps along with some best practices to help set you up for success.
GitOps Basics
GitOps works by using a Git-based source code management system to build code-based infrastructure. It is an evolution of Infrastructure as Code (IaC), a DevOps practice used to provision and configure testing and deployment environments using scripts that are treated as part of the application’s code. The IaC process starts with a developer writing scripts that define the infrastructure configuration parameters. The scripts can be written using a declarative or imperative approach. Essentially, declarative programming lets the user say what infrastructure resources they want while imperative programming lets them say how they want the infrastructure resources created.
The declarative approach requires the user to “declare” what they want the infrastructure to look like by specifying a list of resources and the properties they should have. The IaC tool reads the script then installs and configures all the infrastructure components and manages the code version deployed to them. In the imperative approach, the user specifies a list of commands they want the IaC tool to run to provision the infrastructure step by step. The configuration scripts are sent to a code repository that supports version control, allowing the files to be easily reviewed, edited, copied and shared. When someone edits a file, merge requests and code review workflows validate the changes. Finally, the IaC platform follows the developer’s instructions to create and configure the infrastructure.
GitOps is essentially IaC using Git as the version control system for all infrastructure configurations. GitOps workflows consist of three primary components:
- IaC: In GitOps, everything is defined as code and stored in Git, which is the only source of truth.
- Merge requests: Code changes are reviewed and implemented for all infrastructure updates in merge requests.
- CI/CD: GitOps uses CI/CD to test and deploy code changes in the environment. GitOps automation overwrites any configuration drift, such as manual changes or errors, ensuring the environment maintains the desired state defined in Git.
GitOps workflows consist of three components: IaC, merge requests and CI/CD.
Just like IaC, GitOps requires developers to declaratively describe the desired state of the system. The configuration files and source code are stored and version-controlled in a Git repository — a distributed version control system holding all of the app's code. This repository tracks every change made to files in that project over time and is the control system for creating, updating and deleting infrastructure components.
Merge requests are where code changes are implemented for all infrastructure updates. To roll out a new release, a developer makes a request to modify the state of the system in the Git repository (e.g. git repo). The alerted teams review and ultimately approve or reject the changes. If the changes are approved, a merge commits the code to the main branch and serves as an audit log.
When the changes are approved and merged, they are automatically applied to the live infrastructure. This lets developers continue using their normal GitOps pipeline for workflow and CI/CD practices.
When and why do you use GitOps?
GitOps makes the provisioning of infrastructure easier, faster and more scalable.
Conventional infrastructure management requires developers to file a ticket or email a request to the ITOps team whenever they want to provision infrastructure or cloud infrastructure resources. The ITOps team reviews and logs every request in a queue, then manually allocates the requested resources for each one by entering manual commands in a user interface. Although time consuming, this process worked well enough when most organizations had less infrastructure to manage and little turnover of components.
Today’s infrastructure is much more dynamic due to API-driven cloud environments that enable organizations to provision and decommission resources in seconds, rather than the long commitment required to deploy legacy infrastructure. Today, organizations may spin up thousands of development environments or microservices each week. It’s impractical to manage infrastructure manually; it would require submitting thousands of tickets every day to increase or decrease workload capacity as needed, plus a lot of time waiting for administrators to complete all those requests.
GitOps is a much more efficient solution. It replaces manual processes for provisioning and configuring infrastructure with automation. Developers can declare service-level objectives as part of their application code and have the infrastructure automatically scale to maintain these objectives. And it lets developers use familiar practices and manage infrastructure through the same version control system they use for application development. This means faster, more predictable deployments, more seamless code management and greater overall productivity.
GitOps Core Practices and Immutable Infrastructure
The three core practices of GitOps are:
- Declarative configuration: GitOps uses declarative description to define the desired state of the infrastructure or application in the target environment. The declarative description code is contained in a centralized Git repository or repo as a single source of truth for the whole system.
- The Git workflow: Everything is done by making changes to code or configuration files in Git. When a pull request is approved and merged, it automatically syncs the production environment to the new state of the repository.
- Separate configuration and deployment: Changes are applied to the target environment using a pull-based deployment. This keeps changes separate from the main project.
The three core practices of GitOps include declarative configuration, the Git workflow and separate configuration and deployment.
The Role of Immutable Infrastructure
GitOps encourages developers to work with immutable infrastructures. Mutable and immutable are two different types of infrastructure environments. A mutable infrastructure can be modified after it has been provisioned — to accommodate a new virtual machine, for example. An immutable infrastructure cannot. Each type of environment has advantages and disadvantages.
Mutable infrastructure allows developers to customize existing servers to specific app or development needs rather than requiring them to establish new ones. This enables the infrastructure to meet the needs of each user, which is a significant advantage.
Immutable environments are more rigid. Infrastructure components are built to match particular specifications and once the infrastructure is provisioned, ad hoc modifications can’t be made. Any changes would require a whole new infrastructure to be built based on the new requirements. While there are many disadvantages to this system, immutable infrastructure eliminates configuration drift, makes it easier to identify and resolve technical issues, and ensures testing consistency from development through production.
GitOps takes advantage of immutable infrastructure to minimize the risk of configuration drift and other changes after a deployment and to ensure that the infrastructure maintains its desired state. Together with declarative configuration, immutability also enables the complete recovery of the infrastructure if it crashes. Developers are also increasingly used to immutable infrastructure in another form —– Docker containers are immutable (as a best practice, at least).
GitOps vs. DevOps
Both GitOps and DevOps are designed around the developer experience. While there is some overlap in tools and practices, there are also significant differences between the two.
For one, DevOps is a culture and practice that enables organizations to develop and improve products at high velocity. GitOps is a technique for provisioning and managing infrastructure designed around and tethered to a specific tool: Git. DevOps is mainly concerned with changing an organization’s culture to foster greater collaboration between development and operations teams. GitOps provides tools and a framework to apply DevOps practices such as version control and CI/CD to infrastructure management.
Despite these differences, DevOps and GitOps can easily co-exist within the same organization because of their shared principles, practices and tools. It’s important to note, however, that while an organization using DevOps does not have to adopt GitOps, an organization wanting to adopt GitOps is likely already practicing DevOps.
Using GitOps with Kubernetes
Kubernetes and other modern cloud-native tools are defined declaratively, which is driving the popularity of GitOps. As a declarative system, Kubernetes understands how the ideal cluster configuration should look. It continuously monitors and compares the current state of the system and automatically reconciles it against its desired state.
This methodology lets developers define their ideal state of the system and version control Kubernetes configuration files in Git. The system becomes easier to manage, and it can be rolled back to a previous version if a deployment causes problems. Overall, GitOps facilitates better convergence and end-to-end management of Kubernetes and cloud-native application development, as well as application deployment.
Benefits and Challenges of GitOps
GitOps offers several benefits, including:
- Familiar tools and practices: GitOps allows teams to manage IT infrastructure using many of the same tools and practices they already use in application development. Git is a familiar tool for most developers, and version control and CI/CD are practiced in DevOps.
- Increased productivity: Because GitOps is built around the developer experience, developers are working with tools and processes in which they’re well-versed and don’t have to introduce new tools into their workflow. Similarly, new developers can get up to speed in days rather than months.
- Faster, more frequent deployments: GitOps workflows allow teams to ship multiple times per day, deploy instantaneously and provide real-time visibility so they can roll back changes if a problem arises. This means faster time to market.
- Traceable changes: Because everything happens in the Git repository, all changes to the system are traceable via the Git log. Development teams can easily see who made a change and what changes were made from one version to the next.
- Faster rollbacks: If a deployment brings down the environment, you can simply revert to the previous version using Git.
- Easier knowledge sharing: A Git repository makes it easier for teams to get and share information. If a developer wants to understand the evolution of the infrastructure, for example, they can simply clone the Git repository and review the past commits.
GitOps also comes with a few potential challenges:
- Lack of consistent standards: Like any emerging technology practice, GitOps isn’t defined the same way industry-wide. The good news is that a GitOps working group was established as a CNCF open community project in November 2020 by Amazon, Codefresh, GitHub and Weaveworks, with the goal of defining GitOps core principles more clearly.
- Issues running at scale: As enterprise environments trend toward a mono-repository and the number of deployment repositories scales up with new applications, each one has to be configured with the appropriate access rights and synced to the necessary agents. This can be prohibitively time-consuming in complex environments. A larger number of Git repositories and configuration files also hamper visibility into what’s happening in your environments.
- Lack of skilled talent and buy-in: GitOps is still a relatively new technology, so finding skilled engineers capable of running, managing and evangelizing it can be difficult. Also, with many organizations having only recently adopted DevOps, it can be a tall order to convince leadership to get on board with an even less mainstream methodology.
Getting Started With GitOps
As GitOps uses DevOps practices and tooling, having a mature DevOps culture is a prerequisite for getting started. It ensures developers can get acclimated to GitOps workflows quickly.
The other requirement is infrastructure that can be declaratively managed. For this reason, GitOps is often employed to manage Kubernetes clusters and cloud-native app development. GitOps can be used with other infrastructure and deployment pipelines, such as Ansible, which enables declarative modeling of traditional IT systems.
You will also need tools to implement the GitOps workflow. There are dozens of available GitOps tools, but Flux and ArgoCD are two of the more popular options.
Rather than trying to implement GitOps across your entire ecosystem at once, take an incremental approach. Start with network configuration management, a virtual machine OS, or another resource and define its desired state. Then begin making changes to the current state using infrastructure as code, employing a Git repository to keep track of the resource’s state as it changes, and from there use your CI/CD system to deploy updates. Continue developing a GitOps workflow and documenting processes for your development team until you’re confident applying GitOps to your whole ecosystem.
GitOps Best Practices for Getting Started
Some GitOps principles and best practices include:
- Choose branches over repositories: Limit repositories to one per team and when there’s a need for more repositories, use branches instead. This will make it easier to get an overview of the entire system, save time, and facilitate easier code sharing and feedback among developers.
- Pull instead of push: It’s best to use pull (or merge) requests rather than pushing to publish new code. A pull request typically includes valuable supporting information about why specific changes were made and by whom, easing troubleshooting and debugging tasks.
- Automate: Automating manual tasks is critical for ensuring they can be repeated faithfully with no errors. This means every action should be performed by a script that’s been tested for bugs.
- Monitor: It’s important to continuously monitor the system — including new features — and the state of your applications for any failures, bottlenecks, or other anomalies and enable your CI/CD system to trigger automated responses. This will allow you to resolve issues more quickly and automatically roll back to a previous state if necessary.
The Bottom Line: GitOps is crucial for modern infrastructures
The demands made on modern infrastructures make infrastructure automation a must. GitOps offers an easy way to get started by leveraging the tools and practices your developers are already using. With one small pilot project, you can take the first step toward building a more resilient infrastructure.
What is Splunk?
This posting does not necessarily represent Splunk's position, strategies or opinion.