werf

Werf: The Modern and Platform Agnostic CI/CD Tool

Werf is a Continuous Integration/Continuous Deployment (CI/CD) tool that simplifies the process of building, testing, releasing, and deploying applications to Kubernetes.

Introduction

Werf is a Continuous Integration/Continuous Deployment (CI/CD) tool that simplifies the process of building, testing, releasing, and deploying applications to Kubernetes. It's an Open Source project under the Cloud Native Computing Foundation (CNCF), designated as a sandbox project as of December 13, 2022.

Werf was designed to address the complexities and inconsistencies encountered in CI/CD processes, especially when deploying applications to Kubernetes. It originated in a professional services company that provided DevOps as a Service, Flant. The motivation behind its creation was to standardize and automate the implementation of CI/CD pipelines across various projects.

Background

The motivation behind the creation of werf primarily stems from the desire to improve the CI/CD experience, particularly when it comes to deploying applications on Kubernetes. Here are the key factors that drove its development:

Standardization and Automation
Werf originated in a professional services company providing DevOps as a Service for its customers. In the process of implementing CI/CD pipelines for various projects, there arose a desire to automate and standardize the way CI/CD was done across different projects. This included a wish to standardize the tools used for these processes.

Ease of Use
One of the goals of werf was to simplify the CI/CD process. By allowing users to just create a Dockerfile and a Helm chart, werf aims to manage the rest of the process, making it easier for users to handle CI/CD without getting bogged down by complexities.

Predictability and Reliability
Werf introduces concepts like Giterminism to encourage an Infrastructure as Code (IaC) approach and to use Git as a single source of truth. This leads to a more predictable and reliable delivery process, which is crucial for organizations looking to ensure consistency in their deployments.

Efficiency:
Werf aims to optimize the build and deployment processes by only rebuilding and redeploying the modified components. This is designed to save time and resources, making the CI/CD process more efficient.

Integration with Existing Technologies
Werf was designed to glue commonly used tools (Git, Docker/Buildah, Helm, Kubernetes) and facilitate best practices. This integration with familiar and reliable technologies is aimed at providing a seamless CI/CD experience without requiring users to learn new tools or change their existing workflows drastically.

Community-Driven Development
As an open-source project under the Cloud Native Computing Foundation (CNCF), werf benefits from community-driven development. This community involvement likely contributes to its ongoing improvement and adaptation to meet the evolving needs of CI/CD practitioners.

While there are indeed many similar tools available, werf's unique approach to addressing specific challenges in the CI/CD domain, its emphasis on simplification, and its integration with widely-used technologies contribute to its distinct position in the ecosystem.

Werf Architecture

The architecture of werf is built to streamline the process of CI/CD, especially when dealing with Kubernetes deployments. Here's a breakdown of its architecture and how it interacts with Buildah:

Unified Tool
Werf serves as a unified, all-in-one tool that handles various stages of the CI/CD pipeline including building images, running tests, distributing release artifacts, and deploying applications to Kubernetes¹.

Configuration Files
The primary configuration files used by werf are werf.yaml, Dockerfile, and Helm charts. The werf.yaml file defines project and image details, the Dockerfile contains instructions for building the Docker image, and the Helm charts outline the Kubernetes deployment specifications¹.

Stapel
Werf introduces a concept called Stapel, which is a custom builder for creating Docker images. Stapel allows for efficient image building by reusing existing layers and utilizing a unique image storage mechanism.

Git-Based Configuration
Adhering to the principle of Giterminism, werf relies on Git as the single source of truth for all configurations and code. This Git-based configuration ensures deterministic and reproducible builds and deployments.

Deployment Process
Werf manages the deployment process to Kubernetes, ensuring that only the modified components are redeployed. It handles the synchronization with the container registry, building missing Docker image layers, and redeploying the changed resources in Kubernetes.

Artifact Cleanup
Werf also manages the cleanup of obsolete artifacts in the container registry using a unique algorithm based on Git history and user-defined policies.

Interaction with Buildah

Buildah is one of the technologies that werf integrates with to facilitate the CI/CD process. Buildah is an open-source tool that facilitates building OCI (Open Container Initiative) container images. Here’s how werf interacts with Buildah:

Image Building: Werf uses Buildah for building container images. Buildah provides the mechanism to define and create container images that comply with the OCI image specifications.
Layer Reuse and Efficiency: Buildah's ability to handle image layers efficiently complements werf's objective of optimizing the build process. By reusing existing layers, Buildah helps in achieving incremental builds, which is one of the efficiency features of werf.
Integration: The integration of Buildah within werf’s architecture allows for a seamless image building process which is crucial for the overall CI/CD pipeline. This integration also provides users the flexibility to use a reliable and familiar tool for the image building process.
Content-based Tagging Buildah's capabilities can be leveraged by werf for content-based tagging of images, which is an advanced feature aimed at better resource tracking and management.

Giterminism

Giterminism is a concept central to werf's design philosophy, aimed at making the CI/CD process more predictable and reliable by treating Git as a single source of truth. Here’s a detailed explanation of Giterminism within the context of werf:

Single Source of Truth

Giterminism promotes the idea of having a single source of truth, and in the case of werf, Git serves as that source. All the information required to build, test, and deploy applications is stored in Git. This approach ensures that every aspect of the CI/CD process can be traced back to a version-controlled source.

Reproducibility

By adhering to Giterminism, werf ensures that builds are reproducible. The state of the repository at a particular commit defines the state of the build, making it possible to recreate the same build in the future. This is crucial for debugging and auditing purposes.

Deterministic Behavior

Giterminism encourages deterministic behavior in the CI/CD pipeline. Every operation's outcome is determined by the state of the Git repository, ensuring predictable results across different environments and over time.

Infrastructure as Code (IaC)

Giterminism aligns with the Infrastructure as Code (IaC) paradigm, which emphasizes the codification of infrastructure configurations and operations. This alignment facilitates version control, collaboration, and consistency across the CI/CD pipeline.

Traceability

With Giterminism, every change in the application delivery process is traceable to a commit in Git. This traceability is invaluable for troubleshooting, auditing, and understanding the history of changes.

Elimination of External Configurations

Giterminism discourages the use of external configurations that are not stored in Git. This ensures that the entire process remains contained within a version-controlled environment, reducing the likelihood of configuration drift and unauthorized changes.

Consistency

By requiring all configurations to be stored in Git, Giterminism promotes consistency across different stages of the CI/CD pipeline. This consistency is crucial for ensuring that applications behave as expected across different environments.

Enhanced Collaboration

Giterminism fosters better collaboration among team members by ensuring that all configurations and changes are version-controlled and visible to all stakeholders.

In werf, Giterminism manifests through features like reproducible builds, predictable deployments, and easier troubleshooting by linking release artifacts to Git and CI/CD. By following the Giterminism principle, werf encourages an IaC approach, where “What you Git is what you get” — emphasizing the importance of storing all configurations and code in Git to ensure a predictable and reliable delivery process.

Stapel

Stapel is an alternative syntax within werf for describing assembly instructions of container images. It's built to ease the management of complex configurations and to streamline the image building process in a CI/CD pipeline. Here are the distinctive features and functionalities of Stapel as outlined in the werf documentation:

Complex Configurations Management
Stapel allows for easy support and parameterization of complex configurations. Users can reuse common snippets and generate configurations of images of the same type using YAML format and templating, which simplifies the management of complex setups.

Integration with Git
It provides dedicated commands for integrating with Git to enable incremental rebuilds based on the Git repository history. This is crucial for optimizing the build process by only rebuilding the necessary parts of an application.

Image Inheritance:
Similar to the Dockerfile multi-stage mechanism, Stapel allows for image inheritance and importing files from images. This feature helps in organizing and managing dependencies across different images.

Arbitrary Build Instructions
Users can run arbitrary build instructions, specify directory mount options, and use other advanced tools to build images. This flexibility is beneficial for handling various build requirements.

Efficient Caching Mechanics
Stapel has more efficient caching mechanics for layers, akin to the scheme supported for Dockerfile layers when building with Buildah. This feature contributes to the efficiency and speed of the build process.

Configuration
To build images using the Stapel builder, build instructions need to be defined in the werf.yaml configuration file. Stapel supports both the Docker server builder backend (assembly via shell instructions or Ansible) and Buildah (shell instructions only).

Unique Internal Image Identifier
Each image in werf.yaml is required to have a unique internal image identifier to be referenced during configuration and when invoking werf commands. This identifier is crucial for managing multiple images within a project.

Adding Source Code from Git Repositories
werf, with Stapel, adds files from the repository to the image either by fully transferring them via git archive or by applying patches between commits. This mechanism is used to reflect changes in a Git repository, and is integral for managing the source code in images.

These features make Stapel a powerful and flexible tool within werf, enhancing the image building process by providing a structured yet flexible approach to managing configurations and building instructions.

Stapel vs Dockerfile

Stapel and Dockerfile are both tools used for defining and creating container images, but they differ in their syntax, features, and focus areas. Here’s a detailed comparison between the two based on various aspects.

Criteria	Stapel	Dockerfile
Syntax	YAML-based	Shell-like, procedural
Git Integration	Native commands for Git, incremental builds	No built-in Git integration
Layer Caching	Efficient caching, akin to Buildah	Standard Docker layer caching
Configuration Management	Supports complex configurations via templates	Less suited for complex configurations
Flexibility	Arbitrary build instructions, advanced features	Limited to predefined set of instructions
Build Context	Enhanced control over build context	Build context defined by Dockerfile location
Image Inheritance	Supports complex inheritance patterns	Supports multi-stage builds
User Stages	Defined user stages like beforeInstall, setup	No concept of user-defined stages

Example of Staple Usage

# werf.yaml
configVersion: 1
project: myproject
---
image: myservice
from: ubuntu:20.04
shell:
  beforeInstall:
    - apt update -q
    - apt install -y python3-pip
  install:
    - pip3 install -r requirements.txt
  beforeSetup:
    - echo "Preparing configurations"
  setup:
    - python3 setup.py install
git:
  - add: /app
    to: /src
    stageDependencies:
      setup: ["*.py", "*.ini"]

This configuration uses shell commands across different stages to update system packages, install application dependencies, and perform application setup. The beforeInstall stage is for system-wide updates, install for application dependencies, and setup for final application setup. Git mappings ensure that application source files are included in the build, with dependencies that control caching behavior.

Integration with Drone CI

In the next article, we will combine the power of Drone CI with the flexibility of Werf as platform-agnostic CI/CD Tools. If you never heard about Drone CI before, this article below is a very good start to learn about Drone CI.

Drone CI: A Quest to Implement Open Source Cloud Native CI/CD Tools on Kubernetes (Part 1)

Drone CI is a Continuous Integration system built with simplicity and effectiveness in mind. It’s an open-source platform that orchestrates your build workflows, automating the testing and integration of code changes.