In this article we describe a build process to build images which described in the werf.yaml configuration file. Build process involves sequential building of stages for each image described in the configuration.

Dockerfile-image, Stapel-image and Stapel-artifact each built by a different type of conveyor. But werf handles each stage of such conveyor in a common way: there is the same stage selection rules, the same stage building and saving rules and also the same synchronization rules of multiple werf processes running from arbitrary hosts.

Dockerfile image

werf uses Dockerfile as the principal way to describe how to build an image. Images built with Dockerfile will be referred to as dockerfile images (learn more about a dockerfile image).

How a dockerfile image is being built

werf creates a single stage called dockerfile to build a dockerfile image.

How the dockerfile stage is being built:

  1. Stage digest is calculated based on specified Dockerfile and its contents. This digest represents the resulting image state.
  2. werf does not perform a new docker build if an image with this digest already exists in the stages storage.
  3. werf performs a regular docker build if there is no image with the specified digest in the stage storage. werf uses the standard build command of the built-in docker client (which is analogous to the docker build command). The local docker cache will be created and used as in the case of a regular docker client.
  4. When the docker image is complete, werf places the resulting dockerfile stage into the stages storage (while tagging the resulting docker image with the calculated digest) if the :local stages storage parameter is set.

See the configuration article for the werf.yaml configuration details.

Stapel image and artifact

Also, werf has an alternative tool for building images. The so-called stapel builder:

  • provides an integration with the git and incremental rebuilds based on the git repo history;
  • allows using ansible tasks to describe instructions needed to build an image;
  • allows sharing a common cache between builds with mounts;
  • reduces image size by detaching source data and build tools.

The image built with a stapel builder will be referred to as a stapel image.

See stapel image and stapel artifact articles for more details.

How stapel images and artifacts are built

Each stapel image or an artifact consists of several stages. The same mechanics is used to build every stage.

werf generates a specific list of instructions needed to build a stage. Instructions depend on the particular stage type and may contain internal service commands generated by werf along with user-specified shell commands. For example, werf may generate instructions to apply a prepared patch from a mounted text file using git cli util.

All generated instructions to build the current stage are supposed to be run in a container that is based on the previous stage. This container will be referred to as a build container.

werf runs instructions from the list in the build container (as you know, it is based on the previous stage). The resulting container state is then committed as a new stage and saved into the stages storage.

werf has a special service image called flant/werf-stapel. It contains a chroot /.werf/stapel with all the necessary tools and libraries to build images with a stapel builder. You may find more info about the stapel image in the article.

flant/werf-stapel is mounted into every build container so that all precompiled tools are available in every stage being built and may be used in the instructions list.

How stapel builder processes CMD and ENTRYPOINT

To build a stage image, werf launches a container with the CMD and ENTRYPOINT service parameters and then substitutes them with the base image values. If the base image does not have corresponding values, werf resets service to the special empty values:

  • [] for CMD;
  • [""] for ENTRYPOINT.

Also, werf uses the special empty value in place of a base image’s ENTRYPOINT if a user specifies CMD (docker.CMD).

Otherwise, werf behavior is similar to docker’s.

Stage selection

Werf stage selection algorithm is based on the git commits ancestry detection:

  1. Calculate a stage digest for some stage.
  2. There may be multiple stages in the stages storage by this digest — so select all suitable stages by the digest.
  3. If current stage is related to git (git-archive, user stage with git patch, git cache or git latest patch), then select only those stages which are related to the commit that is ancestor of current git commit.
  4. Select the oldest by the TIMESTAMP_MILLISEC from the remaining stages.

There may be multiple built images for a single digest. Stage for different git branches can have the same digest, but werf will prevent cache of different git branches from being reused for different branch.

Stage building and saving

If suitable stage has not been found by target digest during stage selection, werf starts building a new image for stage.

Note that multiple processes (on a single or multiple hosts) may start building the same stage at the same time. Werf uses optimistic locking when saving newly built image into the stages storage: when a new stage has been built werf locks stages storage and saves newly built stage image into storage stages cache only if there are no suitable already existing stages exists. Newly saved image will have a guaranteed unique identifier TIMESTAMP_MILLISEC. In the case when already existing stage has been found in the stages storage werf will discard newly built image and use already existing one as a cache.

In other words: the first process which finishes the build (the fastest one) will have a chance to save newly built stage into the stages storage. The slow build process will not block faster processes from saving build results and building next stages.

To select stages and save new ones into the stages storage werf uses synchronization service components to coordinate multiple werf processes and store stages cache needed for werf builder.