We propose to divide the assembly proccess into steps, intermediate images (like layers in Docker), with clear functions and assignments. In Werf, such step is called stage and result image consists of a set of built stages. All stages are kept in a stages storage and defining build cache of application (not really cache but part of building context).

Stages

Stages are steps in the assembly process, building blocks for constructing images. A stage is built from a logically grouped set of config instructions, taking into account the assembly conditions and rules. Each stage relates to one Docker image.

The Werf assembly process assumes a sequential build of stages using stage conveyor. A stage conveyor is a sequence with the predefined order and set of stages. Werf uses different stage conveyor for assembling a particular type of build object.

User only needs to write a config corretly the rest of the work with stages are done by Werf.

For every stage at each build, Werf calculates build stage identifier called stage signature. Each stage is assembled in an assembly container based on the previous stage, and saved in stages storage. The stage signature is used for tagging stage in stages storage. Werf does not build stages that already exist in stages storage (like caching in Docker, but more complex).

The stage signature is the checksum of stage dependencies and previous stage signature. In the absence of stage dependencies, the stage is skipped.

It means that the stage conveyor, can be reduced to several stages or even to single from stage.

Stage dependencies

Stage dependency is some piece of data that affects stage signature. Stage dependency may be represented by:

  • some file from git repo with its content;
  • instructions to build stage specified in werf.yaml;
  • arbitrary string specified by user in werf.yaml; etc.

Most stage dependencies are specified in werf.yaml, others relate to a runtime.

Tables below represent Dockerfile image, Stapel image and Stapel artifact stages dependencies. Each row describes dependencies for certain stage. Left column consists of short descriptions of dependencies, right includes related werf.yaml directives and contains relevant references for more information.

stage dockerfile

target dockerfile instructions
hashsum of files related with ADD and COPY dockerfile instructions
args used in target dockerfile instructions
addHost
image: <image name... || ~>
dockerfile: <relative path>
context: <relative path>
target: <docker stage name>
args:
  <build arg name>: <value>
addHost:
- <host:ip>

stage from

from
or from image id
or from artifact id
actual digest from registry (if fromLatest: true)
fromCacheVersion
mounts
from: <image[:<tag>]>
fromLatest: <bool>
fromCacheVersion: <arbitrary string>
fromImage: <image name>
fromImageArtifact: <artifact name>
mount:
- from: build_dir
  to: <absolute or relative path>
- from: tmp_dir
  to: <absolute path>
- fromPath: <absolute or relative path>
  to: <absolute path>

stage beforeInstall

beforeInstall bash commands or ansible tasks
cacheVersion
beforeInstallCacheVersion
shell:
  beforeInstall:
  - <bash command>
  cacheVersion: <arbitrary string>
  beforeInstallCacheVersion: <arbitrary string>

or

ansible:
  beforeInstall:
  - <task>
  cacheVersion: <arbitrary string>
  beforeInstallCacheVersion: <arbitrary string>

stage importsBeforeInstall

imports before install
import:
- artifact: <artifact name>
  before: install
  add: <absolute path>
  to: <absolute path>
  owner: <owner>
  group: <group>
  includePaths:
  - <relative path or glob>
  excludePaths:
  - <relative path or glob>

stage gitArchive

git mappings
git:
- add: <absolute path>
  to: <absolute path>
  owner: <owner>
  group: <group>
  includePaths:
  - <relative path or glob>
  excludePaths:
  - <relative path or glob>
- url: <git repo url>
  branch: <branch name>
  commit: <commit>
  tag: <tag>
  add: <absolute path>
  to: <absolute path>
  owner: <owner>
  group: <group>
  includePaths:
  - <relative path or glob>
  excludePaths:
  - <relative path or glob>

stage install

install bash commands or ansible tasks
installCacheVersion
git files hashsum by install stageDependency
git:
- stageDependencies:
    install:
    - <relative path or glob>

shell:
  install:
  - <bash command>
  installCacheVersion: <arbitrary string>

or

ansible:
  install:
  - <task>
  installCacheVersion: <arbitrary string>

stage importsAfterInstall

imports after install
import:
- artifact: <artifact name>
  after: install
  add: <absolute path>
  to: <absolute path>
  owner: <owner>
  group: <group>
  includePaths:
  - <relative path or glob>
  excludePaths:
  - <relative path or glob>

stage beforeSetup

beforeSetup bash commands or ansible tasks
beforeSetupCacheVersion
git files hashsum by beforeSetup stageDependency
git:
- stageDependencies:
    beforeSetup:
    - <relative path or glob>

shell:
  beforeSetup:
  - <bash command>
  beforeSetupCacheVersion: <arbitrary string>

or

ansible:
  beforeSetup:
  - <task>
  beforeSetupCacheVersion: <arbitrary string>

stage importsBeforeSetup

imports before setup
import:
- artifact: <artifact name>
  before: setup
  add: <absolute path>
  to: <absolute path>
  owner: <owner>
  group: <group>
  includePaths:
  - <relative path or glob>
  excludePaths:
  - <relative path or glob>

stage setup

setup bash commands or ansible tasks
setupCacheVersion
git files hashsum by setup stageDependency
git:
- stageDependencies:
    setup:
    - <relative path or glob>

shell:
  setup:
  - <bash command>
  setupCacheVersion: <arbitrary string>

or

ansible:
  setup:
  - <task>
  setupCacheVersion: <arbitrary string>

stage gitCache

size of git diff between last used commit and actual

stage importsAfterSetup

imports after setup
import:
- artifact: <artifact name>
  after: setup
  add: <absolute path>
  to: <absolute path>
  owner: <owner>
  group: <group>
  includePaths:
  - <relative path or glob>
  excludePaths:
  - <relative path or glob>

stage gitLatestPatch

presence of git diff changes between last used commit and actual

stage dockerInstructions

docker instructions
docker:
  VOLUME:
  - <volume>
  EXPOSE:
  - <expose>
  ENV:
    <env name>: <env value>
  LABEL:
    <label name>: <label value>
  ENTRYPOINT: <entrypoint>
  CMD: <cmd>
  WORKDIR: <workdir>
  USER: <user>
  STOPSIGNAL: <stopsignal>
  HEALTHCHECK: <healthcheck>

Stages storage

Stages storage keeps project stages. Stages can be stored in Docker Repo or locally, on a host machine.

Most commands use stages and require specified stages storage, defined by --stages-storage option or WERF_STAGES_STORAGE environment variable. At the moment, only local storage, :local, is supported.

Stage naming

Stages in local stages storage are named by the following schema — werf-stages-storage/PROJECT_NAME:STAGE_SIGNATURE.

Images

Image is a ready-to-use Docker image, corresponding to a specific application state and tagging strategy.

As it is written above, stages are steps in the assembly process, building blocks for constructing images. Stages are not intended for direct use, unlike images. The main difference between images and stages is cleaning policies due to stored meta-information. The stages storage cleanup is only based on the related images in images repo.

Werf creates images using stages storage. Currently, images can only be created in a publishing process and be saved in images repo.

Images should be defined in the werf configuration file werf.yaml.

See more info about build process.