Migration from v2 to v3

Start with the summary, then check the sections relevant to your build, deployment and CI. For rollback or parallel use of v2, read the compatibility requirements. Removed features require edits before upgrading; renaming deprecated keys can wait.

Default behavior changes

If you enabled experimental features in v2, some changes already apply to your environment.

These changes affect existing projects even without configuration edits:

Building and caching

Deployment and bundles

Before the first werf plan, check sensitive data redaction: the previous annotation no longer hides the entire resource. This is especially important for shared CI logs.

CI and automation

Separately — CI contract changes, not new defaults:

• With --exit-code enabled, code 3 means a release-only update — see Plan exit codes.
• When reusing an image, the JSON build report may contain StagesSkipped: true without Stages — see Build report format.
• The JSON deploy report identifies its format with apiVersion instead of version — see Deploy report format.

Compatibility with v2

Running both versions or rolling back to v2 depends on shared configuration, secrets and registry state, not just the binaries:

• Configuration. After adopting v3 settings, do not assume the same werf.yaml will work in v2. Keep a v2-compatible configuration revision for rollback — see building changes.
• Secrets. v3 reads old secrets, but an old client cannot read new v3 writes. Upgrade all readers before the first rewrite; switching the binary back to v2 does not restore the old format — see encrypted secrets.
• Registry. A shared --repo means shared images: protect rollback images from cleanup.

Building

Expect a full rebuild of images after moving to v3. Account for it when planning the upgrade.

Images instead of artifacts

Required for Stapel using artifact: Replace artifact: builder with this fragment:

image: builder
final: false

Keep the other image settings unchanged.

Required for Stapel using disableGitAfterPatch: remove the directive from werf.yaml; it is no longer supported, and an unknown key is a configuration error. v2 suggested it for keeping artifact behavior when replacing artifact with image. There is no equivalent switch in v3. Use git.stageDependencies to control which changes rerun build instructions, but this does not disable updates to the Git files themselves — see Git dependencies of build stages.

Image names

Nameless stapel images (image: ~) are no longer supported — give every image a name.

Names are now checked when loading the configuration. Latin letters, digits, _, ., - and + are allowed, with optional /-separated segments. Each segment must start with a letter or digit and end with a letter, digit or +. modules/controller is valid; an empty name, whitespace, /api, api- and modules//controller cause a configuration error. Check names produced by Go templates as well.

One from key for image references

Renaming can wait: fromImage, import.image and dependencies.image still work in v3 with a warning. When replacing a key, use only one: specifying both the old and new key is an error.

A name from werf.yaml, such as base, refers to an internal image; a reference with a tag or digest, such as alpine:3.20, refers to an external image. The nested imports block in dependencies still passes image information rather than copying files.

Renaming import.image does not replace the mandatory removal of import.stage — see file imports.

External image with an explicit tag

Required for external references in a base from or import.from: specify a tag or digest. For example, replace from: ubuntu with from: ubuntu:latest, from: ubuntu:TAG, or from: ubuntu@sha256:.... Internal image names from werf.yaml do not need a tag.

Builders and image configuration

For Dockerfiles using the Docker backend: the legacy Docker builder is no longer available. BuildKit was already the default in v2, but DOCKER_BUILDKIT=0 or false allowed opting back into the old builder. In v3, this variable does not restore that mode: check your Dockerfiles and environment with BuildKit.

For staged Dockerfiles: WERF_STAGED_DOCKERFILE_VERSION=v1 no longer selects the old implementation.

Required for Stapel using ansible:: the Ansible builder is removed. Rewrite ansible: steps using the Shell builder; renaming the key is not enough.

Required for Stapel using docker:: the directive is removed. Move its settings to imageSpec.config, translating field names and formats. For example, for a stapel image fragment:

docker:
  WORKDIR: /app
  ENV:
    APP_ENV: production

imageSpec:
  config:
    workingDir: /app
    env:
      APP_ENV: production

See Changing image configuration spec for the full set of fields.

File imports

For Stapel images using the import directive.

Import caching now depends on the source image, rather than checksums of the selected files as it did by default in v2. Changing the source image can rebuild the importing image even if the copied files are unchanged. includePaths/excludePaths still select which files to copy, but no longer isolate the cache from other source-image changes. If those rebuilds are expensive, put the imported output in a separate, narrowly scoped image.

You must remove import.stage. Imports use the completed source image, not a selected intermediate stage. If you need an intermediate result, make it a separate image. before/after still control when the import runs in the destination image.

For example, an import fragment referencing the builder image after removing stage and optionally renaming image to from:

import:
  - from: builder
    add: /out/message.txt
    to: /message.txt
    before: setup

A trailing slash in export/import to: is now an error, except for the root path to: /. Previously werf stripped it with a warning, although users could expect it to mean “copy into this directory”. Check the intended destination before changing to: /usr/sbin/ to to: /usr/sbin:

• If add is a directory, its contents are merged into to.
• If add is a file, it is copied into to when to is an existing directory; otherwise to is the file’s destination path. To avoid depending on whether the containing directory exists, specify the full filename, for example to: /etc/app/config.yaml, and ensure that path is not a directory.

See Destination path rules for details.

Git dependencies of build stages

For Stapel with Git mappings: git.stageDependencies determines which Git file changes trigger Shell-stage rebuilds.

In v3, an omitted setting and an explicit empty list can mean different things:

In a partially filled block, omitted stages before the last explicitly declared stage get [], and those after it get **/*. The order is always install → beforeSetup → setup, regardless of YAML key order. The rule applies independently to each git mapping. In v2, every omitted stage had no Git-file dependency regardless of its position: the new **/* defaults after the last declared stage can cause additional rebuilds.

For example, only beforeSetup is declared:

git:
  - add: /
    to: /app
    stageDependencies:
      beforeSetup:
        - "src/**/*"

install gets [], beforeSetup gets src/**/*, and setup gets **/*. Changing a file outside src therefore does not by itself trigger install or beforeSetup, but triggers setup if it has instructions.

To keep a stage free of direct Git-file dependencies, specify []; to always track all mapped files, specify ["**/*"]. If you specify masks, include every file that affects the stage’s result.

[] disables only the direct Git-file dependency. Changes to preceding stages, base images, commands and other inputs can still trigger a rebuild.

A non-empty stageDependencies entry for a stage with no build instructions now causes an error instead of a warning. Move the masks to the stage with the corresponding commands, add the missing instructions, or remove the unnecessary dependency.

See Dependency on changes in the Git repo.

Validation of werf-giterminism.yaml

Fix unknown or misspelled keys in werf-giterminism.yaml: strict schema validation no longer lets them be silently ignored.

Which images are built and listed

werf build and other build-triggering commands now default to --final-images-only=true, as converge, render, export, plan, lint and bundle publish already did. Orphan non-final images that no final image references are no longer built by default.

werf config list also lists only final images by default; its deprecated --images-only alias is removed.

To keep the previous behavior:

werf build --final-images-only=false
werf config list --final-images-only=false

Buildah

This section applies if you use the native Buildah backend. It switched from CNI/slirp4netns to netavark/pasta: prepare the environment before upgrading according to how you run werf.

Official werf image

Update the werf container image to v3, not just the binary inside it. The image already includes netavark; you do not need to install it separately on the runner host to run werf in that container.

The official Ubuntu werf images are now based on Ubuntu 24.04. Rebuild and check any images you derive from them.

For rootless builds that configure a network, also check that pasta is available inside the container under the conditions below; having netavark alone is not enough. If it is missing, add it to your derived image.

Your own installation or image

If you install werf directly on a host or use your own container image, make sure the dependencies are available where werf runs:

See Network tools for installation paths and considerations when sharing storage with Podman/Buildah.

Network setting changes

These changes apply both inside and outside the official image.

Check network settings. Native Buildah now honors network: and --backend-network for non-staged Dockerfile builds; they used to be silently ignored. Only default, host and none are supported: bridge and Docker network names now cause an error.

Limitations and compatibility:

• Images with staged: true still ignore image-level network settings; only RUN --network= on an individual instruction applies.
• In native-chroot, Buildah forces host networking, so network: none does not provide isolation.
• CNI cannot be restored. You can restore slirp4netns on an individual host via CONTAINERS_CONF_OVERRIDE: default_rootless_network_cmd="slirp4netns" in the [network] section.

Deployment

After configuring sensitive data redaction, review the plan before the first werf converge: changes to values, patches and file exclusion rules can affect manifests without a warning.

Sensitive data in diffs

By default in v2, only identifying fields remained visible for Secret resources and any resource annotated with werf.io/sensitive: "true". In v3, only values at data.* and stringData.* are automatically hidden. Other fields, including metadata and secret key names, may be visible; hidden values are replaced with length and hash information.

The werf.io/sensitive: "true" annotation no longer guarantees redaction of an entire arbitrary resource. If sensitive data is stored in fields such as spec, specify its JSONPath expressions with werf.io/sensitive-paths, for example "$.spec.token". This annotation replaces the default path list rather than extending it: include $.data.* and $.stringData.* as well if you need to keep Secret data redacted. Encrypting secret-values.yaml does not define redaction rules for fields in rendered resources.

Resource and values validation

Kubernetes resource schema validation is enabled by default. In v2, it required an experimental flag. Manifests that previously passed may now be rejected before they are applied. Fix the resource or schema; use --resource-validation-extra-schema for additional schemas. The old --local-resource-validation, --resource-validation-kube-version and --resource-validation-schema flags are removed.

If you need an exception, use --resource-validation-skip; disable all resource validation with --no-resource-validation. Do not disable validation for every resource just to accommodate one unsupported type.

values.schema.json validation already existed in v2, but error handling is stricter. Previously, an error about forbidden service values could turn the entire validation result into a warning, including errors in user values. Now werf retries validation without service values and fails if user values do not match the schema. Fix the values or schema; temporarily disable this check with --no-values-schema-validation.

Automatic patches and null

patches.yaml from the main chart and its dependencies is now read automatically. An existing file with this name that served another purpose may change manifests or cause an error. Check it before deploying. Disable automatic loading with --no-default-patches or WERF_NO_DEFAULT_PATCHES=true.

Fields and list entries whose value is null are now removed recursively. In v2, this required an experimental flag. An absent field and an explicit null are not always equivalent for Kubernetes and CRDs, so check resources that intentionally use null. There is no switch to restore the previous behavior.

Non-string values in metadata.annotations and metadata.labels now fail rendering with a decode resource error. In v2, such values were silently dropped. Quote numbers and booleans in templates, for example example.com/replicas: "3". Resources already stored in a release are not rejected: their invalid entries are dropped with a warning.

External dependency annotations

Required if you wait for resources outside the release: the <name>.external-dependency.werf.io/resource and <name>.external-dependency.werf.io/namespace annotations are no longer supported. werf prints a warning and does not wait for the resource. Replace them with werf.io/deploy-dependency-<name>, specifying kind, version, name and, for a group resource, group.

Before — v2:

annotations:
  secret.external-dependency.werf.io/resource: secret/my-vault-secret
  secret.external-dependency.werf.io/namespace: vault

After — v3:

annotations:
  werf.io/deploy-dependency-secret: state=ready,kind=Secret,version=v1,name=my-vault-secret,namespace=vault,external=true

See Waiting for non-release (external) resources and Resource dependencies.

Resource deletion and recreation

Resources are now deleted with Background propagation by default, instead of Foreground. werf no longer waits for the dependents of a deleted resource, such as the Pods of a Deployment, to disappear before proceeding. If a subsequent step depends on that, set werf.io/delete-propagation: Foreground on the resource or pass --delete-propagation=Foreground / WERF_DELETE_PROPAGATION=Foreground.

A resource with helm.sh/resource-policy can no longer be recreated. When an immutable field changes, the deploy fails with cannot recreate the resource ... because its deletion is prohibited instead of deleting and recreating the resource. Revert the field or remove the annotation for that deploy.

Readiness tracking

Condition types are now matched case-insensitively. In v2, a CamelCase condition such as Ready or Available on a custom resource did not match the built-in ready rules, and the resource counted as ready immediately. Now werf waits for that condition to become True. If a custom resource never sets it, the deploy times out: set werf.io/track-termination-mode: NonBlocking on the resource or fix the controller.

Offline rendering

Without cluster access, werf render, werf lint and werf bundle render now assume Kubernetes 1.36.0 instead of 1.20.0. .Capabilities.KubeVersion changes accordingly, so templates that branch on it may render differently. Pass --kube-version to pin the previous version.

Replacing removed werf helm commands

Deployment and release-inspection commands under werf helm are removed. For werf projects, use the dedicated commands:

These are workflow replacements, not drop-in aliases. converge, render and lint use the werf project rather than Helm’s positional RELEASE CHART arguments. Check the new command’s --help, pass the intended release and namespace explicitly, and adapt scripts that parse output. For example, release get --print-values includes computed values in its structured output; it does not reproduce helm get values output. For standalone Helm chart workflows, use the Helm CLI directly.

werf helm secret and chart-management commands such as werf helm dependency remain available; the whole werf helm group has not been removed.

Uninstalling a release

werf dismiss now always uses the new uninstall implementation, which v2 enabled with WERF_EXPERIMENT_NEW_DISMISS. The old implementation selector and --with-hooks flag are removed. Do not carry that flag into v3 scripts: check release and hook removal in a test environment. Deleting the namespace still requires --with-namespace.

Values and environment variables

The old .Values.global.env key can render as an empty value without an error. Set WERF_LEGACY_VALUES_GLOBAL_ENV=1 for temporary compatibility.

Without replacing HELM_DRIVER, werf uses its default release storage rather than the storage previously selected by that variable.

Helm’s path variables are no longer read. HELM_CACHE_HOME, HELM_CONFIG_HOME and HELM_DATA_HOME are ignored: the chart repository list and the chart cache are located via XDG_CONFIG_HOME and XDG_CACHE_HOME, by default ~/.config/helm and ~/.cache/helm on Linux. These XDG variables specify parent directories: for example, with XDG_CONFIG_HOME=/custom/config, werf reads /custom/config/helm/repositories.yaml, not /custom/config/repositories.yaml. To reuse repositories added in v2 under a custom HELM_CONFIG_HOME, copy their repositories.yaml into the new helm directory, or add the repositories again. Setting XDG_CONFIG_HOME to the old HELM_CONFIG_HOME alone does not preserve the location.

Charts and bundles

In v3, .helmignore applies when reading the chart. Excluded files disappear from the rendered manifests and the published bundle without a warning.

• Even without .helmignore, Helm’s default rules exclude dot-prefixed files and directories directly under templates/. Directories are excluded with their contents.
• ** now causes an error, although it previously had no effect.
• A leading ! previously had no effect. It now excludes everything that does not match the pattern, usually leaving the chart empty. Do not use it to undo earlier exclusions as in .gitignore.

Rules for dependent charts vary by how the chart is included — see Charts and dependencies.

werf bundle publish now defaults to --helm-compatible-chart=true. The chart name in the published bundle’s Chart.yaml becomes the last path component of the repository address, changing .Chart.Name and template paths. To keep the name from your Chart.yaml, pass:

werf bundle publish --helm-compatible-chart=false

Encrypted secret values

An old client cannot read a secret written by v3, and there is no option to write the previous format. Follow this upgrade order:

1. Upgrade every environment that reads the repository’s secrets to werf v3 or nelm v2: developer machines, CI jobs, and jobs applying saved deploy plans.
2. Only then create, edit or rotate secrets. An old werf or nelm must not handle the repository after secrets are re-encrypted.

Additional considerations:

• Old encrypted scalars remain strings. To restore a number, boolean, timestamp or another type, re-enter the value with werf helm secret values edit.
• Comments on encrypted values are kept as cleartext. Do not put secrets in them.

CI and scripts

Registry credentials

If you configure CI with werf ci-env:

werf ci-env now automatically uses a non-empty DOCKER_AUTH_CONFIG if neither --use-docker-auth-config nor WERF_USE_DOCKER_AUTH_CONFIG is set. In v2, this required explicit opt-in.

With this choice, the Docker config is created from DOCKER_AUTH_CONFIG instead of copying the existing config, not merged with it. Credentials and credential helpers from the previous config may no longer be used. Restore the previous choice with --use-docker-auth-config=false or WERF_USE_DOCKER_AUTH_CONFIG=false.

Plan exit codes

With --exit-code enabled, werf plan and werf bundle plan now always distinguish resource changes from release-only updates. In v2, the extended set of codes required an experimental flag.

Update CI if it only accepts 0 and 2. Code 3 is not an error, but is not a reason to automatically skip applying the plan either. Without --exit-code, a successful plan does not start returning 2 or 3.

Build report format

If your scripts read the JSON build report:

When reusing a completed image, the JSON build report may now contain StagesSkipped: true without a Stages field. In v2, Stages was present, although it could be null. Parsers must tolerate the missing field and not treat it as an error or as an indication that the image is not ready.

Deploy report format

If your scripts read the JSON report saved by --save-deploy-report, --save-rollback-report or --save-uninstall-report:

The format identifier changed: the "version": 3 number is replaced with the "apiVersion": "v3" string, and version is gone. Update scripts that read or validate version to check apiVersion == "v3" instead.

Removed flags and modes

Check scripts that pass old options: a removed flag causes an argument parsing error even if it previously did nothing.

• --virtual-merge / WERF_VIRTUAL_MERGE, --skip-image-spec-stage / WERF_SKIP_IMAGE_SPEC_STAGE, --set-runtime-json and --show-verbose-diffs are removed.
• --force-adoption is removed from werf render and werf bundle render, where it had no effect. Other deploy commands keep it.
• Helm mode via WERF_HELM3_MODE and invoking the werf binary under the name helm are removed; use supported werf commands or the standalone Helm CLI.
• Positional image names in converge and plan now take effect without WERF_CONVERGE_ENABLE_IMAGES_PARAMS. Check that stray arguments have not become image selectors; selecting images does not by itself limit which Kubernetes resources are deployed.

Registry cleanup

Protecting rollback images

With a shared --repo, v3 cleanup can delete v2 images. Keep the stage tags needed for rollback in a --keep-list file, one per line: Kubernetes resources may no longer reference these images.

Before the first cleanup, ensure rollback images are not listed for deletion. Use the same repositories, keep-list and Kubernetes access as the scheduled job:

werf cleanup --repo registry.example.com/app --dry-run --keep-list keep-list.txt

Pass the same --final-repo and --meta-repo used for building, if configured. Check coverage of the required clusters and namespaces; --without-kube disables Kubernetes protection. Do not use werf purge to remove only v2 images: it deletes the project’s images without cleanup’s retention policies. See Container registry cleanup.