About bundles and charts
A bundle is a way to distribute a chart and its related images as a single entity.
The werf bundle publish command builds the images and publishes them together with the chart to be deployed later with werf. This way, access to the application’s Git repository is no longer needed during deployment.
The published chart is an ordinary Helm OCI chart: the source chart as is, plus the .werf-data/ directory at its root. .werf-data/ holds everything werf adds at publish time:
- the images built and their tags, as werf auxiliary parameters;
- the values passed with the
--values,--secret-values,--set*,--set-root-jsonoptions and the--patchesfiles; - the user and service annotations and labels to be added to the chart resources;
- the release name and Namespace resolved for the environment.
werf bundle apply (as well as werf bundle plan and werf bundle render) uses all of it, so publishing and then applying the chart deploys the same as werf converge with the same options. Helm, Argo CD, Flux and similar solutions can install the chart too, but they get only the chart’s own values by default, see Using the chart with Helm, Argo CD and Flux.
The published bundle (a chart and its related images) can be copied to another container registry or extracted from/added to the archive using the werf bundle copy command.
Container registry authentication
You must log in to the container registry to work with the images. Use the werf cr login command as follows:
werf cr login <registry url>
For example:
# Log in with a username and password from the command line
werf cr login -u username -p password registry.example.com
# Log in with a token from the command line
werf cr login -u username -p token registry.example.com
# Log in to an insecure registry (over HTTP)
werf cr login --insecure-registry registry.example.com
Note: In supported CI/CD systems, the user gets authenticated to the integrated container registries as part of the ci-env command — you do not have to use the werf cr login command in this case.
Publishing a bundle
You can publish the bundle to the OCI repository as follows:
-
Create
werf.yamlif it does not exist:# werf.yaml: project: mybundle configVersion: 1 -
Put the files in the main chart directory (by default,
.helmin the Git repository root). All files of the chart directory are published, except for the ones excluded by.helmignore(see below). -
The next step is to publish the bundle. Build and publish the images defined in
werf.yaml(if any), and then publish the contents of.helmas theexample.org/bundles/mybundlechart under thelatesttag:werf bundle publish --repo example.org/bundles/mybundle
What the published chart contains
The chart directory is published as is: werf does not merge anything into values.yaml, and secret-values.yaml and the secret/ directory of the chart stay where they are, encrypted. werf changes only the following fields in Chart.yaml, keeping the rest of the file, comments and key order included:
name— the last path component of the repository (see Chart name and version);version— the version derived from the tag;appVersion— only if it is set inwerf.yaml;- the
helm.sh/imagesannotation — the werf-built images and their references, added to the entries the chart author wrote. If an entry written by the author has the same name as a werf image, publishing fails: remove the entry fromChart.yamlor rename the werf image.
werf also vendors the dependent charts listed in Chart.lock into charts/ and builds TypeScript charts.
Everything werf adds at publish time goes to .werf-data/:
.werf-data/
metadata.yaml # the format version (apiVersion: v2), the --set* options, the werf images, the --patches files, annotations, labels, the release name and Namespace
values.yaml # the --values files, merged in order
secret-values.yaml # the --secret-values files, merged in order, encrypted
extra-values.yaml # werf auxiliary parameters: images, repository, environment, commit
set-files/0, 1, ... # the contents of the --set-file files
patches/0, 1, ... # the --patches files
.werf-data/ is always published and kept when the chart is unpacked and packed again, even if .helmignore of the chart matches it.
Note:
.werf-data/values.yaml, theset-files/files and the--set*strings are stored in the registry in plaintext. Pass secrets only with--secret-values: werf checks that every value in these files is encrypted and stores them encrypted.
Publishing multiple bundles from a single Git repository
Place the .helm file with the chart contents and its corresponding werf.yaml in a separate directory for each bundle:
bundle1/
.helm/
templates/
# ...
werf.yaml
bundle2/
.helm/
templates/
# ...
werf.yaml
You can now publish each bundle individually:
cd bundle1
werf bundle publish --repo example.org/bundles/bundle1
cd ../bundle2
werf bundle publish --repo example.org/bundles/bundle2
Excluding files or directories from the chart
The .helmignore file of the chart filters the published chart the same way it filters any chart werf reads, see Excluding files or directories from the chart.
The --disable-default-values, --disable-default-secret-values and --no-default-patches options of the werf bundle publish command are recorded in the chart: werf bundle apply then ignores values.yaml, secret-values.yaml and patches.yaml of the chart, as werf converge does with these options. The files themselves stay in the chart, and Helm still uses values.yaml.
If the chart directory already contains .werf-data/ with these switches, e.g. a committed unpacked published chart, werf bundle publish keeps them: an option passed with true sets the stored switch, while an omitted option or an option passed with false keeps it. To clear a stored switch, remove defaultValuesDisable or defaultSecretValuesDisable under values or defaultPatchesDisable from .werf-data/metadata.yaml of the chart.
Using custom image tags
To use custom image tags for bundles, you need to set the naming template using the --use-custom-tag option (WERF_USE_CUSTOM_TAG).
Example usage:
werf bundle publish --repo example.org/bundles/mybundle --use-custom-tag "%image%-my-custom-tag"
werf bundle publish refuses to publish the chart if its version tag or the tag given with --tag equals the tag of a werf image in the same repository, since pushing the chart would overwrite the image. In this case, publish the bundle with another --tag or use another image tag.
You can read more about custom tags in the documentation.
Chart name and version
The chart name is always the last path component of the repository the chart is published to: werf bundle publish --repo example.org/bundles/mybundle publishes the mybundle chart. Helm finds an OCI chart only in a repository named after the chart.
By default, the chart is published under the latest tag. You can specify a different tag, e.g., a semantic version, using the --tag option:
werf bundle publish --repo example.org/bundles/mybundle --tag v1.0.0
The chart version is derived from the tag:
- a strict semantic version (
MAJOR.MINOR.PATCHwith an optional prerelease and build, optionally prefixed withv) is used as is, without thevprefix:v1.0.0→1.0.0; - any other tag (
latest, a commit SHA, a branch name,1.2, a number such as a date or a pipeline ID) gets the0.0.0-<unix time>-<tag slug>version, where the slug is the lowercased tag with each run of characters other than letters and digits replaced by-:feature_Foo_1→0.0.0-<unix time>-feature-foo-1. If the version would be longer than 128 characters, the maximum length of a tag, the slug is cut and suffixed with-and the first 8 hex digits of the SHA-256 of the tag.
werf pushes the chart under its version tag and also under the given tag if it differs, with + replaced by _ in both, as Helm does: --tag 1.0.0+build.5 publishes 1.0.0_build.5. The command above publishes example.org/bundles/mybundle:1.0.0 and example.org/bundles/mybundle:v1.0.0, both pointing to the same chart.
A tag ending in .prov is not allowed, since such tags address provenance files.
A 0.0.0-… version is a prerelease, so Helm, Argo CD and Flux skip it when selecting a version by a semver range: use a range like >=0.0.0-0 or Helm’s --devel option to select it. Two publications with a non-semver tag within the same second get the same version and overwrite each other. Such versions accumulate in the registry: werf cleanup does not remove them.
If the OCI repository finds that a chart with this tag already exists, the chart in the repository will be overwritten.
Deploying a bundle
werf bundle apply deploys the published chart without access to the application’s Git repository:
werf bundle apply --repo example.org/bundles/mybundle --tag v1.0.0
werf bundle publish followed by werf bundle apply deploys the same as werf converge with the options that were passed to werf bundle publish, and werf release get --print-values shows the same values. The options passed to werf bundle apply (--values, --secret-values, --set*, --set-root-json, --patches, --add-annotation, --add-label) are applied on top of the ones stored at publish time and win over them. The parameters are combined in the following order, from the lowest priority to the highest:
values.yamlandsecret-values.yamlof the chart.- The
--values,--secret-values,--set*and--set-root-jsonoptions ofwerf bundle publish. - werf auxiliary parameters computed at publish time (images, repository, environment, commit).
- The
--values,--secret-values,--set*and--set-root-jsonoptions ofwerf bundle apply. - werf auxiliary parameters computed at apply time (environment, Namespace and others).
The --patches files are applied in the same order: the chart’s own patches.yaml, then the ones of werf bundle publish, then the ones of werf bundle apply. The --disable-default-values and --disable-default-secret-values options of werf bundle apply ignore only values.yaml and secret-values.yaml of the chart, not the values stored at publish time.
The --secret-values of werf bundle publish are decrypted at apply time and saved in the release in decrypted form, as werf converge saves its --secret-values, see Secret parameters in the release. Decrypting them requires the secret key they were encrypted with at publish time.
werf bundle publish records the release name and Namespace resolved the same way as werf converge resolves them for the given --env. werf bundle apply, werf bundle plan and werf bundle render use them for --release and --namespace that are not passed, also when --env is not passed. If you pass a non-empty --env that differs from the one used at publish time, the recorded ones are not used and both --release and --namespace are required.
The one known difference from werf converge is .Chart.Name and .Chart.Version: under werf converge templates get the project name and the version from Chart.yaml (1.0.0 by default), under werf bundle apply they get the repository name and the version derived from the tag. A label such as helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }} therefore differs, and moving a release from werf converge to werf bundle apply restarts its Pods once.
werf converge, werf plan, werf render and werf lint also use .werf-data/ if they find it in the chart directory (for example, when .helm contains an unpacked published chart): its values, annotations and labels are applied below the options of the command.
Using a published chart as a dependent chart
When werf or Nelm deploys a chart that includes a werf-published chart as a dependent chart, the values from .werf-data/ of the dependent chart become its default values, as if it were deployed alone with werf converge. The parent chart overrides them as usual. The secret values of the dependent chart are decrypted with the secret key of the deployment, and, like the parameters from secret-values.yaml of any dependent chart, are saved in the release in decrypted form. Its release-wide inputs are ignored with a warning, since the parent chart decides them: annotations and labels, --patches, --no-default-patches, --set-root-json and the release name and Namespace. werf bundle publish always records the release name and Namespace, so deploying any werf-published dependent chart prints this warning.
The global.* values, including global.werf.*, are shared with the parent chart, as Helm does for global values: where the parent chart and the dependent chart both set a global key, the dependent chart gets the parent chart’s value. The werf.* values of the dependent chart stay its own. For example, in the templates of a dependent chart published with an image app:
image: {{ .Values.werf.image.app }} # the image published with the dependent chart
repo: {{ .Values.global.werf.repo }} # the repository of the parent chart's werf project
Changing a published chart
A published chart can be downloaded, changed and published again without the application’s Git repository, for example, with the Nelm CLI:
-
Download and unpack the chart:
nelm chart download oci://example.org/bundles/mybundle --version 1.0.0 --untar -
Change the chart:
values.yaml, secrets (nelm chart secret values-file edit mybundle/secret-values.yamlormybundle/.werf-data/secret-values.yaml), templates, or any file in.werf-data/. -
Pack the chart and upload it:
nelm chart pack mybundle --set replicas=3 nelm chart upload mybundle-1.0.0.tgz oci://example.org/bundles
nelm chart pack keeps the version from Chart.yaml unless --chart-version is passed, and saves the archive to the current directory unless --destination is passed. The --values, --secret-values, --set*, --set-root-json and --patches options of nelm chart pack are added on top of the ones stored in .werf-data/, as if they were appended to the original werf bundle publish command line: the values files are merged over the stored ones, and the --set*, --set-root-json and --patches options are appended to the stored ones. The --no-default-values, --no-default-secret-values and --no-default-patches options set the stored switches, but can’t clear them. The --reset-values option drops everything stored in .werf-data/ before applying the options of this pack. To remove a single stored option, edit .werf-data/ and pack the chart. When removing a stored values file (.werf-data/values.yaml, .werf-data/secret-values.yaml or .werf-data/extra-values.yaml), also remove its flag from the values section of .werf-data/metadata.yaml (values: true, secretValues: true or extraValues: true), otherwise packing and deploying the chart fail with an error such as .werf-data/metadata.yaml: values: true does not match the presence of values.yaml. nelm chart pack does not need the secret key.
A TypeScript chart is rebuilt when packed, which requires network access to download its modules. To reuse the bundled TypeScript code of the downloaded chart instead, pass --ts-no-rebuild.
A chart published by werf 3.12 or earlier can be repacked too: nelm chart pack moves its extra_annotations.json and extra_labels.json into .werf-data/, while its values.yaml, which contains the values merged at publish time, stays the chart’s own values.yaml.
Instead of nelm chart pack and nelm chart upload, you can use helm package and helm push: they keep .werf-data/ unless the chart’s .helmignore matches it.
Changing the version of the published chart
To change the tag of a published chart, copy the bundle and add the new tag to it using the werf bundle copy command, for example:
werf bundle copy --from example.org/bundles/mybundle:v1.0.0 --to example.org/bundles/renamedbundle:v2.0.0
werf bundle copy --to REPO:TAG sets the version of the copy from TAG by the same rules as werf bundle publish and pushes the copy under the version tag and the given tag. werf bundle copy --to REPO without a tag keeps the version and tag of the source chart. The copy is named after the last path component of the destination repository.
Copying the bundle to a different repository
The werf bundle copy command provides a convenient way to copy the bundle to another repository. Besides copying the chart and its related images, this command regenerates the image references in .werf-data/extra-values.yaml and in the helm.sh/images annotation for the destination repository.
Example:
werf bundle copy --from example.org/bundles/mybundle:v1.0.0 --to other.example.org/bundles/mybundle:v1.0.0
The images are copied by digest, multi-platform images with all their platforms. The images of charts with .werf-data/ included as dependent charts, at any depth, are copied too, and their helm.sh/images annotations and .werf-data/extra-values.yaml are rewritten for the destination repository. A bundle published by werf 3.12 or older and included as a dependent chart has no .werf-data/: its images are not copied and its values are not rewritten.
A bundle published by werf 3.12 or older has no .werf-data/: its images are taken from the werf.image values, and the werf.image, werf.repo and global.werf values in its values.yaml are rewritten for the destination repository.
A signed chart (with a Helm provenance file) keeps its signature when copied unchanged. If the copy changes the chart, e.g. rewrites the image references, renames it or changes its version, the signature is dropped with a warning.
werf bundle apply, werf bundle plan and werf bundle render verify the signature of the bundle pulled from the container registry against the public keys from --provenance-keyring when --provenance-strategy=verify (an unsigned bundle fails) or --provenance-strategy=verify-if-possible (only a signed bundle is verified) is passed. werf bundle render --bundle-dir can’t verify the signature of an unpacked bundle and fails with --provenance-strategy=verify.
Exporting the bundle from the container registry to the archive
After publication, the bundle can be exported from the repository to a local archive for distribution by other means using the werf bundle copy command, for example:
werf bundle copy --from example.org/bundles/mybundle:v1.0.0 --to archive:archive.tar.gz
The archive is a tar file, gzip-compressed if its name ends with .gz or .tgz, with an OCI image layout inside (oci-layout, index.json, blobs/sha256/…). It holds the chart, with its provenance file if the chart is signed, and every image of the chart by digest:
- multi-platform images are kept with all their platforms;
- non-distributable (foreign) image layers are copied into the archive too;
- referrers of the images, such as signatures and SBOMs attached to them, are not copied.
werf 3.12 and older can’t read these archives: import them with the current werf.
Importing the bundle from the archive to the repository
The exported to the archive bundle can be imported back into the same or another OCI repository using the werf bundle copy command, for example:
werf bundle copy --from archive:archive.tar.gz --to other.example.org/bundles/mybundle:v1.0.0
Then the newly published bundle (a chart and its images) can be used as usual.
Archives written by werf 3.12 and older (chart.tar.gz and images/*.tar.gz inside) can still be imported. They have no tag, so without a tag in --to the bundle is pushed under its chart version only.
Using the chart with Helm, Argo CD and Flux
Helm, Argo CD and Flux install the published chart with its own values.yaml and the values they are given at install time. They do not read .werf-data/ by themselves, so the image references and the values passed to werf bundle publish are missing: templates that use .Values.werf.image.* or .Values.global.werf.images.* render empty references.
Note: charts published by werf 3.12 and earlier merged the image references and the
--valuesand--set*options ofwerf bundle publishintovalues.yaml. The new charts do not, so a Helm, Argo CD or Flux deployment of a newly published chart needs the values files below.
.werf-data/values.yaml (the --values of werf bundle publish) and .werf-data/extra-values.yaml (the image references and other werf auxiliary parameters) are plain YAML files inside the chart, so you can pass them as values files. werf bundle publish always writes .werf-data/extra-values.yaml, but .werf-data/values.yaml only when it gets --values: either list only the files the chart has, or tell Flux and Argo CD to skip missing files, as below.
Flux, a HelmRelease whose chart comes from a HelmRepository of the oci type:
spec:
chart:
spec:
chart: mybundle
sourceRef:
kind: HelmRepository
name: bundles
valuesFiles:
- values.yaml
- .werf-data/values.yaml
- .werf-data/extra-values.yaml
ignoreMissingValuesFiles: true
valuesFiles replaces the default list of values files, so values.yaml must be listed first.
Argo CD, the paths are relative to the chart root:
spec:
source:
repoURL: example.org/bundles
chart: mybundle
targetRevision: 1.0.0
helm:
valueFiles:
- .werf-data/values.yaml
- .werf-data/extra-values.yaml
ignoreMissingValueFiles: true
Argo CD pulls the chart from a container registry only if the repository (example.org/bundles) is registered in Argo CD as a Helm repository with enableOCI: "true", e.g. in a repository Secret with type: helm.
The following stays unavailable with Helm, Argo CD and Flux even with these values files:
- the
--set*options ofwerf bundle publish; - secrets: Helm cannot decrypt
secret-values.yaml,.werf-data/secret-values.yamland thesecret/files, and templates that callwerf_secret_filefail; - the annotations and labels of
werf bundle publish; --disable-default-valuesofwerf bundle publish: Helm still usesvalues.yamlof the chart;- resources generated by TypeScript: Helm does not run
ts/; - everything from
.werf-data/when the werf-published chart is a dependent chart of a chart deployed by Helm, Argo CD or Flux.
Helm puts .werf-data/ into .Files and saves it in the release together with the rest of the chart (.werf-data/secret-values.yaml stays encrypted). Templates that read all chart files, e.g. with .Files.Glob "**", get the .werf-data/ files too.
Compatibility with other werf and Nelm versions
Charts published by werf 3.12 and earlier are deployed by the current werf unchanged.
The current chart format (apiVersion: v2 in .werf-data/metadata.yaml) requires werf v3 or Nelm v2 with .werf-data/ support. werf v2 and Nelm v1 releases that include the check (werf/werf#8141, werf/nelm#808) refuse to deploy such a chart, as well as a chart that includes it as a dependent chart. werf v2 and Nelm v1 releases without the check, werf 3.12 and earlier, and Nelm v2 versions without .werf-data/ support do not refuse: they deploy the chart with its own values only, without the image references, the values, secrets, annotations and labels of werf bundle publish.
Container registries that support the publication of bundles
Publishing bundles requires a container registry to support the OCI (Open Container Initiative) specification. Below is a list of the most popular container registries that have been tested and found to be compatible:
| Container registry | Supports bundle publishing |
|---|---|
| AWS ECR | + |
| Azure CR | + |
| Docker Hub | + |
| GCR | + |
| GitHub Packages | + |
| GitLab Registry | + |
| Harbor | + |
| JFrog Artifactory | + |
| Yandex container registry | + |
| Nexus | + |
| Quay | - |