Configuration and secrets

In this chapter, we will show how to use and store sensitive and non-sensitive application configurations properly.

In previous chapters, we added the configuration directly to containers during the build or used the container’s environment variables to pass parameters during the deployment.

Now, you will learn how to store application parameters in ConfigMaps and Secrets for security and flexibility. We will show how you can use Helm chart values and werf secrets and discuss parameterization and configuration reuse approaches. In addition, you will learn how to store sensitive data along with the code in the application’s Git repository.

The application described in this chapter is not intended for use in production environments as-is. Note that successful completion of this entire guide is required to create a production-ready application.

Preparing the environment

If you haven’t prepared your environment during previous steps, please, do it using the instructions provided in the “Preparing the environment” chapter.

If your environment has stopped working or instructions in this chapter don’t work, please, refer to these hints:

docker run hello-world

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:9f6ad537c5132bcce57f7a0a20e317228d382c3cd61edae14650eec68b2b345c
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

docker run hello-world

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:9f6ad537c5132bcce57f7a0a20e317228d382c3cd61edae14650eec68b2b345c
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

sudo systemctl restart docker

sudo systemctl status docker

● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2021-06-24 13:05:17 MSK; 13s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 2013888 (dockerd)
      Tasks: 36
     Memory: 100.3M
     CGroup: /system.slice/docker.service
             └─2013888 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

dockerd[2013888]: time="2021-06-24T13:05:16.936197880+03:00" level=warning msg="Your kernel does not support CPU realtime scheduler"
dockerd[2013888]: time="2021-06-24T13:05:16.936219851+03:00" level=warning msg="Your kernel does not support cgroup blkio weight"
dockerd[2013888]: time="2021-06-24T13:05:16.936224976+03:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"
dockerd[2013888]: time="2021-06-24T13:05:16.936311001+03:00" level=info msg="Loading containers: start."
dockerd[2013888]: time="2021-06-24T13:05:17.119938367+03:00" level=info msg="Loading containers: done."
dockerd[2013888]: time="2021-06-24T13:05:17.134054120+03:00" level=info msg="Daemon has completed initialization"
systemd[1]: Started Docker Application Container Engine.
dockerd[2013888]: time="2021-06-24T13:05:17.148493957+03:00" level=info msg="API listen on /run/docker.sock"

docker run hello-world

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:9f6ad537c5132bcce57f7a0a20e317228d382c3cd61edae14650eec68b2b345c
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

minikube start

kubectl config set-context minikube --namespace=werf-guide-app

😄  minikube v1.20.0 on Ubuntu 20.04
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🎉  minikube 1.21.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.21.0
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/google_containers/kube-registry-proxy:0.4
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
    ▪ Using image registry:2.7.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying registry addon...
🔎  Verifying ingress addon...
🌟  Enabled addons: storage-provisioner, registry, default-storageclass, ingress
🏄  Done! kubectl is now configured to use "minikube" cluster and "werf-guide-app" namespace by default

Restarting existing docker container for "minikube"

minikube tunnel --cleanup=true

minikube start --namespace werf-guide-app

kubectl config set-context minikube --namespace=werf-guide-app

😄  minikube v1.20.0 on Ubuntu 20.04
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🎉  minikube 1.21.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.21.0
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/google_containers/kube-registry-proxy:0.4
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
    ▪ Using image registry:2.7.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying registry addon...
🔎  Verifying ingress addon...
🌟  Enabled addons: storage-provisioner, registry, default-storageclass, ingress
🏄  Done! kubectl is now configured to use "minikube" cluster and "werf-guide-app" namespace by default

Restarting existing docker container for "minikube"

kubectl create namespace werf-guide-app
kubectl create secret docker-registry registrysecret \
  --docker-server='https://index.docker.io/v1/' \
  --docker-username='<Docker Hub username>' \
  --docker-password='<Docker Hub password>'

namespace/werf-guide-app created
secret/registrysecret created

Preparing the repository

cd ~/werf-guide/app

# To see what changes we will make later in this chapter, let's replace all the application files
# in the repository with new, modified files containing the changes described below.
git rm -r .
cp -Recurse -Force ~/werf-guide/guides/examples/laravel/080_configuration/* .
git add .
git commit -m WIP

# Enter the command below to show the files we are going to change.
git show --stat
# Enter the command below to show the changes that will be made.
git show

cd ~/werf-guide/app

# To see what changes we will make later in this chapter, let's replace all the application files
# in the repository with new, modified files containing the changes described below.
git rm -r .
cp -rf ~/werf-guide/guides/examples/laravel/080_configuration/. .
git add .
git commit -m WIP

# Enter the command below to show files we are going to change.
git show --stat
# Enter the command below to show the changes that will be made.
git show

# Clone the example repository to ~/werf-guide/guides (if you have not cloned it yet).
if (-not (Test-Path ~/werf-guide/guides)) {
  git clone https://github.com/werf/website $env:HOMEPATH/werf-guide/guides
}

# Copy the (unchanged) application files to ~/werf-guide/app.
rm -Recurse -Force ~/werf-guide/app
cp -Recurse -Force ~/werf-guide/guides/examples/laravel/050_s3 ~/werf-guide/app

# Make the ~/werf-guide/app directory a git repository.
cd ~/werf-guide/app
git init
git add .
git commit -m initial

# To see what changes we will make later in this chapter, let's replace all the application files
# in the repository with new, modified files containing the changes described below.
git rm -r .
cp -Recurse -Force ~/werf-guide/guides/examples/laravel/080_configuration/* .
git add .
git commit -m WIP

# Enter the command below to show the files we are going to change.
git show --stat
# Enter the command below to show the changes that will be made.
git show

# Clone the example repository to ~/werf-guide/guides (if you have not cloned it yet).
test -e ~/werf-guide/guides || git clone https://github.com/werf/website ~/werf-guide/guides

# Copy the (unchanged) application files to ~/werf-guide/app.
rm -rf ~/werf-guide/app
cp -rf ~/werf-guide/guides/examples/laravel/050_s3 ~/werf-guide/app

# Make the ~/werf-guide/app directory a git repository.
cd ~/werf-guide/app
git init
git add .
git commit -m initial

# To see what changes we will make later in this chapter, let's replace all the application files
# in the repository with new, modified files containing the changes described below.
git rm -r .
cp -rf ~/werf-guide/guides/examples/laravel/080_configuration/. .
git add .
git commit -m WIP

# Enter the command below to show files we are going to change.
git show --stat
# Enter the command below to show the changes that will be made.
git show

ConfigMap and Secret

ConfigMap and Secret Kubernetes resources allow you to separate environment-dependent and context-specific configuration from container images.

Both objects store data in key-value pairs and provide them to Pods as environment variables, command-line arguments, or configuration files mounted in a selected container.

ConfigMap stores non-confidential data, while Secret stores confidential data (various secret types).

You can learn more about these resource types in the official Kubernetes documentation (ConfigMaps, Secrets). This chapter discusses various use cases for ConfigMaps and Secrets using our application as an example.

Storing application’s config files in the ConfigMap

Currently, the nginx.conf config file gets copied into the image during the build. As a result, the image is rebuilt, and Pods are restarted each time the file is modified. In addition, there is no easy way to benefit from templating in our nginx.conf.

You can get around these obstacles by moving .werf/nginx.conf into a dedicated ConfigMap. This way, you can mount nginx.conf when deploying instead of adding the file during the build:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    user nobody;
    worker_processes 1;
    pid /run/nginx.pid;

    events {
      worker_connections 1024;
    }

    http {
      include /etc/nginx/mime.types;
      default_type application/octet-stream;

      log_format json_combined escape=json
          '{'
            '"time_local":"$time_iso8601",'
            '"client_ip":"$http_x_forwarded_for",'
            '"remote_addr":"$remote_addr",'
            '"remote_user":"$remote_user",'
            '"request":"$request",'
            '"status":"$status",'
            '"body_bytes_sent":"$body_bytes_sent",'
            '"request_time":"$request_time",'
            '"http_referrer":"$http_referer",'
            '"http_user_agent":"$http_user_agent",'
            '"request_id":"$request_id"'
          '}';

      access_log /dev/stdout json_combined;

      server {
        listen 8080;
        server_name _;

        root /www;
        index index.php;
        error_page 404 /index.php;

        charset utf-8;

        add_header X-Frame-Options "SAMEORIGIN";
        add_header X-Content-Type-Options "nosniff";

        location / {
            try_files $uri $uri/ /index.php?$query_string;
        }

        location = /favicon.ico { access_log off; log_not_found off; }
        location = /robots.txt  { access_log off; log_not_found off; }

        location ~ \.php$ {
            fastcgi_pass unix:/var/run/php/php-fpm.sock;
            fastcgi_param DOCUMENT_ROOT   /app/public;
            fastcgi_param SCRIPT_FILENAME /app/public/$fastcgi_script_name;
            include fastcgi_params;
        }
      }
    }

Now, let’s add our ConfigMap to the Deployment — we will mount it as a file in the frontend container:

...
- name: frontend
  image: {{ .Values.werf.image.frontend }}
  ports:
    - containerPort: 8080
      name: http
  volumeMounts:
    - mountPath: /var/run/php
      name: run-php
    - mountPath: /etc/nginx/nginx.conf
      subPath: nginx.conf
      name: nginx-conf

We also need to delete the no longer needed .werf/nginx.conf as well as a command to copy this file to an image during the build. The Dockerfile for the frontend image will look as follows:

...
FROM nginx:stable-alpine as frontend
WORKDIR /www

COPY --from=assets /app/public /www

EXPOSE 8080

Handling ConfigMap and Secret changes

Note that by default, changes in ConfigMaps or Secrets mounted in a Deployment, StatefulSet, or DaemonSet will not trigger Pod restarts for the new configuration to take effect. To trigger Pod restart, you need to add annotations with checksums of all ConfigMaps and Secrets used by this Pod. In this case, the annotations will change in response to ConfigMap and Secret modifications, leading to the Pod restart. Below is an example of an annotation that includes the checksum of the nginx.conf ConfigMap:

...
template:
  metadata:
    labels:
      app: werf-guide-app
    annotations:
      checksum/configmap-nginx: '{{ include (print $.Template.BasePath "/configmap-nginx.yaml") . | sha256sum }}'

Note that you will need a unique annotation for each mounted ConfigMap/Secret.

The authors of this tutorial prefer to use operators like stakater/Reloader instead of checksum-containing annotations because they are simpler, more flexible, and easier to work with.

Values

Using a Helm chart to deploy an application has many advantages — for example, you can create manifest templates with Helm charts. Values is the key built-in templating object. With it, you can access values passed to the chart from various sources.

When using werf, all data passed to the chart can be grouped into several categories:

• User-defined regular values: values.yaml parameters and corresponding options.
• User-defined secret values: parameters from the secret-values.yaml file.
• Service values: information about the project, release, built images, etc.

Reusing configuration with Values and ConfigMaps

Frequently used parameters can be put into .helm/values.yaml:

...
mysql:
  storageSize: "100Mi"

… to make manifest configuration easier:

...
volumeClaimTemplates:
- metadata:
    name: mysql-data
  spec:
    accessModes: ["ReadWriteOnce"]
    resources:
      requests:
        storage: "{{ .Values.mysql.storageSize }}"

This way, you can adjust the parameters to the specific environment (we will discuss this in more detail in the following tutorial chapters).

Probably, the most useful feature is the ability to place repetitive configuration snippets into .helm/values.yaml. For example, you can store in this file environment variables used in multiple places:

...
app:
  envs:
    LOG_CHANNEL: stderr
    LOG_STDERR_FORMATTER: Monolog\\Formatter\\JsonFormatter
    LOG_LEVEL: info
    DB_HOST: mysql
    DB_DATABASE: werf-guide-app
    AWS_ENDPOINT: "http://minio:9000"
    AWS_DEFAULT_REGION: us-east-1
    AWS_BUCKET: werf-guide-app
    AWS_USE_PATH_STYLE_ENDPOINT: "true"

The environment variables from .Values.app.envs can be inserted either into the container manifest as env values (as we did before), or you can put them into the ConfigMap and load it into the container via envFrom.

The first option is easier to do, but the ConfigMap option is more convenient if you have a large number of shared environment variables. This way, you will avoid duplicating them between controllers. Your ConfigMap may look like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-envs
data:
  {{- range $key, $val := .Values.app.envs }}
  "{{ $key }}": "{{ $val }}"
  {{- end }}

Now you can use envFrom in the Deployment to define ConfigMap’s data as container environment variables:

...
containers:
  - name: backend
    image: {{ .Values.werf.image.backend }}
    volumeMounts:
      - mountPath: /var/run/php
        name: run-php
    envFrom:
    - configMapRef:
        name: app-envs

Don’t forget to add checksum-containing annotations to trigger Pod re-deployment in response to ConfigMap changes:

...
template:
  metadata:
    labels:
      app: werf-guide-app
    annotations:
      checksum/configmap-nginx: '{{ include (print $.Template.BasePath "/configmap-nginx.yaml") . | sha256sum }}'
      checksum/configmap-app-envs: '{{ include (print $.Template.BasePath "/configmap-app-envs.yaml") . | sha256sum }}'

Similarly, you can load this ConfigMap into other resources that use the same environment variables.

Handling confidential data using Values and Secrets

To start working with secrets, you first need to generate a symmetric encryption key. You can use the following command to do this: werf helm secret generate-secret-key. However, since we have already prepared/encrypted the secrets for you, the encryption key that will be used in this chapter is generated as well. This encryption key is stored in the repository in the .werf_secret_key file; werf uses it automatically.

Caution! You MUST NOT store the key in the repository when working with real-life applications. We recommend keeping it in a safe place and passing via the WERF_SECRET_KEY environment variable. You can read more about using encryption keys in the werf documentation.

The application configuration contains data that should not be stored in unencrypted form in the repository (e.g., the database login and password). Thus, we won’t store the login and password in plain text in the controller manifest. Instead, we will store them encrypted in the .helm/secret-values.yaml file along with other confidential parameters:

app:
  secretEnvs:
    APP_KEY: 10000f02004508feddf9746539c6f937238dfa0bb1cdf3e628547721d868ad1ba5b5a9a97535d7a4d218355bd2a0df22972e90b2a2c91a0f1e754c2c0b40aa4145b44d9c202535f945d95e14f3bb20399b65
    DB_USERNAME: 1000eb0a862c8a80dec5e7cd5c179177e9fe68ecc55896a01f0f16f5212c4ad9522d
    DB_PASSWORD: 10001bd1c973a299589f796896bcdb9afce3840dd3990537a18fd453ed7b2dca6747
    AWS_ACCESS_KEY_ID: 10004ce322cc0d7b02845cfa4b2b2a1f8b9a24dbb1849b6229484af470374b97ca7c
    AWS_SECRET_ACCESS_KEY: 1000ee9f9f758763cdfef3a8e78565bb05b0364bb6b6b43fca06cd4016f149d965f2
mysql:
  secretEnvs:
    MYSQL_ROOT_PASSWORD: 10007d80d8b6dff5ef78b73866b2fc7c167615f5974fa9b4f68b44595b8c203be855
minio:
  secretEnvs:
    MINIO_ROOT_USER: 1000347c3348673966f1f6a0c037ad8bdd3cda7cfedc989b8a0edf7a1b97fad99e8c
    MINIO_ROOT_PASSWORD: 1000216183bc27254749b6012ed68bb6af7d298e6b64d63ade285ed5f5d5da202000

Use the command below to decrypt the secrets stored in .helm/secret-values.yaml:

werf helm secret values decrypt .helm/secret-values.yaml

You should see the following output:

app:
  secretEnvs:
    APP_KEY: base64:GcPVmSxMZwsOJtNOJ9eVNNeU6B5buHuln93+w0TSvfE=
    DB_USERNAME: root
    DB_PASSWORD: password
    AWS_ACCESS_KEY_ID: minioadmin
    AWS_SECRET_ACCESS_KEY: minioadmin
mysql:
  secretEnvs:
    MYSQL_ROOT_PASSWORD: password
minio:
  secretEnvs:
    MINIO_ROOT_USER: minioadmin
    MINIO_ROOT_PASSWORD: minioadmin

Now, you need to pass the secrets contained in .helm/secret-values.yaml back to the application config. To do this, let’s first pass them to the Secret resource:

apiVersion: v1
kind: Secret
metadata:
  name: app-envs
type: Opaque
data:
  {{- range $key, $val := .Values.app.secretEnvs }}
  "{{ $key }}": "{{ $val | b64enc }}"
  {{- end }}

… and then load this resource’s data as a set of environment variables by mounting it in containers:

...
containers:
  - name: backend
    image: {{ .Values.werf.image.backend }}
    volumeMounts:
      - mountPath: /var/run/php
        name: run-php
    envFrom:
    - configMapRef:
        name: app-envs
    - secretRef:
        name: app-envs

You have to make similar changes to the rest of the .helm/templates files (we won’t delve into details here; you can take a look at their contents in the repository.)

You can also use Secrets to store and mount complete secret configuration files. This is similar to using ConfigMap to mount non-secret configuration files (as described above), except that the Secret contents must be stored encrypted in .helm/secret/... or .helm/secret-values.yaml. You can read more about this in the werf documentation.

Note: learn more about working with secrets in the werf documentation.

Checking whether the application runs as expected

Let’s make sure the configuration changes did not affect the application performance:

werf converge --repo <DOCKER HUB USERNAME>/werf-guide-app

You should see the following output:

...
┌ ⛵ image backend
│ ┌ Building stage backend/dockerfile
│ │ backend/dockerfile  Sending build context to Docker daemon    533kB
│ │ backend/dockerfile  Step 1/19 : FROM php:8.0-fpm-alpine as base
│ │ backend/dockerfile   ---> 52c511f481c5
...
│ │ backend/dockerfile  Successfully built 91ec44f42d84
│ │ backend/dockerfile  Successfully tagged d972fcf1-88cc-4a11-a2b8-e76e397914f1:latest
│ ├ Info
│ │      name: <DOCKER HUB USERNAME>/werf-guide-app/werf-guide-app:b12f6a22c119f88d808e9bf5daf5bb0eab8138f0539769e478647e70-1634037356083
│ │        id: 91ec44f42d84
│ │   created: 2022-10-12 14:15:55 +0000 UTC
│ │      size: 58.9 MiB
│ └ Building stage backend/dockerfile (19.73 seconds)
└ ⛵ image backend (24.98 seconds)

┌ ⛵ image frontend
│ ┌ Building stage frontend/dockerfile
│ │ frontend/dockerfile  Sending build context to Docker daemon    533kB
│ │ frontend/dockerfile  Step 1/29 : FROM php:8.0-fpm-alpine as base
│ │ frontend/dockerfile   ---> 52c511f481c5
...
│ │ frontend/dockerfile  Successfully built a61e4952387d
│ │ frontend/dockerfile  Successfully tagged b6cc14ad-f7d6-47d9-a53d-b53cd80838d3:latest
│ ├ Info
│ │      name: <DOCKER HUB USERNAME>/werf-guide-app/werf-guide-app:55170630d763de2af7ac5f4ca4a3c1bdb4f98c9782e99fdd8b07d2bb-1634037415695
│ │        id: a61e4952387d
│ │   created: 2022-10-12 14:16:55 +0000 UTC
│ │      size: 9.4 MiB
│ └ Building stage frontend/dockerfile (77.28 seconds)
└ ⛵ image frontend (82.49 seconds)

┌ Waiting for release resources to become ready
│ ┌ Status progress
│ │ DEPLOYMENT                                                                                      REPLICAS          AVAILABLE           UP-TO-DATE
│ │ werf-guide-app                                                                                  1/1               1                   1
│ │ │   POD                                 READY        RESTARTS          STATUS
│ │ ├── guide-app-54d48f7d4-zvkqd           2/2          0                 Running
│ │ └── guide-app-6b8cf46f5b-zj4d9          2/2          0                 Terminating
│ │ STATEFULSET                                                                                     REPLICAS          READY               UP-TO-DATE
│ │ minio                                                                                           1/1               1                   1
│ │ mysql                                                                                           1/1               0->1                1                            ↵
│ │
│ │ │   POD                                 READY        RESTARTS          STATUS
│ │ └── 0                                   0/1          0                 ContainerCreating
│ └ Status progress
└ Waiting for release resources to become ready (30.34 seconds)

┌ Waiting for helm hook job/migrate-db termination
│ ┌ job/migrate-db po/migrate-db-vvtfd container/migrate-db logs
│ │ mysqld is alive
│ │ Nothing to migrate.
│ └ job/migrate-db po/migrate-db-vvtfd container/migrate-db logs
│
│ ┌ Status progress
│ │ JOB                                                                                             ACTIVE            DURATION            SUCCEEDED/FAILED
│ │ migrate-db                                                                                      0                 13s                 0->1/0
│ │ │   POD                                 READY        RESTARTS          STATUS
│ │ └── db-vvtfd                            0/1          0                 Running -> Completed
│ └ Status progress
└ Waiting for helm hook job/migrate-db termination (13.08 seconds)

┌ Waiting for helm hook job/setup-minio termination
│ ┌ Status progress
│ │ JOB                                                                                             ACTIVE            DURATION            SUCCEEDED/FAILED
│ │ setup-minio                                                                                     0                 13s                 0->1/0
│ │ │   POD                                 READY        RESTARTS          STATUS
│ │ └── minio-d8cmm                         0/1          0                 Running -> Completed
│ └ Status progress
└ Waiting for helm hook job/setup-minio termination (14.02 seconds)

Release "werf-guide-app" has been upgraded. Happy Helming!
NAME: werf-guide-app
LAST DEPLOYED: Tue Oct 12 14:17:12 2022
NAMESPACE: werf-guide-app
STATUS: deployed
REVISION: 20
TEST SUITE: None
Running time 155.26 seconds

Check if the application is available:

curl http://werf-guide-app.test/ping

You should see the following output:

pong

In this chapter, you learned how to securely store the confidential application parameters, minimize duplicate configuration, and turn application configs into templates (thus, no re-builds are necessary when changes are made).