In this chapter, we will implement the ability to download and upload files in the application. In addition, we will discuss some aspects of working with files in Kubernetes and provide a practical example of using S3-compatible storage.

The application described in this chapter is not intended for use in production environments as-is. Note that successful completion of this entire guide is required to create a production-ready application.

Preparing the environment

If you haven’t prepared your environment during previous steps, please, do it using the instructions provided in the “Preparing the environment” chapter.

If your environment has stopped working or instructions in this chapter don’t work, please, refer to these hints:

Is Docker running?

Let’s launch Docker Desktop. It takes some time for this application to start Docker. If there are no errors during the startup process, check that Docker is running and is properly configured:

docker run hello-world

You will see the following output if the command completes successfully:

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:9f6ad537c5132bcce57f7a0a20e317228d382c3cd61edae14650eec68b2b345c
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

Should you have any problems, please refer to the Docker documentation.

Let’s launch the Docker Desktop application. It takes some time for the application to start Docker. If there are no errors during the startup process, then check that Docker is running and is properly configured:

docker run hello-world

You will see the following output if the command completes successfully:

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:9f6ad537c5132bcce57f7a0a20e317228d382c3cd61edae14650eec68b2b345c
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

Should you have any problems, please refer to the Docker documentation.

Start Docker:

sudo systemctl restart docker

Make sure that Docker is running:

sudo systemctl status docker

If the Docker start is successful, you will see the following output:

● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2021-06-24 13:05:17 MSK; 13s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 2013888 (dockerd)
      Tasks: 36
     Memory: 100.3M
     CGroup: /system.slice/docker.service
             └─2013888 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

dockerd[2013888]: time="2021-06-24T13:05:16.936197880+03:00" level=warning msg="Your kernel does not support CPU realtime scheduler"
dockerd[2013888]: time="2021-06-24T13:05:16.936219851+03:00" level=warning msg="Your kernel does not support cgroup blkio weight"
dockerd[2013888]: time="2021-06-24T13:05:16.936224976+03:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"
dockerd[2013888]: time="2021-06-24T13:05:16.936311001+03:00" level=info msg="Loading containers: start."
dockerd[2013888]: time="2021-06-24T13:05:17.119938367+03:00" level=info msg="Loading containers: done."
dockerd[2013888]: time="2021-06-24T13:05:17.134054120+03:00" level=info msg="Daemon has completed initialization"
systemd[1]: Started Docker Application Container Engine.
dockerd[2013888]: time="2021-06-24T13:05:17.148493957+03:00" level=info msg="API listen on /run/docker.sock"

Now let’s check if Docker is available and its configuration is correct:

docker run hello-world

You will see the following output if the command completes successfully:

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:9f6ad537c5132bcce57f7a0a20e317228d382c3cd61edae14650eec68b2b345c
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

Should you have any problems, please refer to the Docker documentation.

Have you restarted the computer after setting up the environment?

Let’s start the minikube cluster we have already configured in the “Preparing the environment” chapter:

minikube start

Set the default Namespace so that you don’t have to specify it every time you invoke kubectl:

kubectl config set-context minikube --namespace=werf-guide-app

You will see the following output if the command completes successfully:

😄  minikube v1.20.0 on Ubuntu 20.04
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🎉  minikube 1.21.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.21.0
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/google_containers/kube-registry-proxy:0.4
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
    ▪ Using image registry:2.7.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying registry addon...
🔎  Verifying ingress addon...
🌟  Enabled addons: storage-provisioner, registry, default-storageclass, ingress
🏄  Done! kubectl is now configured to use "minikube" cluster and "werf-guide-app" namespace by default

Make sure that the command output contains the following line:

Restarting existing docker container for "minikube"

Its absence means that a new minikube cluster was created instead of using the old one. In this case, repeat all the steps required to install the environment using minikube.

Now run the command in the background PowerShell terminal (do not close its window):

minikube tunnel --cleanup=true

Let’s start the minikube cluster we have already configured in the “Preparing the environment” chapter:

minikube start --namespace werf-guide-app

Set the default Namespace so that you don’t have to specify it every time you invoke kubectl:

kubectl config set-context minikube --namespace=werf-guide-app

You will see the following output if the command completes successfully:

😄  minikube v1.20.0 on Ubuntu 20.04
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🎉  minikube 1.21.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.21.0
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/google_containers/kube-registry-proxy:0.4
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
    ▪ Using image registry:2.7.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying registry addon...
🔎  Verifying ingress addon...
🌟  Enabled addons: storage-provisioner, registry, default-storageclass, ingress
🏄  Done! kubectl is now configured to use "minikube" cluster and "werf-guide-app" namespace by default

Make sure that the command output contains the following line:

Restarting existing docker container for "minikube"

Its absence means that a new minikube cluster was created instead of using the old one. In this case, repeat all the steps required to install the environment from scratch using minikube.

Did you accidentally delete the application's Namespace?

If you have inadvertently deleted Namespace of the application, you must run the following commands to proceed with the guide:

kubectl create namespace werf-guide-app
kubectl create secret docker-registry registrysecret \
  --docker-server='https://index.docker.io/v1/' \
  --docker-username='<Docker Hub username>' \
  --docker-password='<Docker Hub password>'

You will see the following output if the command completes successfully:

namespace/werf-guide-app created
secret/registrysecret created
Nothing helps; the environment or instructions keep failing.

If nothing worked, repeat all the steps described in the “Preparing the environment” chapter and create a new environment from scratch. If creating an environment from scratch did not help either, please, tell us about your problem in our Telegram chat or create an issue on GitHub. We will be happy to help you!

Preparing the repository

Update the existing repository containing the application:

Run the following commands in PowerShell:

cd ~/werf-guide/app

# To see what changes we will make later in this chapter, let's replace all the application files
# in the repository with new, modified files containing the changes described below.
git rm -r .
cp -Recurse -Force ~/werf-guide/guides/examples/nodejs/050_s3/* .
git add .
git commit -m WIP
What changes we will make
# Enter the command below to show the files we are going to change.
git show --stat
# Enter the command below to show the changes that will be made.
git show

Run the following commands in Bash:

cd ~/werf-guide/app

# To see what changes we will make later in this chapter, let's replace all the application files
# in the repository with new, modified files containing the changes described below.
git rm -r .
cp -rf ~/werf-guide/guides/examples/nodejs/050_s3/. .
git add .
git commit -m WIP
What changes we will make
# Enter the command below to show files we are going to change.
git show --stat
# Enter the command below to show the changes that will be made.
git show

Doesn’t work? Try the instructions on the “I am just starting from this chapter” tab above.

Prepare a new repository with the application:

Run the following commands in PowerShell:

# Clone the example repository to ~/werf-guide/guides (if you have not cloned it yet).
if (-not (Test-Path ~/werf-guide/guides)) {
  git clone https://github.com/werf/website $env:HOMEPATH/werf-guide/guides
}

# Copy the (unchanged) application files to ~/werf-guide/app.
rm -Recurse -Force ~/werf-guide/app
cp -Recurse -Force ~/werf-guide/guides/examples/nodejs/040_db ~/werf-guide/app

# Make the ~/werf-guide/app directory a git repository.
cd ~/werf-guide/app
git init
git add .
git commit -m initial

# To see what changes we will make later in this chapter, let's replace all the application files
# in the repository with new, modified files containing the changes described below.
git rm -r .
cp -Recurse -Force ~/werf-guide/guides/examples/nodejs/050_s3/* .
git add .
git commit -m WIP
What changes we will make
# Enter the command below to show the files we are going to change.
git show --stat
# Enter the command below to show the changes that will be made.
git show

Run the following commands in Bash:

# Clone the example repository to ~/werf-guide/guides (if you have not cloned it yet).
test -e ~/werf-guide/guides || git clone https://github.com/werf/website ~/werf-guide/guides

# Copy the (unchanged) application files to ~/werf-guide/app.
rm -rf ~/werf-guide/app
cp -rf ~/werf-guide/guides/examples/nodejs/040_db ~/werf-guide/app

# Make the ~/werf-guide/app directory a git repository.
cd ~/werf-guide/app
git init
git add .
git commit -m initial

# To see what changes we will make later in this chapter, let's replace all the application files
# in the repository with new, modified files containing the changes described below.
git rm -r .
cp -rf ~/werf-guide/guides/examples/nodejs/050_s3/. .
git add .
git commit -m WIP
What changes we will make
# Enter the command below to show files we are going to change.
git show --stat
# Enter the command below to show the changes that will be made.
git show

Storing files

Containers running in Kubernetes are often created and deleted automatically, e.g., because of Deployment updates. This means that application files cannot be stored on the container file system since those files will be:

  • available to a single container/application replica rather than all of them;
  • deleted when the container is killed.

Thus, it makes sense to store in the containers only the data you can afford to lose.

By the way, you can set the container file system to read-only. This will improve security and prevent the application from storing data locally.

But what if you need to keep some data? You can use standalone databases for that. For example, NoSQL databases such as object storages are often used for storing regular files. Object stores that provide an Amazon S3-compatible API are especially popular.

Below, we will show you how to store files in S3-compatible storage instead of a local file system. This way, your application can remain stateless and avoid some issues when working in Kubernetes.

Preliminary steps

Let’s add the following libraries:

npm i @aws-sdk/client-s3 express-fileupload readable-web-to-node-stream
  • @aws-sdk/client-s3 — for working with an AWS S3-compatible storage;
  • express-fileupload — for getting a file in the express request;
  • readable-web-to-node-stream — for converting web stream to node stream (this one comes in handy for sending files to the client).

We will use the @aws-sdk/client-s3 since the MinIO interface is compatible with AWS S3. In addition, it makes it easier to migrate to AWS S3 if necessary.

Adding /upload and /download endpoints to the application

Let’s add two new endpoints, /upload (to upload a file to the S3-compatible object storage) and /download (to download a file from the S3-compatible object storage), to see how uploading and downloading works.

Let’s add a new controller:

//@ts-check
const express = require('express');
const router = express.Router();
const asyncHandler = require('express-async-handler');
const config = require('../config/minio.json');
const fileUpload = require('express-fileupload');
const {
  S3,
  GetObjectCommand,
  PutObjectCommand,
} = require('@aws-sdk/client-s3');
const { Readable } = require('stream');
const { ReadableWebToNodeStream } = require('readable-web-to-node-stream');

module.exports = (logger) => {
  const s3 = new S3({ ...config, logger });
  const key = 'thekey';

  router.get(
    '/download',
    asyncHandler(async (req, res) => {
      try {
        const cmd = new GetObjectCommand({
          Bucket: config.bucket,
          Key: key,
        });

        const file = await s3.send(cmd);
        const body = file.Body;
        if (!body) {
          throw new Error('absent object body');
        }

        if (body instanceof Readable) {
          body.pipe(res);
          return;
        }

        if (body instanceof ReadableStream) {
          new Readable(new ReadableWebToNodeStream(body)).pipe(res);
          return;
        }

        if (body instanceof Blob) {
          body.stream().pipe(res);
          return;
        }

        throw new Error('cannot handle S3 response body');
      } catch (e) {
        if (e.name === 'NoSuchKey') {
          res.status(404).send(`You haven't uploaded anything yet.\n`);
          return;
        }
        res.status(500).send(`Something went wrong: ${e.message}\n`);
      }
    })
  );

  router.post(
    '/upload',
    fileUpload(),
    asyncHandler(async (req, res) => {
      try {
        if (!req.files || !req.files.file) {
          res.status(400).send('You forgot to attach a file.\n');
          return;
        }

        let file = req.files.file;
        if (Array.isArray(file)) {
          file = file[0];
        }

        const cmd = new PutObjectCommand({
          Bucket: config.bucket,
          Key: key,
          //@ts-ignore
          Body: file.data,
        });
        await s3.send(cmd);
        res.status(200).send('File uploaded.\n');
      } catch (e) {
        res.status(500).send(`Something went wrong: ${e.message}\n`);
        return;
      }
    })
  );

  return router;
};
//@ts-check const express = require('express'); const router = express.Router(); const asyncHandler = require('express-async-handler'); const config = require('../config/minio.json'); const fileUpload = require('express-fileupload'); const { S3, GetObjectCommand, PutObjectCommand, } = require('@aws-sdk/client-s3'); const { Readable } = require('stream'); const { ReadableWebToNodeStream } = require('readable-web-to-node-stream'); module.exports = (logger) => { const s3 = new S3({ ...config, logger }); const key = 'thekey'; router.get( '/download', asyncHandler(async (req, res) => { try { const cmd = new GetObjectCommand({ Bucket: config.bucket, Key: key, }); const file = await s3.send(cmd); const body = file.Body; if (!body) { throw new Error('absent object body'); } if (body instanceof Readable) { body.pipe(res); return; } if (body instanceof ReadableStream) { new Readable(new ReadableWebToNodeStream(body)).pipe(res); return; } if (body instanceof Blob) { body.stream().pipe(res); return; } throw new Error('cannot handle S3 response body'); } catch (e) { if (e.name === 'NoSuchKey') { res.status(404).send(`You haven't uploaded anything yet.\n`); return; } res.status(500).send(`Something went wrong: ${e.message}\n`); } }) ); router.post( '/upload', fileUpload(), asyncHandler(async (req, res) => { try { if (!req.files || !req.files.file) { res.status(400).send('You forgot to attach a file.\n'); return; } let file = req.files.file; if (Array.isArray(file)) { file = file[0]; } const cmd = new PutObjectCommand({ Bucket: config.bucket, Key: key, //@ts-ignore Body: file.data, }); await s3.send(cmd); res.status(200).send('File uploaded.\n'); } catch (e) { res.status(500).send(`Something went wrong: ${e.message}\n`); return; } }) ); return router; };

Now it is time to add new paths to the routes:

...
const filesRouter = require('./routes/files');
...
app.use('/', filesRouter(logger));
... const filesRouter = require('./routes/files'); ... app.use('/', filesRouter(logger));

New endpoints, /upload and /download, have been added. Next, we need to configure them to work with the storage.

Deploying and configuring MinIO

For illustrative purposes, we will use the MinIO S3-compatible object storage. However, you can use any other S3-compatible storage (such as Amazon S3) instead.

If the different type of S3 storage is used, you do not need to create MinIO’s StatefulSet and Job as described below. However, all further steps remain unchanged.

Let’s add a StatefulSet for MinIO:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: minio
spec:
  serviceName: minio
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      containers:
      - name: minio
        image: minio/minio
        args: ["server", "/data", "--console-address", ":9001"]
        ports:
        - containerPort: 9000
          name: minio
        - containerPort: 9001
          name: console
        volumeMounts:
        - name: minio-data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: minio-data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 100Mi

---
apiVersion: v1
kind: Service
metadata:
  name: minio
spec:
  selector:
    app: minio
  ports:
  - port: 9000
    name: minio
  - port: 9001
    name: console
apiVersion: apps/v1 kind: StatefulSet metadata: name: minio spec: serviceName: minio selector: matchLabels: app: minio template: metadata: labels: app: minio spec: containers: - name: minio image: minio/minio args: ["server", "/data", "--console-address", ":9001"] ports: - containerPort: 9000 name: minio - containerPort: 9001 name: console volumeMounts: - name: minio-data mountPath: /data volumeClaimTemplates: - metadata: name: minio-data spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 100Mi --- apiVersion: v1 kind: Service metadata: name: minio spec: selector: app: minio ports: - port: 9000 name: minio - port: 9001 name: console

The next step is to create a Job for setting up MinIO:

apiVersion: batch/v1
kind: Job
metadata:
  name: "setup-minio-rev{{ .Release.Revision }}"
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: setup-minio
        image: minio/mc
        command:
        - sh
        - -euc
        - |
          is_minio_available() {
            tries=$1
            i=0
            while [ $i -lt $tries ]; do
              curl -sSL http://minio:9000/minio/health/live || return 1
              i=$((i+1))
              sleep 1
            done
          }

          # Waiting for MinIO to become available.
          until is_minio_available 10; do
            sleep 1
          done

          # Configuring access to the MinIO instance.
          mc alias set minio http://minio:9000 minioadmin minioadmin

          # Creating a bucket for the application.
          mc mb --ignore-existing minio/werf-guide-app
apiVersion: batch/v1 kind: Job metadata: name: "setup-minio-rev{{ .Release.Revision }}" spec: backoffLimit: 0 template: spec: restartPolicy: Never containers: - name: setup-minio image: minio/mc command: - sh - -euc - | is_minio_available() { tries=$1 i=0 while [ $i -lt $tries ]; do curl -sSL http://minio:9000/minio/health/live || return 1 i=$((i+1)) sleep 1 done } # Waiting for MinIO to become available. until is_minio_available 10; do sleep 1 done # Configuring access to the MinIO instance. mc alias set minio http://minio:9000 minioadmin minioadmin # Creating a bucket for the application. mc mb --ignore-existing minio/werf-guide-app

Add the configuration for connecting to MinIO: For now, the secrets needed to connect are hardcoded in a file in the project repository. In future articles, we will show you how to get rid of hardcoded secrets.

{
  "region": "us-east-1",
  "endpoint": {
    "protocol": "http",
    "hostname": "minio",
    "port": 9000,
    "path": "/"
  },
  "bucket": "werf-guide-app",
  "forcePathStyle": true,
  "credentials": {
    "accessKeyId": "minioadmin",
    "secretAccessKey": "minioadmin"
  }
}
{ "region": "us-east-1", "endpoint": { "protocol": "http", "hostname": "minio", "port": 9000, "path": "/" }, "bucket": "werf-guide-app", "forcePathStyle": true, "credentials": { "accessKeyId": "minioadmin", "secretAccessKey": "minioadmin" } }

MinIO is now ready to be deployed while our application is configured to store files with it.

Testing the storage

First, let’s deploy our application:

werf converge --repo <DOCKER HUB USERNAME>/werf-guide-app

You should see the following output:


┌ ⛵ image backend
│ ┌ Building stage backend/dockerfile
│ │ backend/dockerfile  Sending build context to Docker daemon  519.2kB
│ │ backend/dockerfile  Step 1/27 : FROM node:12-alpine as builder
    ...
│ │ backend/dockerfile  Successfully built 1483efb79d3e
│ │ backend/dockerfile  Successfully tagged 80de51be-533b-4360-a0a1-9649b8e5f340:latest
│ │ ┌ Store stage into <DOCKER HUB USERNAME>/werf-guide-app
│ │ └ Store stage into <DOCKER HUB USERNAME>/werf-guide-app (15.81 seconds)
│ ├ Info
│ │      name: <DOCKER HUB USERNAME>/werf-guide-app:89aa5375d6575fcbcba7848469df9b7cb1cd5b455c78c403a9922285-1637083461218
│ │        id: 1483efb79d3e
│ │   created: 2022-11-16 20:24:21 +0000 UTC
│ │      size: 52.1 MiB
│ └ Building stage backend/dockerfile (21.74 seconds)
└ ⛵ image backend (29.16 seconds)

┌ ⛵ image frontend
│ ┌ Building stage frontend/dockerfile
│ │ frontend/dockerfile  Sending build context to Docker daemon  519.2kB
│ │ frontend/dockerfile  Step 1/31 : FROM node:12-alpine as builder
    ...
│ │ frontend/dockerfile  Successfully built 479b3ed46505
│ │ frontend/dockerfile  Successfully tagged fdeee494-5aad-4d03-93bd-26884b2b7acf:latest
│ │ ┌ Store stage into <DOCKER HUB USERNAME>/werf-guide-app
│ │ └ Store stage into <DOCKER HUB USERNAME>/werf-guide-app (12.13 seconds)
│ ├ Info
│ │      name: <DOCKER HUB USERNAME>/werf-guide-app:931432640c233a0b42a2fed6311c9d2e7837792b207ce08516fe7384-1637083461046
│ │        id: 479b3ed46505
│ │   created: 2022-11-16 20:24:20 +0000 UTC
│ │      size: 9.4 MiB
│ └ Building stage frontend/dockerfile (17.97 seconds)
└ ⛵ image frontend (25.02 seconds)

┌ Waiting for release resources to become ready
│ ┌ Status progress
│ │ DEPLOYMENT                                                                    REPLICAS      AVAILABLE       UP-TO-DATE
│ │ werf-guide-app                                                                2/1           1               1
│ │ │   POD                          READY      RESTARTS      STATUS              ---
│ │ ├── guide-app-66b78c6dd8-49mxt   0/2        0             Init:0/1            Waiting for: replicas 2->1                          ↵
│ │
│ │ └── guide-app-85c5c7c485-vkv28   2/2        0             Running
│ │ STATEFULSET                                                                   REPLICAS      READY           UP-TO-DATE
│ │ minio                                                                         1/1           0               1
│ │ │   POD                          READY      RESTARTS      STATUS              ---
│ │ └── 0                            0/1        0             ContainerCreating   Waiting for: ready 0->1                             ↵
│ │
│ │ mysql                                                                         1/1           1               1
│ │ JOB                                                                           ACTIVE        DURATION        SUCCEEDED/FAILED
│ │ setup-and-migrate-db-rev19                                                    1             2s              0/0                   ↵
│ │
│ │ │   POD                          READY      RESTARTS      STATUS              ---
│ │ └── and-migrate-db-rev19--1-cw6l 0/1        0             ContainerCreating   Waiting for: pods should be terminated, succeeded 0->1
│ │     l
│ │ setup-minio-rev19                                                             1             2s              0/0
│ │ │   POD                          READY      RESTARTS      STATUS              ---
│ │ └── minio-rev19--1-xxbbv         0/1        0             ContainerCreating   Waiting for: pods should be terminated, succeeded 0->1
│ └ Status progress
│
│ ┌ job/setup-and-migrate-db-rev19 po/setup-and-migrate-db-rev19--1-cw6ll container/setup-and-migrate-db logs
│ │ mysqladmin: connect to server at 'mysql' failed
│ │ error: 'Access denied for user 'root'@'172.17.0.1' (using password: YES)'
│ │ mysqladmin: connect to server at 'mysql' failed
│ │ error: 'Access denied for user 'root'@'172.17.0.1' (using password: YES)'
│ │ mysqladmin: connect to server at 'mysql' failed
│ │ error: 'Access denied for user 'root'@'172.17.0.1' (using password: YES)'
│ │ mysqladmin: connect to server at 'mysql' failed
│ │ error: 'Access denied for user 'root'@'172.17.0.1' (using password: YES)'
│ │ mysqladmin: connect to server at 'mysql' failed
│ │ error: 'Access denied for user 'root'@'172.17.0.1' (using password: YES)'
│ └ job/setup-and-migrate-db-rev19 po/setup-and-migrate-db-rev19--1-cw6ll container/setup-and-migrate-db logs
│
│ ┌ Status progress
│ │ DEPLOYMENT                                                                    REPLICAS      AVAILABLE       UP-TO-DATE
│ │ werf-guide-app                                                                2/1           1               1
│ │ │   POD                          READY      RESTARTS      STATUS              ---
│ │ ├── guide-app-66b78c6dd8-49mxt   0/2        0             Init:0/1            Waiting for: replicas 2->1                          ↵
│ │
│ │ └── guide-app-85c5c7c485-vkv28   2/2        0             Running
│ │ STATEFULSET                                                                   REPLICAS      READY           UP-TO-DATE
│ │ minio                                                                         1/1           0->1            1
│ │ │   POD                          READY      RESTARTS      STATUS
│ │ └── 0                            1/1        0             ContainerCreating
│ │                                                           -> Running
│ │ mysql                                                                         1/1           1               1
│ │ JOB                                                                           ACTIVE        DURATION        SUCCEEDED/FAILED
│ │ setup-and-migrate-db-rev19                                                    1             6s              0/0                   ↵
│ │
│ │ │   POD                          READY      RESTARTS      STATUS              ---
│ │ └── and-migrate-db-rev19--1-cw6l 1/1        0             ContainerCreating   Waiting for: pods should be terminated, succeeded 0->1
│ │     l                                                     -> Running
│ │ setup-minio-rev19                                                             1             2s              0/0
│ │ │   POD                          READY      RESTARTS      STATUS              ---
│ │ └── minio-rev19--1-xxbbv         0/1        0             ContainerCreating   Waiting for: pods should be terminated, succeeded 0->1
│ └ Status progress
│
│ ┌ job/setup-and-migrate-db-rev19 po/setup-and-migrate-db-rev19--1-cw6ll container/setup-and-migrate-db logs
│ │ mysqladmin: connect to server at 'mysql' failed
│ │ error: 'Access denied for user 'root'@'172.17.0.1' (using password: YES)'
│ └ job/setup-and-migrate-db-rev19 po/setup-and-migrate-db-rev19--1-cw6ll container/setup-and-migrate-db logs
│
│ ┌ job/setup-minio-rev19 po/setup-minio-rev19--1-xxbbv container/setup-minio logs
│ │ curl: (7) Failed to connect to minio port 9000: Connection refused
│ └ job/setup-minio-rev19 po/setup-minio-rev19--1-xxbbv container/setup-minio logs
│
│ ┌ job/setup-and-migrate-db-rev19 po/setup-and-migrate-db-rev19--1-cw6ll container/setup-and-migrate-db logs
│ │ mysqladmin: connect to server at 'mysql' failed
│ │ error: 'Access denied for user 'root'@'172.17.0.1' (using password: YES)'
│ │ mysqladmin: connect to server at 'mysql' failed
│ │ error: 'Access denied for user 'root'@'172.17.0.1' (using password: YES)'
│ └ job/setup-and-migrate-db-rev19 po/setup-and-migrate-db-rev19--1-cw6ll container/setup-and-migrate-db logs
│
│ ┌ deploy/werf-guide-app po/werf-guide-app-66b78c6dd8-49mxt container/wait-db-readiness logs
│ │
│ │ Sequelize CLI [Node: 12.22.7, CLI: 6.3.0, ORM: 6.9.0]
│ │
│ │ Loaded configuration file "config/database.json".
│ │ Using environment "production".
│ │ up 20211101064002-create-talker.js
│ └ deploy/werf-guide-app po/werf-guide-app-66b78c6dd8-49mxt container/wait-db-readiness logs
│
│ ┌ job/setup-and-migrate-db-rev19 po/setup-and-migrate-db-rev19--1-cw6ll container/setup-and-migrate-db logs
│ │ mysqladmin: connect to server at 'mysql' failed
│ │ error: 'Access denied for user 'root'@'172.17.0.1' (using password: YES)'
│ │ mysqladmin: connect to server at 'mysql' failed
│ │ error: 'Access denied for user 'root'@'172.17.0.1' (using password: YES)'
│ └ job/setup-and-migrate-db-rev19 po/setup-and-migrate-db-rev19--1-cw6ll container/setup-and-migrate-db logs
│
│ ┌ Status progress
│ │ DEPLOYMENT                                                                    REPLICAS      AVAILABLE       UP-TO-DATE
│ │ werf-guide-app                                                                2/1           1               1
│ │ │   POD                          READY      RESTARTS      STATUS              ---
│ │ ├── guide-app-66b78c6dd8-49mxt   0/2        0             Init:0/1 ->         Waiting for: replicas 2->1                          ↵
│ │
│ │ │                                                         PodInitializing
│ │ └── guide-app-85c5c7c485-vkv28   2/2        0             Running
│ │ STATEFULSET                                                                   REPLICAS      READY           UP-TO-DATE
│ │ minio                                                                         1/1           1               1
│ │ │   POD                          READY      RESTARTS      STATUS
│ │ └── 0                            1/1        0             Running
│ │ mysql                                                                         1/1           1               1
│ │ JOB                                                                           ACTIVE        DURATION        SUCCEEDED/FAILED
│ │ setup-and-migrate-db-rev19                                                    1             6s              0/0                   ↵
│ │
│ │ │   POD                          READY      RESTARTS      STATUS              ---
│ │ └── and-migrate-db-rev19--1-cw6l 1/1        0             Running             Waiting for: pods should be terminated, succeeded 0->1
│ │     l
│ │ setup-minio-rev19                                                             1             11s             0/0
│ │ │   POD                          READY      RESTARTS      STATUS              ---
│ │ └── minio-rev19--1-xxbbv         1/1        0             ContainerCreating   Waiting for: pods should be terminated, succeeded 0->1
│ │                                                           -> Running
│ └ Status progress
│
│ ┌ job/setup-and-migrate-db-rev19 po/setup-and-migrate-db-rev19--1-cw6ll container/setup-and-migrate-db logs
│ │
│ │ Sequelize CLI [Node: 12.22.7, CLI: 6.3.0, ORM: 6.9.0]
│ │
│ │ Loaded configuration file "config/database.json".
│ │ Using environment "production".
│ │ No migrations were executed, database schema was already up to date.
│ └ job/setup-and-migrate-db-rev19 po/setup-and-migrate-db-rev19--1-cw6ll container/setup-and-migrate-db logs
│
│ ┌ Status progress
│ │ DEPLOYMENT                                                                    REPLICAS      AVAILABLE       UP-TO-DATE
│ │ werf-guide-app                                                                2->1/1        1               1
│ │ │   POD                          READY      RESTARTS      STATUS
│ │ ├── guide-app-66b78c6dd8-49mxt   2/2        0             PodInitializing ->
│ │ │                                                         Running
│ │ └── guide-app-85c5c7c485-vkv28   2/2        0             Running ->
│ │                                                           Terminating
│ │ STATEFULSET                                                                   REPLICAS      READY           UP-TO-DATE
│ │ minio                                                                         1/1           1               1
│ │ │   POD                          READY      RESTARTS      STATUS
│ │ └── 0                            1/1        0             Running
│ │ mysql                                                                         1/1           1               1
│ │ JOB                                                                           ACTIVE        DURATION        SUCCEEDED/FAILED
│ │ setup-and-migrate-db-rev19                                                    0             17s             0->1/0                ↵
│ │
│ │ │   POD                          READY      RESTARTS      STATUS
│ │ └── and-migrate-db-rev19--1-cw6l 0/1        0             Running ->
│ │     l                                                     Completed
│ │ setup-minio-rev19                                                             1             11s             0/0
│ │ │   POD                          READY      RESTARTS      STATUS              ---
│ │ └── minio-rev19--1-xxbbv         1/1        0             Running             Waiting for: pods should be terminated, succeeded 0->1
│ └ Status progress
│
│ ┌ job/setup-minio-rev19 po/setup-minio-rev19--1-xxbbv container/setup-minio logs
│ │ Added `minio` successfully.
│ │ Bucket created successfully `minio/werf-guide-app`.
│ └ job/setup-minio-rev19 po/setup-minio-rev19--1-xxbbv container/setup-minio logs
│
│ ┌ Status progress
│ │ DEPLOYMENT                                                                    REPLICAS      AVAILABLE       UP-TO-DATE
│ │ werf-guide-app                                                                1/1           1               1
│ │ │   POD                          READY      RESTARTS      STATUS
│ │ ├── guide-app-66b78c6dd8-49mxt   2/2        0             Running
│ │ └── guide-app-85c5c7c485-vkv28   2/2        0             Terminating
│ │ STATEFULSET                                                                   REPLICAS      READY           UP-TO-DATE
│ │ minio                                                                         1/1           1               1
│ │ │   POD                          READY      RESTARTS      STATUS
│ │ └── 0                            1/1        0             Running
│ │ mysql                                                                         1/1           1               1
│ │ JOB                                                                           ACTIVE        DURATION        SUCCEEDED/FAILED
│ │ setup-and-migrate-db-rev19                                                    0             17s             1/0                   ↵
│ │
│ │ │   POD                          READY      RESTARTS      STATUS
│ │ └── and-migrate-db-rev19--1-cw6l 0/1        0             Completed
│ │     l
│ │ setup-minio-rev19                                                             0             23s             0->1/0
│ │ │   POD                          READY      RESTARTS      STATUS
│ │ └── minio-rev19--1-xxbbv         0/1        0             Running ->
│ │                                                           Completed
│ └ Status progress
└ Waiting for release resources to become ready (22.83 seconds)

Release "werf-guide-app" has been upgraded. Happy Helming!
NAME: werf-guide-app
LAST DEPLOYED: Tue Nov 16 20:24:43 2022
NAMESPACE: werf-guide-app
STATUS: deployed
REVISION: 19
TEST SUITE: None
Running time 58.21 seconds

Now let’s access the /download endpoint to get the file from S3:

curl http://werf-guide-app.test/download

Since we haven’t uploaded any files yet, we’ll get this:

You haven't uploaded anything yet.

Now let’s create a new file and upload it to the S3 storage:

echo "This is file content." > file.txt
curl -F "file=@file.txt" http://werf-guide-app.test/upload
"This is file content." | Out-File -Encoding ascii -FilePath file.txt
curl.exe -F "file=@file.txt" http://werf-guide-app.test/upload

The expected output indicates that the file has been successfully added to the storage:

File uploaded.

Let’s try to pull the file from the storage again:

curl http://werf-guide-app.test/download

You should see the file content:

This is file content.

Let’s make sure that the file gets saved directly to the storage and pulled from it. To do this, run the container having the mc tool to interact with MinIO:

kubectl run mc --image=minio/mc --rm -it --command -- bash

Now, execute the following commands in the container’s shell:

# Connect to MinIO
mc alias set minio http://minio:9000 minioadmin minioadmin
# Get the content of file stored in S3
mc cat "minio/werf-guide-app/$(mc ls minio/werf-guide-app | awk 'NR==1 {print $5}')"

The expected outcome:

This is file content.

In this chapter, we learned how to store files in the S3-compatible object storage instead of the container file system. With this approach, the application Pods can be created and deleted without issues: files are stored safely, and any application replica can quickly access them. At the same time, the file system on Kubernetes nodes will not be filled with unnecessary files.

Keep in mind that you can only store data you can afford to lose in a container. All other data must be stored in the appropriate database/storage. This approach is endorsed by many, e.g. you can find it in the best practices by Google Cloud engineers.

prev
next