You will need the following components to develop in Kubernetes:

  • the locally installed werf tool — done in the previous chapters;
  • the application code stored in Git — done in the previous chapters;
  • a Kubernetes cluster;
  • a container registry;
  • the appropriate DNS records;

Setting up an environment is a difficult task. We do not recommend digging into these problems yourselves. It makes more sense to use the existing services available on the market, seek help from colleagues who have the relevant experience, or ask questions in the werf’s Telegram community.

A brief checklist for self-checking the environment
  • You have a Kubernetes cluster (version 1.14 or later).
    • There is an ingress controller in the cluster with the respective domain (example.com) linking to it (in other words, you can visit it in your browser).
  • You have a registry.
    • The registry is available at registry.example.com.
    • The cluster can pull images from your registry.
  • On the local computer:
    • there is access to the cluster (kubectl version shows K8s version);
    • there is access to the registry (docker push works fine);
    • the browser can access example.com (even if it shows the 404 error page by Ingress).

Now select your preferred method/location for setting up an environment:

Minikube

Minikube is a lightweight Kubernetes version that can run on a local machine.

Installation

Install minikube and run it:

minikube start --driver=docker

IMPORTANT: You need to make sure that Minikube uses a docker driver if it is already running on your system. Otherwise, you have to restart Minikube using the minikube delete command and start it using the command provided above.

The .kube/config file containing keys for connecting to the local cluster will be automatically created, allowing werf to connect to the local registry.

As a result, you must be able to access the cluster using the kubectl utility (you may need to install this utility separately). For example, the command below:

kubectl get ns

… will print a list of all namespaces in the cluster (and not an error message).

Ingress

You need to enable the appropriate addon in Minikube.

How do I verify that Ingress operates as expected?

You can assume that everything works fine by default. If you have difficulties in understanding the points below, please, return to this sub-chapter later.

  • The load balancer is installed.
  • The pod containing the load balancer is up and running fine (use this command for checking: kubectl -n ingress-nginx get po).
  • The application is listening on port 80 (check it with lsof -n grep LISTEN).

Registry

Enable the minikube registry addon:

minikube addons enable registry

And then perform the following actions depending on your operating system:

Windows

The command output will contain a string like the one below:

...
* Registry addon on with docker uses 32769 please use that instead of default 5000
...

Note the port number (e.g., 32769)

Run the following port-forwarding command in a separate terminal, replacing 32769 with your port number:

docker run -ti --rm --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:host.docker.internal:32769"

Start the service at port 5000:

kubectl -n kube-system expose rc/registry --type=ClusterIP --port=5000 --target-port=5000 --name=werf-registry --selector=actual-registry=true

Run the following port forwarding command in the terminal:

kubectl port-forward --namespace kube-system service/werf-registry 5000
macOS

The command output will contain a string like this:

...
* Registry addon on with docker uses 32769 please use that instead of default 5000
...

Note the port number (e.g., 32769).

Run the following port-forwarding command in a separate terminal, replacing 32769 with the number of your port:

docker run -ti --rm --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:host.docker.internal:32769"

Run the following port-forwarding command in a separate terminal, replacing 32769 with the number of your port:

brew install socat
socat TCP-LISTEN:5000,reuseaddr,fork TCP:host.docker.internal:32769
Linux

Run the following port forwarding command in the terminal:

sudo apt-get install -y socat
socat -d -d TCP-LISTEN:5000,reuseaddr,fork TCP:$(minikube ip):5000

Hosts

This tutorial assumes that the cluster (in fact, its Nginx Ingress controller) is located at example.com, and its registry is accessible at registry.example.com. We will be using this domain and its subdomains in the configs. In the case of using different addresses, adjust the configuration accordingly.

Enter the following lines in /etc/hosts:

127.0.0.1           example.com
127.0.0.1           registry.example.com

Authorizing in the registry

The Minikube registry does not require authorization. No further action is required.

Docker Desktop

Installing

Install Docker Desktop for Windows or macOS.

Start Kubernetes and configure the resources allocated to it.

Why are resources important?

Lack of resources in the cluster may prevent your application, Ingress, or even some system components required by the orchestrator from starting. You must be really experienced in cluster administration to figure out what is happening.

The authors of this tutorial tested it using the following configuration: 6 CPU cores, 6 GB RAM, 1.5 GB of swap space, 24 GB of disk space.

The .kube/config file containing keys for connecting to the local cluster will be automatically created, allowing werf to connect to the local registry.

As a result, you must be able to access the cluster using the kubectl utility (you may need to install it separately). For example, the command below:

kubectl get ns

… will print the list of all namespaces in the cluster (and not an error message).

Ingress

You have to manually install the Nginx Ingress controller.

In the case of Docker Desktop, the Ingress controller may be inaccessible in some cases. That is because ports are not forwarded to the host machine. To make sure that Ingress works as intended, check that:

  • The pod containing the Ingress controller is up and running fine (use this command for checking in the case of nginx-ingress: kubectl -n ingress-nginx get po).
  • Kubernetes is listening on port 80 (check it with lsof -n | grep LISTEN or similar way)

If there is no Kubernetes on port 80 (HTTP), you may need to forward the port manually. For this, find out the name of the Ingress controller’s pod:

kubectl -n ingress-nginx get po

… and then run port forwarding:

kubectl port-forward --address 0.0.0.0 -n ingress-nginx ingress-nginx-controller-<random_alphanumeric_sequence> 80:80

After running the above commands, verify that an application is listening on port 80 (lsof -n | grep LISTEN).

Registry

Docker Desktop does not have a built-in registry. The easiest way is to run it manually as a Docker image:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

Note that in this case, the registry does not use SSL encryption by default. Thus, you have to add the --insecure-registry=true parameter when using werf.

Hosts

This tutorial assumes that the cluster (in fact, its Nginx Ingress controller) is located at example.com, and its registry is accessible at registry.example.com. We will be using this domain and its subdomains in the configs. If you are using different addresses, adjust the configuration accordingly.

Insert the following lines in /etc/hosts:

127.0.0.1           example.com
127.0.0.1           registry.example.com

Authorizing in the registry

When running the registry in a local mode (as shown in the example above), no password is necessary, and you do not need to do anything.

I use an external registry or username/password

You need to log in to the local machine using the docker login command for werf to be able to push assembled images to the registry:

docker login <registry_domain> -u <account_login> -p <account_password>

Choosing a cloud provider

There is many Managed Kubernetes offerings. Some of them include server capacity, while others don’t.

The easiest way is to take advantage of AWS (EKS) or Google (GKE) offerings. During the initial registration, they provide a bonus large enough for working with a cluster for a couple of weeks. However, you will need to enter your bank card details to register.

Yandex.Cloud is a viable alternative: the Managed Kubernetes service is available here as well, and you also must provide your card details. However, you can use some variation of a local bank card (such as the YouMoney card that allows for subscription payments within Russia).

You can also try to deploy Kubernetes yourself to the Hetzner cloud using their tutorial (Hetzner is one of the cheapest cloud providers out there). However, you must realize that you will have to deal with a large volume of platform administration tasks. You must find a reliable provider or support team for your production environment. If you have never installed Kubernetes yourself and/or do not have experience in system administration, then you should not try to master this rather voluminous topic as part of this tutorial.

Regardless of the cloud provider chosen, you will need to get the keys for accessing the Kubernetes cluster in the form of .kube/config file.

What is .kube/config?

This file stores the cluster access details. Using this file (located at ~/.kube/config), tools that work with Kubernetes can connect to the cluster.

Here is the example of that file:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ALotOfNumbersAndLettersAndSoOnAveryVERYveryLongStringInBase64=
    server: https://127.0.0.1:6445
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: ManyNumbersAndLettersAVERYveryVerylongString=
    client-key-data: ManyLettersAndNumbersAveryVeryVERYlongString=
apiVersion: v1 clusters: - cluster: certificate-authority-data: ALotOfNumbersAndLettersAndSoOnAveryVERYveryLongStringInBase64= server: https://127.0.0.1:6445 name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: ManyNumbersAndLettersAVERYveryVerylongString= client-key-data: ManyLettersAndNumbersAveryVeryVERYlongString=

Note that in addition to access keys, there is also a path to the Kubernetes API (e.g., server: https://127.0.0.1:6445). Important: this path must be correct! Some .kube/config-generating utilities do not insert a public IP into the file. In this case, you need to adjust it manually.

Checking the operability and access to the cluster

Check your cluster using the checklist provided at the beginning of the article.

As a result, you must be able to access the cluster using the kubectl utility (you may need to install this utility separately). For example, the command below:

kubectl get ns

… will print a list of all namespaces in the cluster (and not an error message).

Ingress

You also have to install Nginx Ingress in your cluster. There are detailed instructions available for all major providers on the NGINX Ingress Controller’s website.

Registry

In most cases, cloud providers provide a registry as one of the services (however, this is not always the case).

My provider doesn't provide a registry, what should I do?

You can deploy Docker Registry by yourself on a separate virtual machine or use some cloud-based solution. werf supports various implementations of container registries.

Ensure the network connectivity between the cluster and the registry: again, in most cases, the cloud provider solves this problem on its own or provides specific manuals on the subject.

Hosts

This tutorial assumes that the cluster (in fact, its Nginx Ingress controller) is located at example.com, and its registry is accessible at registry.example.com. You will have to choose and configure the necessary DNS records yourself and substitute your domains in the code throughout this tutorial.

Authorizing in the registry

You need to log in on the local machine using the docker login command so that werf can push assembled images to the registry:

docker login registry.example.com -u <account_login> -p <account_password>

where <account_login> is the login to the registry, and <account_password> is the password.

I already have a cluster

If you already have a cluster, then you probably have some experience in configuring it. Check your cluster using the checklist provided at the beginning of the article.

Regardless of how the cluster is installed, you will need to get the keys for accessing the Kubernetes cluster in the form of .kube/config file.

What is .kube/config?

This file stores the cluster access details. Using this file (located at ~/.kube/config), tools that work with Kubernetes can connect to the cluster.

Here is the example:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ALotOfNumbersAndLettersAndSoOnAveryVERYveryLongStringInBase64=
    server: https://127.0.0.1:6445
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: ManyNumbersAndLettersAVERYveryVerylongString=
    client-key-data: ManyLettersAndNumbersAveryVeryVERYlongString=
apiVersion: v1 clusters: - cluster: certificate-authority-data: ALotOfNumbersAndLettersAndSoOnAveryVERYveryLongStringInBase64= server: https://127.0.0.1:6445 name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: ManyNumbersAndLettersAVERYveryVerylongString= client-key-data: ManyLettersAndNumbersAveryVeryVERYlongString=

Note that in addition to connection keys, there is also a path to the Kubernetes API (e.g., server: https://127.0.0.1:6445). Important: this path must be correct! Some .kube/config-generating utilities do not insert a public IP into the file. In this case, you need to adjust it manually.

Checking the operability and accessibility of the cluster

Check your cluster using the checklist provided at the beginning of the article.

As a result, you must be able to access the cluster using the kubectl utility (you may need to install it separately). For example, the command below:

kubectl get ns

… must print a list of all namespaces in the cluster (and not an error message).

Ingress

Nginx Ingress must be installed in your cluster. The installation guide is available in the Kubernetes documentation.

Registry

werf supports various registry implementations.

Make sure that there is network connectivity between the cluster and the registry.

Hosts

This tutorial assumes that the cluster (in fact, its Nginx Ingress controller) is located at example.com, and its registry is accessible at registry.example.com. You will have to choose and configure the necessary DNS records yourself and substitute your domains in the code throughout this tutorial.

Authorizing in the registry

You need to log in to the local machine using the docker login command so that werf can push assembled images to the registry:

docker login <registry_domain> -u <account_login> -p <account_password>

Final checks

After verifying the cluster functionality, you have to check if the registry and ingress work well. To do this, run the following commands:

docker tag ubuntu:18.04 registry.example.com/ubuntu:18.04
docker push registry.example.com/ubuntu:18.04

They should push the Ubuntu image to the registry without any errors. And the command below:

curl example.com

… should return a nginx ingress error page (if you haven’t already deployed an application to the cluster).