Install

The Kubernetes docs may make you feel lost. We will start from https://kubernetes.io/docs/setup/production-environment/#production-control-plane

We choose to use the default tool kubeadm to deploy a control plane used to manage the overall state of the cluster. Take a look at https://kubernetes.io/docs/concepts/overview/components/ to have an overview of Kubernetes components.

Thus, we will follow the kubeadm install guide from https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

Prerequisites

The first step is to install a container runtime. A container runtime allows the run of containers in a fisical or virtual host. Kubernetes is not a container runtime but can use one of them to orchestrate containers in a cluster. The most famous of them is Docker, but you can use any one that follow the Container Runtime Interface, there is a list of suggestion and we will use the first one.

from from https://kubernetes.io/docs/setup/production-environment/container-runtimes/

The docs exaplain how to enable IPv4 packey forward, we can do that configuring the follow

### configure the IPv4 packet forward to be enabled at boot time
cat > /etc/sysctl.d/k8s.conf << 'EOF'
net.ipv4.ip_forward = 1
EOF
### enable the IPv4 packet forward right now without reboot
sysctl -w net.ipv4.ip_forward=1

Then, we will install Containerd as our container runtime

Install base packages

apt-get -q -y install curl ethtool iproute2 socat conntrack kmod
apt-get -q -y clean

Install Containerd

The docs tell to follow the official docs from https://github.com/containerd/containerd/blob/main/docs/getting-started.md to install Containerd

### define current arch
ARCH="amd64"
# or ARCH="arm64"
### install containerd
VERSION="2.1.3"
curl -fsSL "https://github.com/containerd/containerd/releases/download/v${VERSION}/containerd-${VERSION}-linux-${ARCH}.tar.gz" | tar -C /usr/local/ -xz
### configure containerd to be started with systemd
mkdir -p /usr/local/lib/systemd/system/
curl -fsSL -o /usr/local/lib/systemd/system/containerd.service "https://raw.githubusercontent.com/containerd/containerd/main/containerd.service"
systemctl daemon-reload
systemctl enable containerd
### install runc
VERSION="1.3.0"
curl -fsSL -o /usr/local/sbin/runc "https://github.com/opencontainers/runc/releases/download/v${VERSION}/runc.${ARCH}"
chmod 0755 /usr/local/sbin/runc
### install cni plugins
VERSION="1.7.1"
mkdir -p /opt/cni/bin
curl -fsSL "https://github.com/containernetworking/plugins/releases/download/v${VERSION}/cni-plugins-linux-${ARCH}-v${VERSION}.tgz" | tar -C /opt/cni/bin -xz

Get default config

mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

Then, going back to https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd-systemd to configure Containerd to use systemd as cgroup driver and start it

sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
systemctl start containerd
systemctl status -n 100 -o cat containerd

Install Kubeadm

Then, going back to https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

The docs tell us to install cni-plugins but we already installed them following the Containerd docs. Then, we will install the optional crictl tool as well as kubeadm, kubelet and kubelet service install:

### define current arch
ARCH="amd64"
# or ARCH="arm64"
cd /usr/local/bin
### install crictl
CRICTL_VERSION="v1.33.0"
curl -fsSL "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gz" | tar -xz
chmod 0755 crictl
chown root:root crictl
### install kubeadm and kubelet
KUBE_VERSION="v1.33.2"
curl -fsSL --remote-name-all "https://dl.k8s.io/release/${KUBE_VERSION}/bin/linux/${ARCH}/{kubeadm,kubelet}"
chmod 0755 {kubeadm,kubelet}
chown root:root {kubeadm,kubelet}
### install kubernetes service and confs
RELEASE_VERSION="v0.16.2"
curl -fsSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/krel/templates/latest/kubelet/kubelet.service" | sed "s:/usr/bin:/usr/local/bin:g" > /usr/local/lib/systemd/system/kubelet.service
mkdir -p /usr/local/lib/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/krel/templates/latest/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:/usr/local/bin:g" > /usr/local/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
### start kubelet in a crashloop waiting for our manual kubeadm init cluster
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status -n 100 -o cat kubelet

Note from the doc: "The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do."

Install Kubectl

Finally, we will install kubectl tool following the docs from https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

Note, I don't understand why the docs don't put it togheter with kubeadm and kubelet

### define current arch
ARCH="amd64"
# or ARCH="arm64"
cd /usr/local/bin
### install kubectl
KUBE_VERSION="v1.33.2"
curl -fsSL --remote-name-all "https://dl.k8s.io/release/${KUBE_VERSION}/bin/linux/${ARCH}/kubectl"
chmod 0755 kubectl
chown root:root kubectl

Cluster setup

Now, if you want to have an High Availability (HA) kubernetes cluster, apparently you need an external load balancer. See https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/ .

So, choose your setup to keep going:

NON HA cluster (no load balancer required)

# enable kubelet service but not start it, kubeadm init will create the initial config and start itself
# note the use of --ignore-preflight-errors=NumCPU,Mem option when machines have less than 2 cores or 1.7Gb Ram
systemctl enable kubelet.service
kubeadm init --ignore-preflight-errors=NumCPU,Mem --cri-socket unix:/run/containerd/containerd.sock --pod-network-cidr=192.168.0.0/16
...
# PLEASE TAKE NOTE OF THE OUTPUT COMMAND TO JOIN MORE WORKERS TO THE CLUSTER

HA cluster (load balancer REQUIRED)

please note, a loadbalancer domain name is useful here, however at least a loadbalancer responding to an internal ip and correctly configured to forward to the kubernetes control-plane https request is required. It must be defined with the option --control-plane-endpoint and forward the requests to the control-plane nodes of the cluster.

# enable kubelet service but not start it, kubeadm init will create the initial config and start itself
systemctl enable kubelet.service
kubeadm init --ignore-preflight-errors=NumCPU,Mem --cri-socket unix:/run/containerd/containerd.sock --pod-network-cidr=192.168.0.0/16 --control-plane-endpoint "LOADBALANCER_INTERNAL_IP_OR_NAME:6443" --upload-certs
# PLEASE TAKE NOTE OF THE OUTPUT COMMAND TO JOIN MORE CONTROL PLANES TO THE CLUSTER

add a pod network

if you want to use flannel pod network:

from https://github.com/flannel-io/flannel

export KUBECONFIG=/etc/kubernetes/admin.conf
VERSION="0.22.1"
curl -fsSL "https://github.com/flannel-io/flannel/releases/download/v${VERSION}/kube-flannel.yml" | sed "s:10.244.0.0/16:192.168.0.0/16:g" | kubectl create -f -
# wait ready state
kubectl get nodes

ALTERNATIVELY, if you want to use calico pod network:

from https://docs.tigera.io/calico/latest/getting-started/kubernetes/quickstart

export KUBECONFIG=/etc/kubernetes/admin.conf
VERSION="3.26.1"
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v${VERSION}/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v${VERSION}/manifests/custom-resources.yaml
# wait ready state
kubectl get nodes

Links

add more nodes to the cluster

For each node,

again, enable (without start) the kubelet service

# enable kubelet service but not start it, kubeadm init will create the initial config and start itself
systemctl enable kubelet.service

use the command outputted by the kubeadm init to add nodes to the cluster

remember to add the --ignore-preflight-errors=NumCPU,Mem option if your machines have less than 2 cores or 1.7Gb Ram

kubeadm join --ignore-preflight-errors=NumCPU,Mem ...

NOTE, if you are configuring an HA cluster, and if you want to use the control plane nodes to also run your containers, you must allow that running the following command in the first node:

kubectl get nodes # note that the node ROLES is control-plane
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

Exposing cluster applications outside the cluster

Usually we can find docs or tutorials using the kubernates Service of type LoadBalancer to expose an app outside the cluster. If I understand it correctly, LoadBalancer service requires a direct integration with a cloud provider in a way that kubernates itself will setup a cloud loadbalancer in such provider. In case we have not (or we do not want) to use such direct integration, we can not use services of type LoadBalancer.

Usually, however, we want to use a loadbalancer to provide some level of High Availability to our applications. We can be using a homemade solution or a free cloud loadbalancer (like the one I'm using Oracle Network Load Balancer) so that we must find a solution to integrate your load balancer with your cluster.

For example, we can setup the loadbalancer to forward the requests incoming from the loadbalancer ip to all of our cluster nodes that are listening to a specific port (30080 in this example). Then we can setup a proxy application (one that implements the ingress controller specification, availables in this list), to listen directly on that port in the node were the proxy is deployed. This can be done using the hostPort:30080 directive, or we can put a kubernetes Service of type NodePort in front of it using a specific nodePort:30080. There are pro and cons to each solution (see for instance the answers on stackoverflow or reddit). In this example we will use hostPort:30080, and we will deploy the proxy application as a type DaemonSet (instead of Deployment) to have an easy way to run the a proxy instance in each node of our cluster.

Note that, if I understand correctly, in both cases we cannot use an Ingress kubernates object in front of proxy to change the routes to proxy itself, because we are skipping the Ingress and making our external loadbalancer to communicate directly with the proxy. However, we can configure an Ingress that defines a route to a Service that forward the request to the proxy again. It sound strange, however it can make sense to request that should be answered from the proxy itself (like the proxy healthcheck) but must be managed by an Ingress to do some route stuffs (e.g. to replace the Traefik path /ping, where it answers its health status, to the path /healthcheck that can be a path that make sense for us).

Using Traefik as ingress controller

from https://doc.traefik.io/traefik/v3.0/providers/kubernetes-crd/

To use traefik middlewares (useful to modify requests like replace paths or strip prefixes) we need to use the traefik kubernetescrd provider and add the traefik crd resources, so:

Traefik Resource Definitions 00-kubernetes-crd-definition-v1.yml

curl -SsL https://raw.githubusercontent.com/traefik/traefik/v3.0/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml -o 00-kubernetes-crd-definition-v1.yml
kubectl apply -f 00-kubernetes-crd-definition-v1.yml

Traefik RBAC 01-kubernetes-crd-rbac.yml

curl -SsL https://raw.githubusercontent.com/traefik/traefik/v3.0/docs/content/reference/dynamic-configuration/kubernetes-crd-rbac.yml -o 01-kubernetes-crd-rbac.yml
kubectl apply -f 01-kubernetes-crd-rbac.yml

Then, we setup traefik with the kubernetescrd provider for middlewares and kubernetesingress to use the classic Ingress object to define routes. We setup also a kubernetes Service of type ClusterIP for traefik to be able to use Ingress:

Traefik application resource 02-traefik.yml

cat > 02-traefik.yml << 'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-sa
  labels:
    stack: traefik

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: traefik-crb
  labels:
    stack: traefik
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-sa
    namespace: default

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: traefik-dashboard-ing
  labels:
    stack: traefik
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
  rules:
    - http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: traefik-srv
                port:
                  number: 8080
          - path: /dashboard
            pathType: Prefix
            backend:
              service:
                name: traefik-srv
                port:
                  number: 8080

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-srv
  labels:
    stack: traefik
spec:
  type: ClusterIP
  selector:
    stack: traefik
  ports:
    - protocol: TCP
      port: 80
      name: web
      targetPort: 80
    - protocol: TCP
      port: 443
      name: websecure
      targetPort: 443
    - protocol: TCP
      port: 8080
      name: traefik
      targetPort: 8080

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: traefik-dst
  labels:
    stack: traefik
spec:
  selector:
    matchLabels:
      stack: traefik
  template:
    metadata:
      labels:
        stack: traefik
    spec:
      serviceAccountName: traefik-sa
      containers:
        - name: traefik-cnt
          image: docker.io/traefik:v3.0
          imagePullPolicy: Always
          readinessProbe:
            httpGet:
              path: /ping
              port: 8080
            failureThreshold: 1
            initialDelaySeconds: 10
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /ping
              port: 8080
            failureThreshold: 3
            initialDelaySeconds: 10
            periodSeconds: 10
          args:
            - --log.level=INFO
            - --accesslog=true
            - --entrypoints.web.address=:80/tcp
            - --entrypoints.websecure.address=:443/tcp
            - --entrypoints.traefik.address=:8080/tcp
            - --ping=true
            - --ping.entryPoint=traefik
            - --api=true
            - --api.dashboard=true
            - --api.insecure=true
            - --providers.kubernetesingress=true
            - --providers.kubernetescrd=true
          ports:
            - name: web
              containerPort: 80
              hostPort: 30080
            - name: websecure
              containerPort: 443
              hostPort: 30443
            - name: traefik
              containerPort: 8080
EOF
kubectl apply -f 02-traefik.yml

see all

watch kubectl get all,ing,cm,secret -A -o wide

see traefik definition

kubectl describe daemonset traefik

see traefik logs

kubectl logs -f --all-containers --prefix=true -l stack=traefik

delete all

kubectl delete -f 00-kubernetes-crd-definition-v1.yml -f 01-kubernetes-crd-rbac.yml -f 02-traefik.yml

Configure Cert-manager with letsencrypt

Add cert-manager resources:

curl -SsL https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml -o 00-cert-manager.yaml
kubectl apply -f 00-cert-manager.yaml

Test a staging certificate

Test a staging certificate to validate letsencrypt integration

(Note, letsencrypt has a rate limit for request in production environment, so it is better to test the first setup in the staging one)

Create the issuer

Please, replace <YOUR_EMAIL_FOR_NOTIFICATION> with your email, this is useful to have notification of problems with certificate renewals.

cat > 01-letsencrypt-issuer-staging.yaml << 'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  labels:
    stack: cert-manager
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: <YOUR_EMAIL_FOR_NOTIFICATION>
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
      - http01:
          ingress:
            ingressTemplate:
              metadata:
                annotations:
                  "traefik.ingress.kubernetes.io/router.entrypoints": "web"
EOF
kubectl apply -f 01-letsencrypt-issuer-staging.yaml

Add a staging certificate

Please, replace <YOUR_CERT_NAME> with a name for your certificate, e.g. www-example-com-staging-cert

Please, replace <YOUR_DOMAIN> with your domain name, e.g. www.example.com

cat > 02-certificate-staging.yml << 'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: <YOUR_CERT_NAME>-staging-cert
  labels:
    stack: cert-manager
spec:
  secretName: <YOUR_CERT_NAME>-staging-cert
  commonName: <YOUR_DOMAIN>
  dnsNames:
    - <YOUR_DOMAIN>
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-staging
EOF
kubectl apply -f 02-certificate-staging.yml

validate staging certificate

kubectl get clusterissuer
kubectl describe clusterissuer letsencrypt-staging

kubectl get certificate
kubectl describe certificate <YOUR_CERT_NAME>-staging-cert
kubectl describe certificaterequest <YOUR_CERT_NAME>-staging-cert-XXXXX

kubectl describe order <YOUR_CERT_NAME>-staging-cert-XXXXX-XXXXXXXXXX
kubectl describe challenge <YOUR_CERT_NAME>-staging-cert-XXXXX-XXXXXXXXXX-XXXXXXXXXX

kubectl get ingress
kubectl describe ingress cm-acme-http-solver-XXXXX
kubectl describe service cm-acme-http-solver-XXXXX

Create a production certificate

Create the issuer

Please, replace <YOUR_EMAIL_FOR_NOTIFICATION> with your email, this is useful to have notification of problems with certificate renewals.

cat > 03-letsencrypt-issuer-production.yaml << 'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <YOUR_EMAIL_FOR_NOTIFICATION>
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
      - http01:
          ingress:
            ingressTemplate:
              metadata:
                annotations:
                  "traefik.ingress.kubernetes.io/router.entrypoints": "web"
EOF
kubectl apply -f 03-letsencrypt-issuer-production.yaml

Add a production certificate

Please, replace <YOUR_CERT_NAME> with a name for your certificate, e.g. www-example-com-production-cert

Please, replace <YOUR_DOMAIN> with your domain name, e.g. www.example.com

cat > 04-certificate-production.yml << 'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: <YOUR_CERT_NAME>-production-cert
spec:
  secretName: <YOUR_CERT_NAME>-production-cert
  commonName: <YOUR_DOMAIN>
  dnsNames:
  - <YOUR_DOMAIN>
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-production
EOF
kubectl apply -f 04-certificate-production.yml

validate production certificate

kubectl get clusterissuer
kubectl describe clusterissuer letsencrypt-production

kubectl get certificate
kubectl describe certificate <YOUR_CERT_NAME>-production-cert
kubectl describe certificaterequest <YOUR_CERT_NAME>-production-cert-XXXXX

kubectl describe order <YOUR_CERT_NAME>-production-cert-XXXXX-XXXXXXXXXX
kubectl describe challenge <YOUR_CERT_NAME>-production-cert-XXXXX-XXXXXXXXXX-XXXXXXXXXX

kubectl get ingress
kubectl describe ingress cm-acme-http-solver-XXXXX
kubectl describe service cm-acme-http-solver-XXXXX

References

Add a whoami stateless application

Then we can add an example resources to test the add of a new application on our cluster and configure the http route to the application:

testing whoami resouce 01-whoami.yml

Please, replace <YOUR_CERT_NAME> with a name for your certificate, e.g. www-example-com-production-cert

Please, replace <YOUR_DOMAIN> with your domain name, e.g. www.example.com

cat > 01-whoami.yml << 'EOF'

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami-ing
  labels:
    stack: whoami
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: 'true'
    certmanager.k8s.io/cluster-issuer: letsencrypt-production
spec:
  rules:
    - http:
        paths:
          - path: /whoami
            pathType: Exact
            backend:
              service:
                name:  whoami-srv
                port:
                  number: 80
  tls:
    - hosts:
        - <YOUR_DOMAIN>
      secretName: <YOUR_CERT_NAME>-production-cert

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami-http-ing
  labels:
    stack: whoami
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
    traefik.ingress.kubernetes.io/router.middlewares: default-whoami-redirectscheme@kubernetescrd
spec:
  rules:
    - http:
        paths:
          - path: /whoami
            pathType: Exact
            backend:
              service:
                name:  traefik-srv
                port:
                  number: 80

---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: whoami-redirectscheme
  labels:
    stack: whoami
spec:
  redirectScheme:
    scheme: https
    permanent: true

---
apiVersion: v1
kind: Service
metadata:
  name: whoami-srv
  labels:
    stack: whoami
spec:
  ports:
    - name: http
      port: 80
  selector:
    stack: whoami

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami-dpl
  labels:
    stack: whoami
spec:
  replicas: 1
  selector:
    matchLabels:
      stack: whoami
  template:
    metadata:
      labels:
        stack: whoami
    spec:
      containers:
        - name: whoami-cnt
          image: docker.io/traefik/whoami
          imagePullPolicy: Always
          ports:
            - containerPort: 80

EOF
kubectl apply -f 01-whoami.yml

Add a stateful applications

Kubernetes allows local storage volumes to be used in your apps, but does not takes decision about the lifecycle of the data. If you choose local storages, the persistent volumes are automatically claimed to be used and released when not needed, as usual, but they will remains in a released state, with the data untouched. It is responsibility of an human admin (or third part software managers) to decides what to do with the data, remove the claim to the volume, and set the volume in a usable state.

Usually, you may need to:

remove the pvc:

kubectl delete persistentvolumeclaim/YOUR_PVC_NAME

manually clean or reset volume data if needed (rm -rf)

mark the pv as reusable:

kubectl patch persistentvolume/YOUR_PV_NAME -p '{"spec":{"claimRef": null}}'

from https://medium.com/building-the-open-data-stack/reclaiming-persistent-volumes-in-kubernetes-5e035ba8c770

and https://kubernetes.io/docs/concepts/storage/persistent-volumes/#lifecycle-of-a-volume-and-claim

Postgresql example

from https://sweetcode.io/how-to-use-kubernetes-to-deploy-postgres/

and https://medium.com/@suyashmohan/setting-up-postgresql-database-on-kubernetes-24a2a192e962

A stateful application requires a persistent volume to store the data. In this example, we will use a local persistent volume to deploy a StatefulSet app. See also https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/ for an example.

Setup a storage of type local, a persistent volume with a max size mapped to a path of a local-storage, and add a postgres StatefulSet app claiming a part (or full) size of the persistent volume

Please note that you need to fix the local volume to a node, so you need to change the node name tf-instance2 to your node name, and you need to change the path /var/local/kube-local-storage/postgres-pv-2 to an existing path

cat > postgresql.yml << 'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
#  annotations:
#    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-pv-2
  labels:
    app: postgres
    type: local
spec:
  capacity:
    storage: 8Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /var/local/kube-local-storage/postgres-pv-2
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - tf-instance2

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-configuration
  labels:
    app: postgres
data:
  POSTGRES_DB: mydb
  POSTGRES_USER: mydbuser
  POSTGRES_PASSWORD: mydbpass

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-statefulset
  labels:
    app: postgres
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  volumeClaimTemplates:
  - metadata:
      name: postgres-pv-claim-template
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: local-storage
      resources:
        requests:
          storage: 8Gi
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:15
        envFrom:
        - configMapRef:
            name: postgres-configuration
        ports:
        - containerPort: 5432
          name: postgresdb
        volumeMounts:
        - name: postgres-pv-claim-template
          mountPath: /var/lib/postgresql/data

---
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  type: ClusterIP
  selector:
    app: postgres
  ports:
    - protocol: TCP
      port: 5432
      name: postgresdb
EOF

apply

kubectl apply -f postgresql.yml

logs

kubectl logs -f -l app=postgres

see volumes in your node

kubectl get pv

delete

kubectl delete -f postgresql.yml

if volumes is stuck in state Terminating, edit it and remove its constraint

  finalizers:
  - kubernetes.io/pv-protection

using the command

kubectl edit pv

Add postgres High Availability stateful app

from https://www.postgresql.org/download/products/3-clusteringreplication/

Using Kubegres

from https://www.kubegres.io/doc/getting-started.html

Add kubegres operator

curl -SsL https://raw.githubusercontent.com/reactive-tech/kubegres/v1.17/kubegres.yaml -o 00-kubegres.yml
kubectl apply -f 00-kubegres.yml

Setup a storage of type local, a Persistent Volume for each node, and remember to create the Persistent Volume paths on local nodes

First setup a storage class:

cat > 01-storage-class.yml << 'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: kubegres-storage-class
  labels:
    stack: kubegres
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
EOF
kubectl apply -f 01-storage-class.yml

Next, for each node instance:

Please, first define the following env vars:

STORAGE_PATH=<YOUR_INSTANCE_STORAGE_PATH> # example /var/local/kubernetes-storage/kubegres-persistent-volume
STORAGE_SIZE=<YOUR_INSTANCE_STORAGE_SIZE> # example 8Gi
INSTANCE_NAME=<YOUR_INSTANCE_NAME_N>      # example tf-instance0

Then setup the 02-persistent-volume-${INSTANCE_NAME}.yml resource file:

cat > 02-persistent-volume-${INSTANCE_NAME}.yml << EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: kubegres-persistent-volume-${INSTANCE_NAME}
  labels:
    stack: kubegres
spec:
  capacity:
    storage: ${STORAGE_SIZE}
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: kubegres-storage-class
  local:
    path: ${STORAGE_PATH}
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
              - ${INSTANCE_NAME}
EOF
kubectl apply -f 02-persistent-volume-${INSTANCE_NAME}.yml

create the Postgres cluster

Please, first define the following env vars:

SUPERUSER_PASSWORD=<YOUR_SUPERUSER_PASSWORD>
REPLICAUSER_PASSWORD=<YOUR_REPLICAUSER_PASSWORD>
REPLICAS=<YOUR_PERSISTENT_VOLUME_COUNT>          # example 4
STORAGE_SIZE=<YOUR_INSTANCE_STORAGE_SIZE>        # example 8Gi

Then, create a secret to store the passwords:

cat > 03-secret.yml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: kubegres-cluster-scr
  labels:
    stack: kubegres
type: Opaque
stringData:
  superUserPassword: ${SUPERUSER_PASSWORD}
  replicationUserPassword: ${REPLICAUSER_PASSWORD}
EOF
kubectl apply -f 03-secret.yml

Finally, create the cluster:

cat > 04-kubegres-cluster.yml << EOF
apiVersion: kubegres.reactive-tech.io/v1
kind: Kubegres
metadata:
  name: kubegres-cluster
  labels:
    stack: kubegres
spec:
  replicas: ${REPLICAS}
  image: docker.io/postgres:16
  port: 5432
  database:
    size: ${STORAGE_SIZE}
    storageClassName: kubegres-storage-class
    volumeMount: /var/lib/postgresql/data
  resources:
    limits:
      memory: 2Gi
    requests:
      memory: 200Mi
  env:
    - name: POSTGRES_PASSWORD
      valueFrom:
        secretKeyRef:
          name: kubegres-cluster-scr
          key: superUserPassword
    - name: POSTGRES_REPLICATION_PASSWORD
      valueFrom:
        secretKeyRef:
          name: kubegres-cluster-scr
          key: replicationUserPassword
EOF
kubectl apply -f 04-kubegres-cluster.yml

To see kubegres resources:

kubectl get all,ing,sc,pv,cm,secret -A -o wide --show-labels | grep gres

To check node logs:

kubectl logs --all-containers --prefix=true -l app=kubegres-cluster | sort

To access postgres database:

kubectl exec $(kubectl get pod -l app=kubegres-cluster,replicationRole=primary -o name) -it -- bash
su - postgres
psql
...

To solve problems with claimed but unused persistent volumes

Kubegres will create claims on persistent volumes (i.e. persistentvolumeclaim a.k.a. pvc) to run pods as statefulset apps.

When something goes wrong with the pod, it may decides to drop it and create a new one. If there is an available persistent volume for that, it will use the new one. But it cannot reuse the old claim (not sure if in some cases or in all cases) or the previous used pv, so if there is no pv available the satefulset app will be unable to create the pod. To unlock the situation, you need to remove the pvc, eventually clean the data if needed, and mark the volume as reusable as explained before.

Using CloudNativePG

from https://cloudnative-pg.io/docs/

and https://medium.com/supportsages/set-up-cloudnativepg-postgresql-operator-for-kubernetes-on-aks-2dbc00c770ca

Add cloudnative-pg operator

curl -SsL https://github.com/cloudnative-pg/cloudnative-pg/releases/download/v1.21.3/cnpg-1.21.3.yaml -o 00-cloudnative-pg.yml
kubectl apply -f 00-cloudnative-pg.yml

Setup a storage of type local, a Persistent Volume for each node, and remember to create the Persistent Volume paths on local nodes

First setup a storage class:

cat > 01-storage-class.yml << 'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cloudnative-pg-storage-class
  labels:
    stack: cloudnative-pg
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
EOF
kubectl apply -f 01-storage-class.yml

Next, for each node instance:

Please, first define the following env vars:

STORAGE_PATH=<YOUR_INSTANCE_STORAGE_PATH> # example /var/local/kubernetes-storage/cloudnative-pg-persistent-volume
STORAGE_SIZE=<YOUR_INSTANCE_STORAGE_SIZE> # example 8Gi
INSTANCE_NAME=<YOUR_INSTANCE_NAME_N>      # example tf-instance0

Then setup the 02-persistent-volume-${INSTANCE_NAME}.yml resource file:

cat > 02-persistent-volume-${INSTANCE_NAME}.yml << EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: cloudnative-pg-persistent-volume-${INSTANCE_NAME}
  labels:
    stack: cloudnative-pg
spec:
  capacity:
    storage: ${STORAGE_SIZE}
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: cloudnative-pg-storage-class
  local:
    path: ${STORAGE_PATH}
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
              - ${INSTANCE_NAME}
EOF
kubectl apply -f 02-persistent-volume-${INSTANCE_NAME}.yml

create the Postgres cluster

Please, first define the following env vars:

SUPERUSER_PASSWORD=<YOUR_SUPERUSER_PASSWORD>
INSTANCES=<YOUR_INSTANCES_COUNT>                 # example 3
STORAGE_SIZE=<YOUR_INSTANCE_STORAGE_SIZE>        # example 8Gi

Then, create a secret to store the passwords:

cat > 03-secret.yml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: cloudnative-pg-cluster-scr
  labels:
    stack: cloudnative-pg
type: Opaque
stringData:
  password: ${SUPERUSER_PASSWORD}
  username: postgres
EOF
kubectl apply -f 03-secret.yml

Finally, create the cluster:

cat > 04-cloudnative-pg-cluster.yml << EOF
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cloudnative-pg-cluster
  labels:
    stack: cloudnative-pg
spec:
  instances: ${INSTANCES}
  imageName: ghcr.io/cloudnative-pg/postgresql:16.1
  primaryUpdateStrategy: unsupervised
  enableSuperuserAccess: true
  superuserSecret:
    name: cloudnative-pg-cluster-scr
  storage:
    pvcTemplate:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: ${STORAGE_SIZE}
      storageClassName: cloudnative-pg-storage-class
      volumeMode: Filesystem
  resources:
    limits:
      memory: 2Gi
    requests:
      memory: 200Mi
EOF
kubectl apply -f 04-cloudnative-pg-cluster.yml

To see kubegres resources:

kubectl get all,ing,sc,pv,cm,secret -A -o wide --show-labels | grep cloudnative

To check node logs:

kubectl logs --all-containers --prefix=true -l cnpg.io/cluster=cloudnative-pg-cluster | sort

To access postgres database:

kubectl exec $(kubectl get pod -l cnpg.io/cluster=cloudnative-pg-cluster,cnpg.io/instanceRole=primary -o name) -it -- psql -U postgres
...

Notes

Kubectl

To see most of the resources:

kubectl get all,ing,sc,pv,pvc,cm,secret,cronjob -A -o wide

Example to run a command in a standalone container

# bash inside a debian container
kubectl run --rm -it --image debian:12.1-slim -- bash
# psql inside a postgres container
kubectl run -it --rm --image=postgres:15.4 -- psql -H <MY_POSTGRES_HOST> -U postgres;

You can convert helm files to native kubectl files, this is an example to convert hydra helm package:

./helm template hydra ory/hydra --output-dir ./hydra

kubectl apply does a three-way diff

from https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#kubectl-apply

kubectl apply does a three-way diff between the previous configuration, the provided input and the current configuration of the resource, in order to determine how to modify the resource.

Containerd

Note that, if you follow this guide, you are using containerd instead of docker, so some usual commands changes, for instance

list node images in the kubernetes namespace

ctr -n k8s.io i ls

remove all images containing a text in the name

ctr -n k8s.io i rm $(ctr -n k8s.io i ls -q | grep PARTIALNAME)

References

Free Oracle Cloud Kubernetes cluster with Terraform A Kubernetes guide for Docker Swarm lovers

Kubernetes

Contents

Install

Prerequisites

Install base packages

Install Containerd

Install Kubeadm

Install Kubectl

Cluster setup

NON HA cluster (no load balancer required)

HA cluster (load balancer REQUIRED)

add a pod network

Links

add more nodes to the cluster

Exposing cluster applications outside the cluster

Using Traefik as ingress controller

Configure Cert-manager with letsencrypt

Test a staging certificate

Create a production certificate

References

Add a whoami stateless application

Add a stateful applications

Postgresql example

Add postgres High Availability stateful app

Using Kubegres

Using CloudNativePG

Notes

Kubectl

Containerd

References

Navigation menu

Kubernetes

Install

Prerequisites

Install base packages

Install Containerd

Install Kubeadm

Install Kubectl

Cluster setup

NON HA cluster (no load balancer required)

HA cluster (load balancer REQUIRED)

add a pod network

Links

add more nodes to the cluster

Exposing cluster applications outside the cluster

Using Traefik as ingress controller

Configure Cert-manager with letsencrypt

Test a staging certificate

Create a production certificate

References

Add a whoami stateless application

Add a stateful applications

Postgresql example

Add postgres High Availability stateful app

Using Kubegres

Using CloudNativePG

Notes

Kubectl

Containerd

References

Navigation menu

Search