Intro

I’m finally getting into Kubernetes, also known as k8s. This behemoth of infrastructure and Container Orchestration has long haunted me and long have I alluded it. No more!
The Tutorials | Kubernetes on the Kubernetes website has several tutorials to follow and try. I’ll be starting with Minikube to get a single cluster running on localhost

Starting with Minikube

Topics To Cover

Services deployment and exposure

Load Balancing

Automatic Scaling

Ingress for external access and domain names

ConfigMaps & Secrets: move any hardcoded config (API URLs, ports, env vars) out of your Docker images and into Kubernetes-managed config. IMPORTANT!

Persistent Volumes: add Redis and DBs with PersistentVolumeClaim for state

Automated update flows: for rolling out new app images

Helm

Grafanna and Prometheus

What is Kubernetes?

Kubernetes is a cluster orchestration tool for docker containers. What’s that mean? It manages running containers where and when you want them to, finds and allocates resources to them, manage their tools, update the software they need, etc.
It’s an abstraction layer for handling many containers running in a manner that decouples them from which machine’s running them.

Kubectl

Is the CLI that unlocks Kubernetes, whether that’s with Minikube on single cluster or K8s.
The common format of a kubectl command is: kubectl action resource.
For more commands, check the kubectl reference.

Important flags

-l: to specify a lable type and its value. E.g. kubectl get service -l app=<app-name>

Troubleshooting Commands

kubectl get - list resources
kubectl describe - show detailed information about a resource
kubectl logs - print the logs from a container in a pod
kubectl exec - execute a command on a container in a pod

Workflows: The Kubernetes Dev Cycle

It looks something like this, a waterfall of manifests being written and deployed.

Write Deployment (Runs the code) 
   └──> Write Service (Internal networking) 
           └──> Write Ingress (External access/domain names)

Deployments: write and apply them to create pods to run your code and apps
Services: write them and apply them to have a constant service that your pods can use to communicate, since pods are ephemeral and get replaced with new IP addresses constantly
Ingress: write and apply these to route actual public internet traffic like mysite.com to your Service

More on this here, Full K8s Lifecycle

YAML Manifest

Before getting into the different parts that make up K8s, you should know that Kubernetes really is all about YAML files defining what and how to do things. These are known as Manifests.
Each manifest has two distinct parts:

Kubernetes Envelope (the metadata that tells K8s what this object is)
Specification (the actual settings for your application).
Example Outline:

# 1. THE ENVELOPE (Every K8s object needs these 4 fields)
apiVersion: apps/v1      # Which version of the K8s API to talk to (e.g., v1, apps/v1)
kind: Deployment         # What type of object you are creating
metadata:
  name: my-app-deploy    # The unique name of this object inside the cluster
  namespace: default     # (Optional) The logical folder/space to put it in
  labels:
    app: my-web-app      # Tags used to organize and link objects together
 
# 2. THE SPECIFICATION (The "What I want you to do" part)
spec:
  # This section changes completely depending on the "kind" above.
  # For a Deployment, it usually looks like this:
  
  replicas: 3            # How many identical copies (Pods) you want running
  
  selector:
    matchLabels:
      app: my-web-app    # Tells the deployment which Pods it is responsible for
      
  # 3. THE TEMPLATE (The blueprint for the actual containers)
  template:
    metadata:
      labels:
        app: my-web-app  # MUST match the selector above!
    spec:
      containers:
      - name: web-container
        image: nginx:1.25  # The Docker/OCI image to run
        ports:
        - containerPort: 80 # The port the app listens on inside the container

Important to remember:

apiVersion & kind: tells Kubernetes, “Hey, look up the rules for a Deployment under the apps/v1 API group.”
metadata: Assign your object a identity (name) and organize it with labels (key-value pairs).
spec: Outlines state, You are telling K3s, “Make the reality match these settings, and if something breaks, fix it until it matches again.”

Kubernetes Parts and Components

Kubernetes like most complex systems, is comprised of various components and parts that make up the whole system.

Clusters

K8s coordinates clusters of computers to work as a single unit in order to run containerized apps. The applications aren’t tied to any specific

Clusters consist of two types of resources:

Control Plane: this coordinates the cluser
Nodes: these are workers that actually run the application
// Think of a Supervisor & Worker dynamic

Control Planes

These manage the cluster. They coordinate activities within such as:

Scheduling apps
Maintaining apps state
Scaling apps
Rolling out updates

Nodes

Nodes are either VMs or physical computers serving as worker machines, they run the apps.
Each node is managed by a control plane.
Every Node runs at least:

kubelet, a process responsible for communication between the Kubernetes control plane and the Node; it manages the Pods and the containers running on a machine.
A container runtime (like Docker) responsible for pulling the container image from a registry, unpacking the container, and running the application

Node-level components like kubelets communicate with the control plane using the Kubernetes API, which is exposed by the control plane itself.
Users can also use it to interact with the cluster.

Pods

Pods are the smallest unit of a computing in the Kubernetes ecosystem. A Kubernetes Pod is a group of 1+ containers, that are bundled together in terms of admin and networking. By default a pod is only reachable by its internal IP address within the Kubernetes virtual network.

Pods are only visible to other pods and services on the same Kubernetes network, i.e. on the same cluster.
As such, they share storage, network resources, and specs on how to run the containers.

Pods house tightly coupled logical-units in one or more containers. An example can be a Node.js app in one container and another container containing a service that feeds data to the app, like a SQLite instance.
Since the containers in a Pod share an IP Address and port space, they’re also always co-located and co-scheduled, and run in a shared context on the same Node.

In order to use the pod externally it must be exposed. // Exposed pods are known as **Kubernetes Services**.
For a more in depth look, read this.

Executing Commands on a Pod

To run commands within a pod’s environment we run: kubectl exec "$POD_NAME"
For example, listing environment variables:

kubectl exec "$POD_NAME" -- env

Or starting a bash session in the container:

kubectl exec -ti $POD_NAME -- bash

Services

Services are another abstraction layer, one that defines logical set of Pods and enables external traffic exposure, load-balancing, and discoverability for those pods. I,e, it exposes them so they can be used while also managing the load and traffic from the outside.

Services are defined using YAML manifests, just like deployments. The set of Pods targeted by a service is determined by the label selector.

Exposure and Service Types

Despite pod’s having unique IP address (not exposed to public), a service allows them to receive traffic.
There are a few ways to specify what type in the spec portion of the manifest:

ClusterIP (default) : Exposes the Service on an internal IP in the cluster. This type makes the Service only reachable from within the cluster
NodePort : Exposes the Service on the same port of each selected Node in the cluster using NAT. Makes a Service accessible from outside the cluster using NodeIP:NodePort, it’s superset of ClusterIP
LoadBalancer : Creates an external load balancer in the current cloud (if supported) and assigns a fixed, external IP to the Service. Superset of NodePort
ExternalName : Maps the Service to the contents of the externalName field (e.g. foo.bar.example.com), by returning a CNAME record with its value. No proxying of any kind is set up. This type requires v1.7 or higher of kube-dns, or CoreDNS version 0.0.8 or higher

Headless Services

These are services with ClusterIP: None when gotten via kubectl. Their DNS is resolved directly through the pod IP. Standard for StatefulSets (see Persistent Volumes) and needs no special handling to resolve.

Labels

Services match a set of Pods using labels and selectors, allowing logical operations on objects in Kubernetes.
Labels: are key/value pairs attached to objects and used in many ways:

Designate objects for development, test, and production
Embed version tags
Classify an object using tags

Creating a Service

This can be done using a manifest YAML file or on an existing deployment.
The latter is done using this syntax:

kubectl expose deployment/<deployment-name> --type="NodePort" --port 8080

// The NodePort type exposes to external traffic.
Getting the services should show the external IP for the service.

The other method, using manifests by following the template:

apiVersion: v1
kind: Service
metadata:
  name: my-backend-service # how pods reach this service, needed for internal DNS
  namespace: default
spec:
  type: ClusterIP
  selector: # Tells the service which pods to direct traffic
    app: backend-api
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080

Breakdown:

Identity: The metadata, like name field dictates the internal DNS routing. Other Pods in the same namespace can reach this service simply by calling http://my-backend-service.
The Bridge: This is the selector field, the most crucial part. The selector tells the Service which Pods to send traffic to. It looks for Pods that have the exact matching label (app: backend-api in their own metadata). If this doesn’t match perfectly, your traffic hits a dead end. // `selector` must match pod `app` field
Ports: These are the routing engine, and involves two different ports:
- port: The port exposed by the Service itself. Other Pods talk to the service on this port (e.g., port 80).
- targetPort: The port that the actual application inside the Pod is listening on (e.g., port 8080).

Common Mistake

If deployed service returns a 503 Service Unavailable or a timeout, run kubectl get ep .
“EP” stands for Endpoints. Should show if your Service selector labels do not match your Pod labels.

Accessing Service

The service should be accessible via internalIP:serviceport for example:

curl http://<nodeIp>:<service-port>

Deployments

A Deployment checks on health of a pod, and restarts it if it terminates for whatever reason. They’re the recommended way for managing creation, and updating, scaling of pods.

Once a deployment’s in place, the control plane schedules the app instances included in that Deployment to run on a node in that cluster.
If something goes wrong and the instance dies, the deployment will manage replacing it on a different node. Providing a mechanism that’s fault-tolerant and self-healing.

What They Do

Define the details for deploying pods of containers using YAML config files known as Manifests.
These files outline details including:

Number of pods
Which images to run in them
Which port(s) to expose

Deploying an app

Using the command kubectl create deployment <deployment-name> --image=<image-url>
We can deploy a containerized (Docker) application to our cluster.
A few things happen to achieve this:

Searched for a suitable node where an instance of the application could be run
Scheduled the application to run on that Node
Configured the cluster to reschedule the instance on a new Node when needed
All automatically managed by the Control Plane of that cluster.

Ingress

An Ingress acts as a single entry point to a cluster (not a node, but the whole cluster) and occupies a public IP port (80 or 443). It’s job is to:

Read incoming web requests
Route them to different internal Services based on domain name or URL path

Components

For an ingress to work, two components are needed.

Ingress Controller: the engine behind it all. It’s a reverse proxy (like Nginx or Traefik) and runs inside the cluster, but stands at the cluster entrance listening for external traffic and routes
Ingress Resource: this is the YAML map that’s written. A set of rules telling the controller how to behave. For example: If a user navigates to api.example.com, send them to the backend-service but If they go to https://example.com/blog, send them to the blog-service

In standard Kubernetes, an Ingress controller like Ingress-nginx needs to be set up and exposed via manifests or tools like Helm.

Manifest template

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  annotations:
    # This tells Kubernetes which controller engine should process this rule
    kubernetes.io/ingress.class: "traefik" 
spec:
  rules:
  - host: my-app.vps-domain.com       # The domain name you want to use
    http:
      paths:
      - path: /                       # Match all paths after the domain
        pathType: Prefix
        backend:
          service:
            name: demo-web-service   # Send traffic to your internal Service!
            port:
              number: 80              # The port the Service is listening on

Environment Variables and Secrets

ConfigMaps

These YAML files define non-sensitive environment variables for your pods.
Ex:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "info"
  API_URL: "https://api.production.com"

Secrets

Secrets are stored in the cluster as base64 strings and are not encrypted by default.
Ex:

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  # 'supersecretkey' base64 encoded:
  API_KEY: c3VwZXJzZWNyZXRrZXk=

Enable Encryption at Rest in your Kubernetes cluster/etcd to truly secure them).

Applying Them

There’s two main ways to apply/inject these into your pods.

Env Vars
Volume Mounts (Recommended)

Env Vars

In the Deployment manifest, add the name of these variables/secrets in addition to their reference under the spec:containers: fields
For example:

# Deployment
# ...
spec:
	# ...
	containers:
      - name: app-container
        image: my-app:latest
        env:
        # Pulling from a ConfigMap
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: LOG_LEVEL
        # Pulling from a Secret
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: API_KEY

Injecting Mounted Volumes

This is much more secure than the previous method, isn’t susceptible to log-dumps, etc.

# Deployment
# ...
spec:
	# ...
	containers:
	- name: app-container
        image: my-app:latest
        volumeMounts:
        - name: secret-volume
          mountPath: "/etc/secrets"
          readOnly: true
      volumes:
      - name: secret-volume
        secret:
          secretName: app-secrets

The app reads the secrets at the mountPath

Proxy

Kubernetes cluster proxies forward communications into the KVN for that cluster.

Creating a Proxy

Run the kubectl proxy command.
This provides a connection between the host and the cluster, enabling direct access to the API.

We can see all those APIs hosted through the proxy endpoint. For example, we can query the version directly through the API using the curl command:
curl http://localhost:8001/version

Accessing Pods

The API creates endpoints per pod automatically, naming the endpoints based on the pod name.
To get the Pod name, and store it in the environment variable POD_NAME:

export POD_NAME=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
echo Name of the Pod: $POD_NAME

Now to access that pod:

curl http://localhost:8001/api/v1/namespaces/default/pods/$POD_NAME:8080/proxy/

Scaling Applications

Kuberentes can automatically scale the number of pods for a given service, based on the applied manifest.
Scaling is accomplished by changing the number of replicas in a Deployment.

Getting the set of replicas for a deployment:kubectl get rs.
The name of the ReplicaSet is always formatted as [DEPLOYMENT-NAME]-[RANDOM-STRING]. The random string is randomly generated and uses the pod-template-hash as a seed. For example:

NAME                    DESIRED   CURRENT   READY   AGE
nginxfirst-54ffcff76c   1         1         1       98m

Scaling Manually

This is done using the scale command on deployments.

kubectl scale deployments/<deployment-name> --replicas=4

When run, K8s will begin to scale until the current number of replicas matches the desired, in this case 4.

To scale Down we run the same command but set the replicas flag to a smaller number.

Load Balancing

Using Klipper

To avoid typing the ports like :31045 every time and instead want to hit your server on standard web ports (80 for HTTP or 443 for HTTPS), change the service type from NodePort to type: LoadBalancer.

Updating Images

It’s not advised to enter pods and update the apps there, since pods aren’t permanent, whatever reason a pod goes down, Kubernetes will replace it and the changes will be lost.
The Modus Operandi is:

Update code and apps locally where they live, rebuild the images (Docker) and push them to their registry. Examples include DockerHub, GitHub, etc.
Update the Deployment YAML files with new image tags and K8s will take care of rolling the changes out gracefully

The Update Flow

To update the image of the application to version 2, get your deployment and pod to verify details.
Then use the set image subcommand, followed by the deployment name and the new image version:

kubectl set image deployments/<deployment-name> image-name=<new-image-src>

Now the Deployment’s notified of the update and the rolling update is initiated. Pods that aren’t busy are replaced by pods with the new version.

You can also confirm the update by running the rollout status subcommand:

kubectl rollout status deployments/<deployment-name>

Debugging Exception

The only exception is debugging, you can go into a pod strictly for troubleshooting using kubectl exec like so:

kubectl exec -it  -- /bin/bash

This lets you look at logs, test database connections, or check local files, but not suitable for permanent configuration.

Rolling Back An Update

To roll back the deployment to your last working version, use the rollout undo subcommand:

kubectl rollout undo deployments/kubernetes-bootcamp

The rollout undo command reverts the deployment to the previous known state of the image. Updates are versioned and you can revert to any previously known state of a Deployment.

Networking

Every app has a targetport field (in the Service manifest) that matches the containerPort in the Deployment manifest.
That’s what the Service listens to on behalf of the pods running the application.
A web server (see next section) is the only component that’s listening to external traffic on ports 80 and 443 for the server’s IP address. I,e, the only one listening to external traffic.

The traffic flow looks something like this:

User → sub.domain.com:443
           │
        Traefik (listening on :80 and :443 on the node)
           │
        Ingress rules (route by path/host)
           │
        Service ClusterIP:80
           │
        Pod :8080 (targetPort, internal only)

Adding New Apps

When adding a new app to the cluster somethings need adding or updating

New targetPort for that app’s service
Service port is 80 by convention
Update the Ingress for that service, by adding host/path field(s) rules
What stays the same:
Traefik will continue listening to port 443
VPS IP doesn’t change obviously.
In summary, per new app, add a new Ingress rule, either a new path on the same host, or a new host entirely:

# New subdomain for a second app
- host: anotherapp.domain.come
  http:
    paths:
      - path: /
        backend:
          service:
            name: anotherapp-svc
            port:
              number: 80

Traefik will pick up the new rule automatically, no need to restart or change configs.

DNS and Web Servers

In order to be able to access apps from your cluster over the internet via a domain, you need to set up a Web Server and DNS.

Web Server / Reverse Proxy

Like mentioned in the Ingress#Components section, a reverseproxy or web server needs to be configured to handle routing and traffic for your cluster.
Examples include Nginx, Apache, and Traefik. K3s includes Traefik by default as its Ingress Controller.

DNS

Needed to have your domain properly point to the desired app.

TLS/SSL

Crucial for serving https and being accepted by browsers and other clients.
There are a few ways to do this, I’ll mention the quick bypass for dev/testing and the “proper” way…

There's also the K3s way 👀 Setting up K3s Cluster > Automated SSL Traefik + Cert-Manager + Let's Encrypt]

Quick Self-Signed Bypass

This is for development, please don’t use this for production!
Steps:-

Generate Keys: first use openssl to create a public certificate (tls.crt) and a private key (tls.key).
Create a Secret: then store these files inside a Kubernetes object called a Secret of type kubernetes.io/tls. // lookup the docs
Bind to Ingress: Then reference that Secret name inside your Ingress manifest.

Slight Hiccup

Browsers on your PC will still not trust this because it’s self-signed and throw a giant warning! But you can brute past it and by clicking “proceed” or the like.
Don’t keep this in PROD!!!

K8s way with cert-manager & Nginx

In a standard or enterprise K8s cluster, the certificates aren’t built directly into the Ingress-Controller. Instead, a modular architecture’s implemented using a industry-standard called cert-manager
cert-manager: watches the cluster’s ingress files, and communicates with Let’s Encrypt (a free, automated public certificate authority) to grab SSL certificates and spin up Kubernetes Secrets. // doesn't sound complex at all...

The Standard Multi-Step Workflow:

[ Your Ingress ] ──(Spotted by)──> [ cert-manager Operator ]
                                            │
                                    (Automated ACME Challenge)
                                            ▼
[ K8s TLS Secret ] <──(Saves Cert)── [ Let's Encrypt CA ]

Step 1: Install `cert-manager`

Usually installed via Helm (the Kubernetes package manager) or a single massive manifest:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml

Step 2: Configure the ClusterIssuer

You have to create a cluster-wide configuration file (ClusterIssuer) that defines how to talk to Let’s Encrypt. This configuration sits separately from your apps:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - http01:
        ingress:
          class: nginx   # Works whether you use nginx, traefik, or kong

Step 3: Write Ingress Resource

Now cert-manager is running in the background, add a simple annotation line to your regular Ingress manifest. This tells cert-manager to monitor the file and handle SSL while adding the results to a secret named cortado-vanilla-tls.”

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ar-ingress
  namespace: audioreaper-astro
  annotations:
    kubernetes.io/ingress.class: "nginx"
    # This magic annotation triggers cert-manager to fetch the SSL certificate automatically!
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - cortado.aabuharrus.dev
    secretName: cortado-vanilla-tls
  rules:
  - host: cortado.aabuharrus.dev
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: ar-astro-svc
            port:
              number: 80

Persistent Volumes

Two things to know when you have an app that needs to retain data beyond the lifespan of a any particular pod.

Persistent Volume (PV): storage within the Kubernetes cluster
Persistent Volume Claims (PVC): A claim/request for storage

Any database or storage engine running in the cluster should be a StatefulSet, not a Deployment.

// Continue this!!!

Clarification on Structure

I found myself confused on where to deploy the volumes (DBs / caches), so here’s the rundown.

Inside the Cluster

This is the common approach for dev and hobby projects.
The DB can run like any other service in pods in the cluster. Deploy it via Helm or kubectl, it gets a ClusterIP Service, and your app pods reach it over the internal cluster network by its service name.

Outside the Cluster

Installed directly on the system via a package manager. Then the pods connect to it via the node’s internal IP address.

Apparently it’s common practice to have a single PV instance of something like Redis or Postgres that’s shared by the apps.
I know for example that a single Postgres instance can house multiple databases. E.g. app1_db and app2_db live in the same Postgres instance.
Whereas Redis has namespace or key-prefixes to avoid keycollision.

Package Management

There are a few ways to handle packages cluster wide, the most popular is Helm.
While kubectl lets us write our apps’ manifests, configure, and deploy them; Helm is more suited for projects that aren’t ours. // Other people's software onto our cluster
Examples include Postgres, Redis, Grafana, cert-manager, ingress controllers, etc. This spares us the need to update piles of complex manifests, let people more familiar do that…

Logic Flow

If you’re deploying your own app, use kubectl. If you’re deploying a third-party service or tool onto your cluster, reach for Helm first and see if a chart exists.

Abdu's Garden

Explorer

Starting with Minikube

Intro

What is Kubernetes?

Kubectl

Important flags

Troubleshooting Commands

Workflows: The Kubernetes Dev Cycle

YAML Manifest

Kubernetes Parts and Components

Clusters

Control Planes

Nodes

Pods

Executing Commands on a Pod

Services

Exposure and Service Types

Labels

Creating a Service

Accessing Service

Deployments

What They Do

Deploying an app

Ingress

Components

Manifest template

Environment Variables and Secrets

ConfigMaps

Secrets

Applying Them

Env Vars

Injecting Mounted Volumes

Proxy

Creating a Proxy

Accessing Pods

Scaling Applications

Scaling Manually

Load Balancing

Using Klipper

Updating Images

The Update Flow

Debugging Exception

Rolling Back An Update

Networking

Adding New Apps

DNS and Web Servers

Web Server / Reverse Proxy

DNS

TLS/SSL

Quick Self-Signed Bypass

K8s way with cert-manager & Nginx

The Standard Multi-Step Workflow:

Step 1: Install cert-manager

Step 2: Configure the ClusterIssuer

Step 3: Write Ingress Resource

Persistent Volumes

Clarification on Structure

Inside the Cluster

Outside the Cluster

Sharing Instances

Package Management

Resources

Step 1: Install `cert-manager`