Your own Kubernetes environment

03 Jul 2023

How to build a complete Kubernetes environment on a single spare PC or EC2 VM, complete with storage, load balancer(s) and ingress. This is great for testing. For a more production grade setup add some more PCs or VMs.

For easy DNS support, add static hostnames on your router for any services you want to expose.

PC Setup

Wipe everything and install Debian 12. If you have a spare high capacity drive mount it somewhere like /data and create paths under this directory for your PVs. Consider buying more RAM if planning to run heavyweight services.

K3s (Kubernetes)

A complete Kubernetes environment, from Rancher: https://k3s.io/.

Install

curl -sfL https://get.k3s.io | sh -s - --disable traefik --disable servicelb

kubectl

Copy /etc/rancher/k3s/k3s.yaml from the node to ~/.kube/config on your workstation
Adjust server: to be match the fqdn of your k3s node
Test: kubectl cluster-info

Storage

K3s has built-in support for simple node-local files through local-path storageClassName. This is fine for simple testing. Take a look at Longhorn if replicated storage is required.

To use local-path storage you will need a pv and a pvc.

Example

Note storage capacity is required by the spec but local-path does not support applying the limit.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: something-pv
spec:
  storageClassName: "local-path"
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  hostPath:
    path: /data/something
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: something-pvc
  namespace: yourapp
spec:
  volumeName: something-pv
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: local-path

MetalLB (load balancer)

MetalLB is the perfect load balancer implementation when not running on cloud-vendor Kubernetes distributions. It attaches additional IP addresses to the NIC and adds an external IP address to Kubernetes Service with spec.type: LoadBalancer.

Install

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.10/config/manifests/metallb-native.yaml

Configure

kubectl apply -f each file:

pool.yaml

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default
  namespace: metallb-system
spec:
  addresses:
  # Address pool MetalLB will allocate from, adjust your router to not
  # allocate address in this range to prevent clash
  - 192.168.1.5-192.168.1.99

advertisement.yaml

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default-advertisement
  namespace: metallb-system

Use

Annotate Service of type: LoadBalancer with the IP address from the pool to allocate:

  annotations:
    metallb.universe.tf/loadBalancerIPs: 192.168.1.11

TLS/CA

Optional but recommended: Create a self-signed CA and wildcard certificate to use with Istio to expose HTTPS web services.

# CA
openssl genrsa -out ca-key.pem 2048

openssl req -new -key ca-key.pem -x509 \
  -days 1000 \
  -out ca.pem \
  -subj "/C=AU/ST=NSW/L=Sydney/O=DSys/O=someorg/CN=something"

# CSR
openssl req -nodes -newkey rsa:2048 -keyout wildcard.your.domain.key -out wildcard.your.domain.csr -subj "/C=AU/ST=NSW/L=Sydney/OU=something/CN=*.your.domain"

# Cert
openssl x509 -req -in wildcard.your.domain.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out wildcard.your.domain.pem -days 1000

Istio (ingress)

Istio routes HTTP/S traffic into Kubernetes, its preferred to use one or more ingress vs attaching load balancers to individual Service unless the service must expose a TCP wire protocol. This lowers infrastructure overhead (and cost if using cloud)

Install Istio

Note Based on the Install with Helm instructions

# install helm chart
helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update

# main istio components
kubectl create namespace istio-system
helm install istio-base istio/base -n istio-system --set defaultRevision=default
helm install istiod istio/istiod -n istio-system --wait

Install Istio gateway

Istio gateway is the component that creates the Kubernetes load balancer Service. Use MetalLB annotations to give it a static IP address and configure it to listen on ports 80 and 443:

values.yaml

service:
  annotations:
    metallb.universe.tf/loadBalancerIPs: 192.168.1.8
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
  - name: https
    port: 443
    targetPort: 443
    protocol: TCP

Install with helm:

helm upgrade --install istio-ingress istio/gateway -n istio-ingress --values values.yaml

Istio Gateway resource

Gateway resource binds to the loadbalancer. If using TLS also load the credential created earlier as a secret, then apply gateway.yaml:

kubectl create -n istio-ingress secret tls tls-wildcard --key=wildcard.your.domain.key --cert=wildcard.your.domain.pem

gateway.yaml

---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  namespace: istio-ingress
  name: gateway
spec:
  selector:
    app: istio-ingress
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
  # Omit if not configuring TLS
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: tls-wildcard
    hosts:
    - "*"

kubectl apply -f gateway.yaml
kubectl create namespace istio-ingress

# if using TLS
kubectl create -n istio-ingress secret tls tls-wildcard   --key=/home/geoff/ca/wildcard.some.domain.key   --cert=/home/geoff/ca/wildcard.some.domain.pem

DNS

Just add static hostnames from your router to the IP addresses allocated by MetalLB. Bonus points: install OpenWRT.

Example deployment

Test Istio and MetalLB, load the httpbin sample web server:

kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml

httpbin.yaml

---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin
  namespace: istio-ingress
spec:
  hosts:
  - 'httpbin.your.domain'
  gateways:
  - gateway
  tls:
  - match:
    - port: 443
      sniHosts:
      - 'httpbin.your.domain'
    route:
    - destination: 
        host: httpbin.default.svc.cluster.local
        port:
          number: 8000
  http:
  - route:
    - destination:
        host: httpbin.default.svc.cluster.local
        port:
          number: 8000

kubectl apply -f httpbin.yaml, then add the host httpbin.your.domain as a static record on your router pointing to the IP address of the Istio load balancer.

Test the service, you should see a teapot:

curl -k https://httpbin.your.domain/status/418

    -=[ teapot ]=-

       _...._
     .'  _ _ `.
    | ."` ^ `". _,
    \_;`"---"`|//
      |       ;/
      \_     _/
        `"""`

geoffwilliams@home:~$