Migrating My Site to Kubernetes

META:kubernetes:

March 4th, 2018

Previously when I brought my my site back online I briefly mentioned the simple setup I threw together with Caddy running on a tiny GCE VM with a few scripts — Since then I’ve had plenty of time to experience the awesomeness that is managing services with Kubernetes at work while developing Kubernetes’s testing infrastructure (which we run on GKE).

So I decided, of course, that it was only natural to migrate my own service(s) to Kubernetes for maximum dog-fooding. :kubernetes: ↔ 🐶

This turned out to be even easier than expected and I was quickly up and running on a toy single-node cluster running on a spare linux box at home with the help of the excellent official docs for setting up a cluster with kubeadm. After that I set up ingress-nginx to handle ingress to my service(s) and kube-lego to manage letsencrypt certificates. I then replaced Caddy with my own minimal containerized Go service to continue having GitHub webhooks trigger site updates. :go_gopher:

I did run into the following hiccups:

To get DNS resolution within the cluster of external services I needed to configure kube-dns with kubectl apply -f ./k8s/kube-dns-configmap.yaml where my ./k8s/kube-dns-configmap.yaml contained:

k8s/kube-dns-configmap.yaml YAML
1# Use Google's public DNS to resolve external services
2apiVersion: v1
3kind: ConfigMap
4metadata:
5  name: kube-dns
6  namespace: kube-system
7data:
8  upstreamNameservers: |
9        ["8.8.8.8", "8.8.4.4"]

I also needed to configure RBAC for kube-lego which doesn’t currently ship with RBAC configured out of the box. Again, this was just involved applying a config update based on the comments at jetstack/kube-lego#99 with kubectl apply -f k8s/kube-lego.yaml. The config below is probably giving kube-lego a lot more access than it needs, but I wasn’t particularly concerned about this since this is on a toy “cluster” for my personal site and the service is already managing my TLS certificates. 🤷

My k8s/kube-lego.yaml contained:

k8s/kube-lego.yaml YAML
  1# Complete setup for kube-lego.
  2# The only thing specific to my cluster here is the lego.email setting,
  3# the rest is just kube-lego with RBAC.
  4# Thanks to comments at: https://github.com/jetstack/kube-lego/issues/99
  5apiVersion: v1
  6kind: Namespace
  7metadata:
name: kube-lego
  9---
 10apiVersion: v1
 11metadata:
name: kube-lego
namespace: kube-lego
 14data:
# modify this to specify your address
lego.email: "bentheelder@gmail.com"
# configure for letsencrypt's production api
lego.url: "https://acme-v01.api.letsencrypt.org/directory"
 19kind: ConfigMap
 20---
 21apiVersion: rbac.authorization.k8s.io/v1beta1
 22kind: ClusterRole
 23metadata:
  name: lego
 25rules:
 26- apiGroups:
- ""
- "extensions"
resources:
- configmaps
- secrets
- services
- endpoints
- ingresses
- nodes
- pods
verbs:
- list
- get
- watch
 41- apiGroups:
- ""
resources:
- services
verbs:
- create
 47- apiGroups:
- "extensions"
- ""
resources:
- ingresses
- ingresses/status
verbs:
- get
- update
- create
- list
- patch
- delete
- watch
 61- apiGroups:
- "*"
- ""
resources:
- events
- certificates
- secrets
verbs:
- create
- list
- update
- get
- patch
- watch
 75---
 76apiVersion: rbac.authorization.k8s.io/v1beta1
 77kind: ClusterRoleBinding
 78metadata:
name: lego
 80roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: lego
 84subjects:
- kind: ServiceAccount
  name: lego
  namespace: kube-lego
 88---
 89apiVersion: v1
 90kind: ServiceAccount
 91metadata:
name: lego
namespace: kube-lego
 94---
 95apiVersion: extensions/v1beta1
 96kind: Deployment
 97metadata:
name: kube-lego
namespace: kube-lego
100spec:
replicas: 1
template:
  metadata:
    labels:
      app: kube-lego
  spec:
    serviceAccountName: lego
    containers:
    - name: kube-lego
      image: jetstack/kube-lego:0.1.5
      imagePullPolicy: Always
      ports:
      - containerPort: 8080
      env:
      - name: LEGO_EMAIL
        valueFrom:
          configMapKeyRef:
            name: kube-lego
            key: lego.email
      - name: LEGO_URL
        valueFrom:
          configMapKeyRef:
            name: kube-lego
            key: lego.url
      - name: LEGO_NAMESPACE
        valueFrom:
          fieldRef:
            fieldPath: metadata.namespace
      - name: LEGO_POD_IP
        valueFrom:
          fieldRef:
            fieldPath: status.podIP
      readinessProbe:
        httpGet:
          path: /healthz
          port: 8080
        initialDelaySeconds: 5
        timeoutSeconds: 1
139---

After applying these two changes the rest of my very simple config to deploy the Go service behind automatic TLS termination worked flawlessly. Since then managing the site has been an excellent experience with the power of kubectl, the Kubernetes “swiss army knife”.

In conclusion:

If you haven’t given Kubernetes a try but you are already comfortable with Docker you should give it a try. Kubernetes makes managing services easy and portable. The same kubectl commands I use to debug our services for the Kubernetes project’s infrastructure on GKE work just as well on my toy cluster at home. 😄
If you want to give hosting on Kubernetes a try with much less effort, Google Cloud offers a free 12 month, $300 credit and an always-free tier which both include Google Kubernetes Engine.
We use GKE heavily for the project infrastructure and I can speak highly to it’s ease of use and freedom to focus on your services without worrying about setting up and maintaining all of the pluggable Kubernetes bits such as logging, master upgrades, node auto repair, IAM, cluster networking, etc.

If my site were a serious production service instead of a toy learning experience I would seriously look towards GKE instead of a one node “cluster” running on a DIY “server” sitting by my desk at home, but setting up a toy cluster with kubeadm was a great experience for experimenting with Kubernetes. I can recommend using kubeadm for similar experiments, it’s quite simple to use once you have all the prerequesites installed and configured and the docs are quite good, however it won’t solve many of the things you’ll want for a production cluster.

You may also want to look around the list of the many CNCF certified Kubernetes conformant products for other options if for some reason neither of these sound appealing to you.

If you really just want to play with it first (and not host anything), check out minikube.

Addendum

I also used Calico for my overlay network, but I haven’t really exercised it yet so I can’t really comment on it.
Kubernetes secrets are awesome. My simple Go service can just read in the GitHub webhook secret as an environment variable injected into the container without worrying about how the secret is loaded and stored.
To get a one node cluster working you need to remove the master taint. This is terrible idea for a production cluster but great for tinkering and effectively using kubelet as your PID1.

✎ Update

My site is on Netlify now, but I still run my own Kubernetes cluster to host other small projects. Hosting it on a toy Kubernetes cluster worked well, except when the power went out at my apartment … I’d like my site to be online even then, hence Netlify 🙃

✎ Update

I’ve since simplified my home server further to minimize time spent on upkeep, my website remains on Netlify since the last update. If you’re interested, you can read more on that in “Self-Driving Debian”.

I would still recommend considering Kubernetes for larger application infrastructure management, but unless you are doing so for educational purposes, I’m not sure I’d recommend it for simple small-scale deployments … Your results may vary.

Migrating My Site to Kubernetes

# Addendum

Addendum