Migrating My Site to Kubernetes


March 4th, 2018

Previously when I brought my my site back online I briefly mentioned the simple setup I threw together with Caddy running on a tiny GCE VM with a few scripts — Since then I’ve had plenty of time to experience the awesomeness that is managing services with Kubernetes at work while developing Kubernetes’s testing infrastructure (which we run on GKE).

So I decided, of course, that it was only natural to migrate my own service(s) to Kubernetes for maximum dog-fooding. :kubernetes::dog:

This turned out to be even easier than expected and I was quickly up and running on a toy single-node cluster running on a spare linux box at home with the help of the excellent official docs for setting up a cluster with kubeadm. After that I set up ingress-nginx to handle ingress to my service(s) and kube-lego to manage letsencrypt certificates. I then replaced Caddy with my own minimal containerized Go service to continue having GitHub webhooks trigger site updates. :go_gopher:

I did run into the following hiccups:

1) To get DNS resolution within the cluster of external services I needed to configure kube-dns with kubectl apply -f ./k8s/kube-dns-configmap.yaml where my ./k8s/kube-dns-configmap.yaml contained:

k8s/kube-dns-configmap.yaml YAML
1# Use Google's public DNS to resolve external services
2apiVersion: v1
3kind: ConfigMap
5  name: kube-dns
6  namespace: kube-system
8  upstreamNameservers: |
9    ["", ""]

2) I also needed to configure RBAC for kube-lego which doesn’t currently ship with RBAC configured out of the box. Again, this was just involved applying a config update based on the comments at jetstack/kube-lego#99 with kubectl apply -f k8s/kube-lego.yaml. The config below is probably giving kube-lego a lot more access than it needs, but I wasn’t particularly concerned about this since this is on a toy “cluster” for my personal site and the service is already managing my TLS certificates. :shrug:

My k8s/kube-lego.yaml contained:

k8s/kube-lego.yaml YAML
  1# Complete setup for kube-lego.
  2# The only thing specific to my cluster here is the lego.email setting,
  3# the rest is just kube-lego with RBAC.
  4# Thanks to comments at: https://github.com/jetstack/kube-lego/issues/99
  5apiVersion: v1
  6kind: Namespace
  8  name: kube-lego
 10apiVersion: v1
 12  name: kube-lego
 13  namespace: kube-lego
 15  # modify this to specify your address
 16  lego.email: "bentheelder@gmail.com"
 17  # configure for letsencrypt's production api
 18  lego.url: "https://acme-v01.api.letsencrypt.org/directory"
 19kind: ConfigMap
 21apiVersion: rbac.authorization.k8s.io/v1beta1
 22kind: ClusterRole
 24    name: lego
 26- apiGroups:
 27  - ""
 28  - "extensions"
 29  resources:
 30  - configmaps
 31  - secrets
 32  - services
 33  - endpoints
 34  - ingresses
 35  - nodes
 36  - pods
 37  verbs:
 38  - list
 39  - get
 40  - watch
 41- apiGroups:
 42  - ""
 43  resources:
 44  - services
 45  verbs:
 46  - create
 47- apiGroups:
 48  - "extensions"
 49  - ""
 50  resources:
 51  - ingresses
 52  - ingresses/status
 53  verbs:
 54  - get
 55  - update
 56  - create
 57  - list
 58  - patch
 59  - delete
 60  - watch
 61- apiGroups:
 62  - "*"
 63  - ""
 64  resources:
 65  - events
 66  - certificates
 67  - secrets
 68  verbs:
 69  - create
 70  - list
 71  - update
 72  - get
 73  - patch
 74  - watch
 76apiVersion: rbac.authorization.k8s.io/v1beta1
 77kind: ClusterRoleBinding
 79  name: lego
 81  apiGroup: rbac.authorization.k8s.io
 82  kind: ClusterRole
 83  name: lego
 85  - kind: ServiceAccount
 86    name: lego
 87    namespace: kube-lego
 89apiVersion: v1
 90kind: ServiceAccount
 92  name: lego
 93  namespace: kube-lego
 95apiVersion: extensions/v1beta1
 96kind: Deployment
 98  name: kube-lego
 99  namespace: kube-lego
101  replicas: 1
102  template:
103    metadata:
104      labels:
105        app: kube-lego
106    spec:
107      serviceAccountName: lego
108      containers:
109      - name: kube-lego
110        image: jetstack/kube-lego:0.1.5
111        imagePullPolicy: Always
112        ports:
113        - containerPort: 8080
114        env:
115        - name: LEGO_EMAIL
116          valueFrom:
117            configMapKeyRef:
118              name: kube-lego
119              key: lego.email
120        - name: LEGO_URL
121          valueFrom:
122            configMapKeyRef:
123              name: kube-lego
124              key: lego.url
125        - name: LEGO_NAMESPACE
126          valueFrom:
127            fieldRef:
128              fieldPath: metadata.namespace
129        - name: LEGO_POD_IP
130          valueFrom:
131            fieldRef:
132              fieldPath: status.podIP
133        readinessProbe:
134          httpGet:
135            path: /healthz
136            port: 8080
137          initialDelaySeconds: 5
138          timeoutSeconds: 1

After applying these two changes the rest of my very simple config to deploy the Go service behind automatic TLS termination worked flawlessly. Since then managing the site has been an excellent experience with the power of kubectl, the Kubernetes “swiss army knife”.

In conclusion:

If my site were a serious production service instead of a toy learning experience I would seriously look towards GKE instead of a one node “cluster” running on a DIY “server” sitting by my desk at home, but setting up a toy cluster with kubeadm was a great experience for experimenting with Kubernetes. I can recommend using kubeadm for similar experiments, it’s quite simple to use once you have all the prerequesites installed and configured and the docs are quite good, however it won’t solve many of the things you’ll want for a production cluster.

You may also want to look around the list of the many CNCF certified Kubernetes conformant products for other options if for some reason neither of these sound appealing to you.

If you really just want to play with it first (and not host anything), check out minikube.


UPDATE: My site is on Netlify now, but I still run my own Kubernetes cluster to host other small projects. Hosting it on a toy Kubernetes cluster worked well, execept when the power went out at my apartment … I’d like my site to be online even then, hence Netlify :upside_down_face: