Kubernetes: Overkill or Overlooked?

Do you really need it?

So if you listened to my last Engineering Evolved podcast then you’d know that the general answer is no. But obviously it is a little more neuanced than that.

Let me direct you to a Linkedin Post that caught my eye the other day because I thought it was a joke, then it became apparant in the comments that he wasn’t joking.

Now the argument that you should use “more modern tooling” because its hot is one that’s been raging since the dawn of time. But of course, why change what works? And if a VM and SSH works for you, and you have a small team, have at it!

So this then leads me on to Kubernetes, what is it and should you use it. Kubernetes span out of Google where their Borg platform was dealing with massive Google sized deployments on a daily basis. Now if that doesn’t tell you what Kubernetes is good at nothing will. But for those of us who like to hack around on things, Kubernetes basically takes Docker images and a lot of YAML descriptors to deploy your application, or applications into a cluster of machines that then mangages your container, your networking, ingress, egress, permissions etc for you. Sounds great right?

Well in a number of cases it is great. It can provide a lot of fault tollerance, auto scaling, auto healing, all the things your business may be interested in leveraging and some of that, but not all of it comes out of the box.

Downsides

Okay so what are the downsides of Kubernetes? Lets start on a downer and finish more positively. Its super complex. Sure every cloud vendor has their own hosted solution, and there are a number of vendors offering up support for their Kubernetes variant if you’d rather deploy it elsewhere, but that doesn’t mean its easy to wrap your head around. Azure attempt to use Azure App Service to wrap Kubernetes in something that makes it easier to deploy apps. AWS just launched AKS Auto Mode in an attempt to reduce the management burden. They all know its hard work. Cluster security, cluster upgrades etc etc they all take planning and forethought on top of managing your own pods. And if you’re self hosting them your host VM also needs patching, upgrades and remediations.

On top of this the majority of Kubernetes is driven by YAML in some form or other. Some times this is simple, often its not, but its also the way you manage it, how do you deal with change requests, version control, config drift, different people applying policies etc? All of this requires careful management and its not really a technical problem its more of a policy and people problem, do you have the bandwidth to manage it?

At smaller scales Kubernetes is just more expensive. This obviously manifests itself in a number of ways and isn’t just 1 number. For example the additional complexity means more man hours. But it also means you’ve got more nodes, a control plane (sure it may be “free” in a vendor but you’re paying for it somewhere), more infrastructure for ingress etc. It all adds up.

Slower developer loops are also a real issue. You can, if you’re a sadist, run Kubernetes on a local machine. But this still requires the same YAML setup, the same config tweaking, the same maintenance. It can be great to debug issues in the cluster but in reality its still hard to manage and certainly not developer friendly.

Lastly on the downside, the mirage of Kubernetes portability. Of course Kubernetes is designed somewhat to offer universal configuration across cloud providers, this often doesn’t really play out in practice. You still need to have a good understanding of your vendors networking, its ingress options, its available storage etc. When you deploy storage, do you need slow disks? Cheap disks? High performance disks? Maybe both, how do you pick them how do you configure your setup to use them? It all adds to the operational complexity of a Kubernetes deployment.

Upsides

Portability is obviously real. We can deploy docker containers into almost any environment, on prem, in any cloud environment. And whilst there is a degree of needing to understand the environment and what it provides the core product itself probably doesn’t need any changes making it easily deployable in a range of different environments. Of course the other thing about docker containers, developers can also just run them using Docker Desktop, Podman etc making them runnable across an number of different control services.

The other part of deploy Kubernetes processes is the declarative infrastructure. Whilst having everything defined as a YAML file can certainly add more control complexity it also is a positive. Once you’ve described the state you want, Kubernetes will do its best to keep everything in sync. Self healing, rollbacks on failure, consistency across different environments. That all comes from the bit that also makes Kubernetes hard to manage from a process perspective.

Of course in reality there is a rich ecosystem of additional services, products and people who can all plug into your Kubernetes infrastructure. Products like Argo can provide excellent application management. Prometheus provides out of the box support for monitoring of your processes and so on. There is an endless amount of additional tooling that can both help and hinder depending on what you’re trying to achieve.

Resource Isolation provides an element of multi tenancy or security patterns across your resources. Obviously this could be achieved outside of Kubernetes as well but RBAC roles, network boundaries etc all provided out of the box. Now of course, these can be hard to configure and maintain, also you want to hook them into your main security systems, probably easier to do inside of a cloud vendors system but still obscure, ensuring you’ve got namespaces scoped properly, maintaining that network firewall at the correct place etc. This is what I mean by the man management of the change requests, it only takes one badly changed YAML file to blow that right open, but when it works well, having all your processes on a single scalable cluster can really make managing Kubernetes clusters a breeze.

Next on the list of positives is the extensibility. If it can’t do what you want it to the operator framework, the ability to extend and improve the environment is absolutely possible. The APIs have been standardised for a long time and this is what drives that ecosystem we mentioned earlier.

Deployment patterns can really help you, of course some of these need additional tooling if you’re going to do them well but things like Rolling Updates, Canary and Blue Green releases are all absolutely doable out of the box. Of course though what you need to ask is, do you really need Blue Green releases? Or did you just read about them and figured they sounded cool?

Last on the list of positives is observability hooks. One of the biggest problems with running anything inside a container is, “whats going on inside?”. Hooks for logs, metrics, traces can really help here, both inside your containers but also the traffic through and across the platform. The ecosystem across the observability sector has really grown over the past decade, standards like OpenZipkin(founded by my friend from back in the JClouds days Adrian Cole) have really aided this drive.

So how do I choose?

Now this is the million dollar(in some cases actually million dollar) question. What is it you’re trying to achieve? Can you achieve the same outcomes by doing less? If the answer to this question is “yes”, then do less. I honestly believe in the modern tech landscape if you can do something by spinning up a VM and running a process on there, even if that thing sits inside a docker container for convenience, then it probably should. Of course there are times that doesn’t hold true. But even if its not true, do you still need a Kubernetes cluster? Or will something like Lambda for backend, ECS for frontend, Google Cloud Run, or Azure App Services work just as well?

Ponding life choices whilst looking at servers

In the 21st century there are multiple ways to spin this cat. Even if you’re looking for portability, would it cost more to package your app as a Docker container and then write separate deployment scripts for each cloud providers bog standard application deployment?

These are the types of questions you need to ask, not just the cost in running the thing, people think instantly to hosting costs, but when you’ve got a platform as complex as Kubernetes, whats the human cost, both monetarily and in terms of raw time.

Kubernetes is a powerful beast, in the right places it makes a lot of sense. But in the wrong places it can be a security, maintenance and management disaster waiting to happen.