Table of Contents
Kubernetes is Kubernetes—what difference does it make which cloud provider you choose? Well, quite a lot, actually. While GKE, EKS, AKS, and DOKS all conform to CNCF Kubernetes Certification standards and are each valid choices, they each have their unique advantages and disadvantages, ranging from pricing to upgrades, to node repair.
A managed Kubernetes offering means you don't have to worry about the certificates between nodes, setting up new servers, and whatever else comes with running your Kubernetes cluster. Running in the cloud also means you get support from a giant cloud vendor, rather than only the community itself, which can help you troubleshoot your production issues faster and reduce downtime.
In this article, we'll break down the "big four" cloud providers, giving you information on why you might want to choose one over the other for your specific use case.
Google Kubernetes Engine
GKE is the original cloud Kubernetes offering. Launched back in August 2015, only one month after the launch of Kubernetes itself, GKE provided an easy way for engineers to get started with this exciting orchestrator. Many consider Google's Borg as the unofficial predecessor to Kubernetes, so it's no wonder they were the first to make it a cloud offering. But how do they fare in 2021?
By spinning up a Kubernetes cluster in GKE, you get a few benefits you won't get from other providers. One such benefit is gVisor. GKE uses gVisor to create sandboxes for greater isolation between pods. Besides that, you also get the possibility of using a container-optimized OS, whereas other providers may only provide an Ubuntu image.
In the case that your cluster experiences failure, GKE has a few tools to help you out. You can enable auto-repair on your cluster, which will make GKE check your nodes continuously. If a node reports consecutive NotReady status reports, GKE will initiate a repair process. Typically this means the node is re-created. GKE also has your back when new versions of Kubernetes come out, as they have automatic upgrades enabled by default. Be sure to check whether this is a behavior you want, as in some cases it could break your application.
When using GKE, you pay an administration fee of $0.10 per hour per cluster, and otherwise, you only pay for the underlying resources. However, this administration fee can be waived by running GKE on-prem, with the help of Anthos.
Elastic Kubernetes Service
Coming out in June 2018, EKS launched quite a while after Google's offering. EKS is one of Amazon's many possibilities for running containers in the cloud, as they also offer an Elastic Container Service and Elastic Compute Cloud. The latter being their IaaS solution, which can also host containers.
Running a Kubernetes cluster in EKS, you get the possibility of using either a standard Ubuntu image as the OS for your nodes, or you can use their optimized EKS AMIs. This can help you get some better speed and performance rather than running a generic OS.
Once the cluster is running, there's no way to enable automatic upgrades of the Kubernetes version. While EKS does have excellent documentation on how to upgrade your cluster, it is a manual process.
If your nodes start reporting failures, EKS doesn't have a way of enabling auto-repair like in GKE. This means you'll have to either monitor that yourself and manually fix nodes or set up your own system to repair broken nodes.
As with GKE, you pay an administration fee of $0.10 per hour per cluster when running EKS, after which you only pay for the underlying resources. If you want to run your cluster on-prem, it's possible to do so either by using AWS Outposts or EKS Anywhere, which launches sometime in 2021.
Azure Kubernetes Service
Like EKS, AKS came out in June 2018. AKS is Azure's shot at a managed Kubernetes service, and it offers many of the same advantages as GKE and EKS. One big thing to note here is that AKS doesn't have any optimized node images you can use. The only images you can use as the base OS for your nodes are based on regular Linux distributions, like Ubuntu.
When your cluster is running, it's not upgraded automatically by default, but it is possible to enable. AKS allows you to put your cluster into four categories: none, patch, stable, and rapid. This gives you great control over exactly how you want your cluster to be upgraded.
Auto-repair can be enabled in AKS but is not default. Like GKE, it checks whether the node reports consecutive NotReady statuses. If AKS decides a node is unhealthy, it starts by trying to reboot the node. If this doesn't work, it reimages the node. The last course of action it takes is to delete the node and re-create it.
Unlike GKE and EKS, Azure doesn't require you to pay an administration fee for your clusters, meaning you only pay for the underlying resources. This can be a big selling point for small clusters.
Azure doesn't offer a solution specifically meant for running AKS on-prem, but their ARC solution allows you to run Azure resources on-prem in general and can be used for AKS.
DigitalOcean Kubernetes Service
DigitalOcean is the latest major cloud provider to offer a Kubernetes solution, coming out in May 2019. DigitalOcean continues with its philosophy of ensuring the products they do have are well-supported and stable, even if they're not the most feature-rich. This can be seen in the fact that they offer automatic upgrades, but only for patch versions. There's also no option for enabling automatic node repair.
As of writing, there's no documentation on how to run a DOKS cluster on-prem. Maybe this is something they will add in the future, as some customers are asking for it. Like AKS, they don't charge you for the cluster itself, only for the underlying resources. As with many other DigitalOcean products, this means you can get started relatively cheaply. The cheapest Kubernetes cluster can be set up for only $10 per month.
Comparing Pricing
When comparing any offerings, it's not just a matter of what feature-set it has. Pricing should also play a significant role in your decision. So let's look at how they compare and what tools are available to keep up with costs once your cluster is running.
These prices were calculated using the respective provider's pricing calculator, except DigitalOcean, which doesn't have a calculator. So instead, their pricing page was used and then manually calculated. The basis for the calculation is to have 20 nodes that together give you 80vCPUs and 320GB RAM.
GKE only gets you 300GB RAM since they vary a bit from standard sizes, and in this case, the node chosen has 15GB of RAM. These prices don't include the cost of traffic, as that is not specific to Kubernetes and is a function of how each cloud provider manages traffic. Also, GKE's pricing calculator automatically applied a sustained use discount.
| AKS | GKE | EKS | DOKS |
|---|---|---|---|
| $3,416.40 | $1,747.61 | $2,928.00 | $2,400.00 |
Prices are listed on a per month basis.
It's possible to bring these costs down in GKE, AKS, and EKS by committing to 1 or 3-year use. Note that the above table is excellent for getting a quick overview of how these providers compare. If you are looking into setting up your own cluster and want to know the cost, I recommend using their calculators to calculate your specific scenario.
Reducing cost
Once you have your cluster set up and you know what it costs, you may be interested in not just monitoring the cost over time but also how you can keep it as low as possible. To monitor your cost, you can use kubecost. This tool can help you continuously monitor your cost, giving insights that can help you and your team lower your spending.
A more automatic approach is to use Loft. This tool will install into your cluster, monitor your resources, and automatically put idle namespaces to sleep. You can also set it up to auto-delete idle namespaces. If your workload fluctuates highly, for example, based on seasonal traffic, this can significantly reduce your cost.
Conclusion
As you can see, not all Kubernetes providers are created equal. However, whichever offering you go with, you will get a stable cluster that conforms to CNCF's standards, so you should pick your cloud provider based on what advantage they can bring to your organization.
EKS is great if you're already deeply integrated with AWS. GKE is great at innovating around Kubernetes, adding support for new container runtimes, and optimized operating systems. AKS gives you a free cluster, access control can be controlled via your Azure AD, and it is currently the only major provider keeping up with the newest versions. Lastly, DigitalOcean is great for the same reasons you would otherwise choose DigitalOcean: they're a simple but excellent solution that can be easily set up at a great price.
In any case, all four providers offer free credits to new users, meaning you can sign up and test out which one fits your specific use case the best.
Photo by Ibrahim Boran on Unsplash
