Skip to content

EKS-Anywhere#

For On-Premisis, the Platform Team chose to go with EKS Anywhere, an Amazon-created, Cluster-API-centric opinionated cluster that comes with full Kubernetes support. In addition to suport and a reasonable set of opinionated defaults, it is designed with Bottlerocket in mind, a custom hardened appliance distro for hosting containers.

The On-Premisis implemntation of EKS utilizes a small vSphere cluster connected to the central NISSO VCenter. The the tools shipped with EKS Anywhere (eksctl anywhere) are able to ue the vCenter API to fully manage the cluster lifecycle.

Pre-cluster Environment#

The on-prem pre-cluster environment primarily consists of two hosts in the main vCenter cluster, maintained by NISSO with Platform Team acting as a tenant/app admin.

  • plat-dhcp.cc.vt.edu - Provides DHCP for our subnets - TEMPORARY WORKAROUND FOR NETWORKING LIMITATIONS
  • plat-gitlab.cc.vt.edu - Gitlab Runner for pre-cluster and life-cycle management

Manual Stages#

DHCP#

The apps server hosting DHCP does so via docker-compose. This is a temporary host until networking figures out how to host DHCP on our subnets. Code/Instructions for this hose can be found here.

Gitlab Runner#

The gitlab runner has some minimal manual steps to become functional if the host is wiped. These primarily consist of prep tasks for the runner for resources not governed by the OS package manager and some additional tasks like etcd backups. Instructions can be found in the main EKS Anywhere Vsphere project.

Code and Runner#

The EKSA Vsphere runner runs out of its own project and is composed of gitlabci logic and shell-centric commands to support eksctl which leverages cluster-api and some custom lifecycle code to create the clusters and their dependencies. To facilitate inventory management between the cluster environments, we leverage a helmchart inside the project to create the manifests that the eksctl consumes.

  • Implementation:

    • Kubernetes Cluster: Main cluster implementation

    https://code.vt.edu/it-common-platform/infrastructure/eks-cluster

  • Upgrade Concerns:

    • Plugins: Although Terraform is generally compatible between versions, the underlying plug-ins are likely not and will have various requirements, dependencies, and changes in default behavior.

EKS Anywhere Limitations#

The primary limitation we find working with EKS Anywhere is that it is not nearly extensive as most of the Kubernetes technologies we have used previous. EKSA is a young project. As a life-cycle management tool written by AWS resting on a few other tools, it is often the case that we find ourselves wanting or needing something that is not yet supported, won't be supported, or will require us to home-brew the default Bottlerocket OS images the cluster uses.

EKS Anywhere Kubernetes Upgrades#

One of the primary advantages of running EKS Anywhere is the lifecycle management. Although the usual caveats for upgrading a cluster are present. The actual work for updating the cluster is done simply by incrementing a config and running the appropriate tool.

EKS Anywhere Updates#

EKS Anywhere has its own updates independent to Kubernetes. EKS Anywhere is primarily a life-cycle management tool, so each of these usually support a wide range of Kubernetes versions, allowing teams to update the EKSA version independently. These are conveniently governed by a config and applied after mitigating any caveats by running the pipeline.

EKS Anywhere Documentation Version

There is a versioning drop down on the top right of the site. To date, it has not had any but the current version.

EKS Anywhere Update Concerns#

  • Any Interruption of the CNI will cause an outage - which is a regular part of EKS Anywhere Updates
  • KIND Cluster - EKSA Upgrades spawn a KIND Cluster that various piece of the current cluster are trasnferred to during EKS-A upgrades. Any issues may result in a situation where the temporary KIND cluster is still in-place and recovery is needed.
  • EKS Anywhere is a new project and moves quickly. Looking over documentation is important to see if there are any changes.