Skip to content

GitLab#

Our Platform Team heavily integrates Infrastructure as Code (IaC) into our workflow, with all code repositories managed through GitLab. This centralized approach via GitLab ensures a streamlined, version-controlled, and collaborative environment for code development, deployment, and management.

Infrastructure as Code in GitLab#

We use Terraform as our infrastructure as code (IaC) tool for provisioning and managing our cloud (AWS) and on-prem resources. We use GitLab's CI/CD pipelines to automate the execution of Terraform commands such as init, plan, and apply

How we store and access Terraform state in GitLab#

The Platform Team leverages GitLab's HTTP backend for centralized Terraform state storage and management. As exemplified in the configuration snippet below, our state files are securely stored on our GitLab instance (code.vt.edu). The state for our eksapprd environment, for instance, resides in project 8221. Terraform operations like state locking and unlocking are managed through designated API endpoints, ensuring concurrency control. As an example, you can view the state configuration for our eksapprd project at this link or in the code snippet below: https://code.vt.edu/it-common-platform/infrastructure/eks-cluster/-/blob/main/cluster-bootstrap/environments/eksa/pprd-backend.tf.


terraform {
  backend "http" {
    address        = "https://code.vt.edu/api/v4/projects/8221/terraform/state/eksapprd"
    lock_address   = "https://code.vt.edu/api/v4/projects/8221/terraform/state/eksapprd/lock"
    unlock_address = "https://code.vt.edu/api/v4/projects/8221/terraform/state/eksapprd/lock"
    lock_method    = "POST"
    unlock_method  = "DELETE"
    retry_wait_min = 5
  }
}

Platform Team's Use of Artifacts in GitLab CI/CD#

In our continuous integration and deployment workflows, the Platform Team harnesses the power of GitLab CI/CD's artifact management. As showcased in the provided example from our .gitlab-ci.yml for the eks-cluster https://code.vt.edu/it-common-platform/infrastructure/eks-cluster/-/blob/main/.gitlab-ci.yml, after certain operations are executed, such as selecting a Terraform workspace and extracting the EKS cluster ID, we save specific environment variables as artifacts. These artifacts are named based on the commit reference slug ($CI_COMMIT_REF_SLUG) and contain important data like CLUSTER_ID. The saved artifact, specifically the dotenv report, is stored at a path like ./eks/dvlp-$CI_COMMIT_SHORT_SHA-deploy.env. This approach ensures that crucial data derived from CI processes are retained for subsequent stages or jobs in the pipeline. Notably, the artifacts are transient and expire after a specified duration, ensuring our CI storage remains optimized without unnecessary clutter.


.eks_kubeconfig_update:
  ...
  artifacts:
    name: $CI_COMMIT_REF_SLUG
    reports:
      dotenv: ./eks/dvlp-$CI_COMMIT_SHORT_SHA-deploy.env
  resource_group: dvlp

Platform Team's Container Image Management#

We employ a comprehensive approach to container image management. Consider this example from our Infrastructure project. In the .gitlab-ci.yml file located at code.vt.edu/it-common-platform/infrastructure/eks-cluster/-/blob/main/.gitlab-ci.yml, we set a default image from our internal GitLab container registry using the address code.vt.edu:5005/it-common-platform/support/platform-infra-ci-image:latest. This setup ensures that all CI jobs utilize this specific container image unless another is indicated. In our container registries on Gitlab, for example code.vt.edu/groups/it-common-platform/-/container_registries/212, we keep detailed records of each image, capturing attributes like size, publication date, and digest. Such tracking is invaluable for both traceability and ensuring the security and integrity of our images. Furthermore, our use of the image proxy, highlighted by the registry's digest and configuration digest, offers more precise image referencing, bolstering the safety and efficiency of our operations.


default:
  image: code.vt.edu:5005/it-common-platform/support/platform-infra-ci-image:latest
...

Platform Team CI/CD and the templates#

At the Platform Team, we've adopted a systematic and automated approach to our CI/CD processes, harnessing templates to maintain uniformity across workflows. By segmenting our pipelines into stages such as "build", "deploy", and "scan", we ensure our container images undergo a thorough lifecycle from creation to vulnerability scanning. Our secret sauce lies in the it-common-platform/tenant-support/ci-templates project, which provides us with predefined CI/CD blueprints. Templates like docker-build.yml, repo-update.yml, and trivy-scan.yml serve as the backbone of our CI/CD components, allowing individual jobs to extend these modules seamlessly. For instance, the .docker-build template streamlines our build process, while the .add-files-to-repo template simplifies our manifest deployments. Not stopping there, we've also integrated Helm charts into our CI/CD routine. With Helm, we ensure that chart versions are consistently updated in tandem with code changes and that these charts meet the gold standard in Kubernetes packaging.


Build image:
  stage: build
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  extends: .docker-build

EKS Cluster Configuration in GitLab: https://code.vt.edu/it-common-platform/infrastructure/eks-cluster/-/tree/main/eks:

Our EKS cluster configurations, stored in the eks directory of our GitLab repository, embody the principles of IaC. These files hold the blueprints of our Kubernetes cluster on AWS, enabling automated, consistent, and efficient cluster provisioning, management, and scaling.

Cluster Bootstrapping with IaC#

Post the creation of our EKS cluster, a critical next step is the process of bootstrapping - setting up essential configurations, workloads, and integrations to ensure the cluster is production-ready. This process is deeply embedded in our Infrastructure as Code (IaC) methodology.

Cluster Bootstrap Configuration in GitLab: https://code.vt.edu/it-common-platform/infrastructure/eks-cluster/-/tree/main/cluster-bootstrap

Housed in the cluster-bootstrap directory of our GitLab repository, these Terraform configurations encapsulate all the requisite tasks for initializing our newly minted EKS cluster. By defining our bootstrapping processes as code, we can guarantee a standard, repeatable, and automated setup across our clusters.

Key elements within this directory include:

  • Utilities & Workloads: gatekeeper.tf and others: These configurations deploy vital workloads and utilities on the cluster, from security constraints with Gatekeeper to authentication via Kubernetes OIDC and VT Middleware.

  • Environment-specific Setups: Directories such as aws, onprem, and eksa contain configurations tailored for specific environments, ensuring that our bootstrapping process adapts based on where our cluster resides.

  • Remote State Consumption: The bootstrapping configurations are designed to consume output variables from the cluster's Terraform remote state. This seamless integration ensures that the bootstrapping process is informed by the actual state of our EKS cluster, promoting consistency and reducing potential configuration drift.

By translating our cluster bootstrapping process into Terraform configurations, we've established a robust, repeatable, and auditable method to initialize our EKS clusters, reinforcing our commitment to infrastructure automation and best practices.

Note: Always refer to official GitLab documentation for the latest information - Documentation: https://docs.gitlab.com