Skip to content

Local Terraform Execution Guide#

Description#

This guide outlines the process for running Terraform locally instead of through the pipeline, which may be necessary in certain cases.

Prerequisites#

1. Install necessary software#

  • tfswitch

    • For older versions of Terraform or certain modules/providers, use the amd64 binary rather than the arm64 one.
    • Ensure your Terraform version matches the one used in the pipeline.
    • Projects should have a .terraform-version file containing the correct version.
    • Running tfswitch in a directory with this file will automatically switch to the correct version.

  • aws-cli

  • aws-iam-authenticator

  • kubectl (within one minor version of the cluster)

  • awsume

2. Connect to the VT VPN.#

3. Obtain a personal access token from code.vt.edu/-/profile/personal_access_tokens with read_api permissions only.#

4. Set the environment variable TF_VAR_gitlab_token equal to this token.#

5. Request admin access for the production cluster:#

6. Add your PID here and create a merge request.#

7. Add AWS STS client functions to your .bashrc/.zshrc:#

Procedure#

  1. Clone a new copy of the repository code where state is locked. You need a new copy of the code because we will be copying files around that we don’t want pushed upstream

  2. Check out the desired commit.

  3. Run vault-auth.sh / vault-auth.bash script in the root of the eks-cluster project.

  4. Run the export commands output by the vault-auth script:

    • export VAULT_TOKEN=.......
    • export VAULT_ADDR=.......

  5. Recursively copy the environment you're working with to the cluster-bootstrap directory: 

       cp -R environments/<environment>/* .

  6. For on-prem: Remove unnecessary backend definition files. For example, for the dvlp cluster: 

       rm pprd-backend.tf prod-backend.tf

  7. For AWS clusters:

    • Authenticate to the needed account (e.g., vt-platform-prod), and switch to that profile.

    • New way with baseline (AWS DVLP and PPRD)

      1. Log in: AWS Baseline Login
      2. Select the cluster you want to access
        • Click the "AWS Account" button
        • DVLP: VT-AWS-ACCT_50
        • PRD: VT-AWS-ACCT_143
      3. Click on "Command line or programmatic access"
      4. Use the "AWS IAM Identity Center credentials"

    • Run: aws_sts_set:

          aws_sts_set vtnis-ss 772415023108 NISAdmin

  8. For On-prem: Authenticate using Headlamp

  9. Run tfswitch to ensure you're using the correct version of Terraform.

  10. Initialize Terraform:

    • For AWS: 

      terraform init
terraform workspace list
terraform workspace select <workspace name>

    • For on-prem: 

      export TF_VAR_gitlab_token=<your gitlab token>
terraform init -backend-config=username="gitlab-ci-token" -backend-config=password="${TF_VAR_gitlab_token}"

  11. You can now run commands such as terraform show, terraform taint, and terraform force-unlock.

  12. If you need to run a terraform plan or apply, create a kubeconfig:

    • The kubeconfig is stored in the state file for eks
    • terraform init
    • terraform workspace list
    • terraform workspace select <workspace name>
    • Run terraform output kubeconfig to make sure the output is valid
    • Take the output from the last command and write it to a file called kubeconfig in the directory you need to run terraform from (eks-cluster/cluster-bootstrap)
    • Edit the file
      • Remove the header and footer - <<EOT and EOT
      • Change the apiVersion from v1alpha1 to v1beta1
    • The kubeconfig file is referenced in the provider

  13. Run Terraform plan:

    • Use the personal access token generated previously when prompted for var.gitlab_token. 
      terraform plan --var-file=<cluster name>.tfvars -out=plan.tfplan

      terraform apply -input=false plan.tfplan

Tips#

Show all helm releases:

terraform show | grep '# helm'

Cleanup#

  1. If you cloned the eks-cluster repository in a new location, delete it to avoid accidental commits.
  2. If granted admin access for the AWS production cluster, remove your PID from here and create a merge request.

Troubleshooting#

If the terraform state is locked, run:

  1. Run Terraform plan: 
       terraform force-unlock <Lock ID>

In the AWS vt-platform-auth account, there's an S3 vt-common-platform-env bucket that stores the tfstate for each cluster.

Additional Resources#

Note

Always ensure you're using the correct versions of tools and following the most up-to-date procedures when working with infrastructure as code.