Logging Infrastructure#
Collectors#
Filebeat#
Filebeat is an open source file harvester, mostly used to fetch logs files and feed them into the CLS Cribl installation via load balanaced IP where applicable.
We use filebeat running in containers on nodes to collect logs from all running pods. Various options can be used to configure the logging settings on a per pod/container basis. Using a custom annotation of platform-logging-splunkindex can be used to set the index in splunk that the log will be sent to. Also a custom annotation of platform-logging-sourcetype can be used to set the sourcetype that is sent to splunk. Additionally filebeat has additional built-in annotations that can be configured to support things like multi-line log formats.
Cloudwatch via S3 and SQS#
Many of the logs we need in the AWS environment are surfaced via AWS Cloudwatch. To centralize them into our central logging environment, we use a lambda configured in each of the projects ( NIS AWS| CCS AWS ) which runs periodically and dumps the logs to S3 buckets in the Platform Auth account (754180419460). In Platform Auth we have configured an SQS which watches the buckets and streams them to Cribl. The queue is created as part of a terraform module imported from CLS which also implements a role with which CLS' Cribl accesses the Bucket.
Post Collection#
Cribl#
Additionally, Cribl can be configured to watch an SQS url, allowing easy export of logs from Amazon using an SQS to consume Cloud Watch logging saved in S3 buckets by giving adequate permissions. More information on this from the logging team can be found here.
Splunk#
The ingested logs are transformed as necessary and passed to the SOC and Splunk for aggregation and analysis. Splunk has a broad query language that facilitates analysis. Splunk is capable of using the output of queries to generate alerts based on the data.
Links#
-
Documentation:
- General Documentation: https://www.elastic.co/guide/en/beats/filebeat/current/index.html
- Kubernetes Documentation: https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html
- Kubernetes Metadata Documentation: https://www.elastic.co/guide/en/beats/filebeat/7.17/add-kubernetes-metadata.html
- Built-in Annotations Documentation: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html
- CLS Cribl S3 SQS Git w/ Documentation: https://code.vt.edu/NIS/customer-support-tools/cls/cribl_aws_customer_infra
-
Helm:
-
Implementation:
-
Cluster Config: Bootstrap https://code.vt.edu/it-common-platform/infrastructure/eks-cluster/-/blob/main/cluster-bootstrap/filebeat.tf
-
Filebeat Config File: https://code.vt.edu/it-common-platform/infrastructure/eks-cluster/-/blob/main/cluster-bootstrap/files/filebeat/filebeat.yml
-
AWS lambdas - NIS: https://code.vt.edu/it-common-platform/infrastructure/infra/-/blob/master/cloudwatch-export-lambda.tf
-
AWS lambdas - CCS: https://code.vt.edu/it-common-platform/infrastructure/infra-ccs/-/blob/main/cloudwatch-export-lambda.tf
-
Platform Auth AWS Common Core: https://code.vt.edu/it-common-platform/infrastructure/meta-infra/-/blob/fa2dd0c6697c3784eaf36859b7ed33dd9177abf2/logs-buckets.tf
- Splunk Links:
- Main VT Site: https://splunk.it.vt.edu
- Splunk Company Site: https://splunk.com
-