Prometheus Stack

License: Apache 2.0 GPL 3

We use Prometheus, Alertmanager, and Grafana to do metric collection and alerting.

Prometheus is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts when specified conditions are observed.

Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations such as email, PagerDuty, OpsGenie, or many other mechanisms thanks to the webhook receiver. It also takes care of silencing and inhibition of alerts.

Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.

We deploy all of this using the helm chart kube-prometheus-stack.

The kube-prometheus-stack helm chart installs the kube-prometheus stack, a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

Data collected by prometheus is viewable in grafana and alertmanager uses prometheus data to send alerts to opsgenie and slack.

Links

• Prometheus Code: https://github.com/prometheus/prometheus
• Alertmanager Code: https://github.com/prometheus/alertmanager
• Grafana Code: https://github.com/grafana/grafana

• kube-prometheus-stack Code: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

• Prometheus Releases: https://github.com/prometheus/prometheus/releases

• Alertmanager Releases: https://github.com/prometheus/alertmanager/releases
• Grafana Releases: https://github.com/grafana/grafana/releases

• kube-prometheus-stack Releases: https://github.com/prometheus-community/helm-charts/releases?q=kube-prometheus-stack

• Documentation:

  • Prometheus Documentation: https://aquasecurity.github.io/trivy-operator/
  • Alertmanager Documentation: https://aquasecurity.github.io/trivy-operator/latest/getting-started/installation/upgrade/
  • Grafana Documentation: https://grafana.com/docs/
  • kube-prometheus-stack Documentation: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/README.md

• Helm:

  • Chart: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

• Implementation:

  • Install Stack

  https://code.vt.edu/it-common-platform/infrastructure/eks-cluster/-/blob/main/cluster-bootstrap/utils/monitoring/prometheus.tf

Update Concerns

• While not every release of the kube-prometheus-stack helm chart has CRD upgrade, when it does, they have to be updated manually.
• There are a large number of other componants that depend on prometheus for monitoring, and may fail to install if prometheus isn't installed correctly.