GitHub Enterprise Server External Monitoring and Alerting with Prometheus and Grafana

[oskarpie]

Abstract

GitHub Enterprise Server includes a monitoring dashboard that helps administrators visualize system performance and detect anomalies. This is helpful for manually analyzing graphs at a glance, though may not be sufficient at scale. This article covers how to automate early detection and notifications by using exported metrics with Prometheus and Grafana.

Problem Statement

Proactive monitoring and alerting can prevent many critical incidents caused by resource overuse (disk, CPU, memory, network). Many GitHub Enterprise Server customers have asked in the past if they can install third-party monitoring agents inside the GitHub Enterprise Server to natively export metrics to their existing monitoring systems. While this is technically possible, it is not advisable due to the need of protecting the integrity of the GitHub Enterprise Server appliance.

Proposed Solution

Instead, of installing a monitoring agent inside of the GitHub Enterprise Server appliance we recommend setting up external monitoring in conjunction with the GitHub Enterprise Server Prometheus metrics exporter. This new feature simplifies the process of setting up a monitoring pipeline that integrates with an external Prometheus/Grafana installation and allows for building customer dashboards along with configuration of alerting.

Why Prometheus/Grafana?

• Prometheus Metrics Exporter: Released as a private beta in GHES 3.12 and general availability in GHES 3.16.
• Open-Source Community: Prometheus has a large and active community of developers.
• Industry Adoption: Widely used to monitor applications and infrastructure, both on-premise and in the cloud, with many integrations and exporters available.
• Powerful Data Model: Uses a time-series metrics data model stored in a database, enabling powerful queries and analyses.
• PromQL: A flexible query language that allows users to create and customize dashboards and alerts easily.
• Grafana Integration: Built-in support for Prometheus as a data source, requiring no additional development. PromQL is used to query metrics and build dashboards.
• Robust Alerting: Fully integrated alerting system that lets users set up alerts based on specific conditions or thresholds. Alerts can be sent via email, Slack, and other channels.
• Scalability: Prometheus scales easily ex. Thanos which handles failures and outages of individual components without disrupting service.
• Security: IP-based rule access to exposed metrics

Cons:

• Open Source Software: Requires technical expertise for deployment and configuration.
• Global Visibility: Needs additional components.
• Query Language Incompatibility: GHES monitoring dashboards use Graphite as data source and Graphite query language, while external Grafana use Prometheus as data source and PromQL for querying, necessitating translation of dashboards.

How to achieve it

1. On your GHES 3.16+ instance enable Collectd-Prometheus Exporter

Enable Prometheus metrics exporter and scraping metrics via three available methods:

• Management Console
• Command line interface
• REST API call

💡 Note: On GHES 3.12 - 3.15 you can only use CLI and there is no trusted IP option:

ghe-config app.prometheus.enabled true
ghe-config-apply

To confirm that the plugin is exporting metrics, query the endpoint on port 9103 with curl

curl localhost:9103/metrics

You should see metrics that start with collectd_ prefix

💡 NOTE: On your instance the local firewall is already open on port 9103.

2. On your external Prometheus/Grafana monitoring system

Configure Prometheus server to scrape newly created GitHub Enterprise Server exported metrics endpoint

In your prometheus.yaml configuration file add a scraping section for the GHES metrics scraping

scrape_configs:
  #  Collectd
  - job_name: "collectd-prom-agent"
    scrape_interval: 10s
    static_configs:
      - targets: ["http(s)://HOSTNAME:9103/metrics"]

💡 NOTE: You might have to restart your Prometheus service

Build simplified GitHub Enterprise Server dashboard

An Example of a simple GitHub Enterprise Server dashboard built in Grafana 11

• Configure and test Grafana alerting

The latest version of Grafana ships with built-in alerting with a wide variety of contact points to choose from. You can use this functionality to pro-actively monitor hardware resources such as disk, cpu or memory reaching pre-defined thresholds.

Summary

Hopefully this article provided enough information to get you started with configuring proactive monitoring and alerting for GitHub Enterprise Server (GHES) using Prometheus and Grafana software.

[senaraya]

@oskarpie How did you get the 'User Data DIsk Space' usage ? collectd doesnt seem to export it. I have added the below config in collectd.conf though

LoadPlugin df

IgnoreSelected true
MountPoint "/^/tmp/hooks-/"
MountPoint "/^/data/user/docker/overlay2/"
MountPoint "/^/run/"
MountPoint "/^/data/user/nomad/alloc/"
MountPoint "/^/data/user/"

LoadPlugin disk
<Plugin "disk">
Disk "/data/user"