Create Observability Stack for Monitoring and Logging

[MooseQuest]

Generating the observability stack serves the following purposes:

• Allows for monitoring of both the cluster and application.
• Identify resource contention and scale issues through metrics
• Allows developers to pinpoint errors and surface them for ops alerts

Components to generate:

• The data pipeline which will deliver logs from the environment and application
• The metrics visualization
• Data management layer either on the cluster, or provide the connectivity to another component which will surface.

Technologies and software to consider:

• Elasticsearch
• Grafana
• Prometheus
• Splunk

[lottspot]

Going to try out this pre-rolled stack as a starting point: https://github.com/coreos/kube-prometheus

Even if all goes well, this doesn't get us a logging stack; just metrics and monitoring.

[lottspot]

Roll out went well and we have metrics dashboards running at https://metrics.chime-live-cluster.phl.io/

The manifests used for the rollout are currently sitting in the issues/3 branch, where they will remain until the freeze on PR to masters is lifted.

[rcknplyr]

@lottspot would we consider this completed?

[lottspot]

We don't have anything capturing logs yet so this is technically not completed

[MooseQuest]

I'll be pushing up what we have so far onto a branch and will reference here.