Add alerts and dashboards for TSA

[haydentherapper]

Description

[jku]

This is the current state:

SLO alerts

terraform-modules: gcp/modules/monitoring/timestamp/slo.tf

• SLO Fast Burn: Timestamp Authority 99.5% Availability (Prober) : /api/v1/timestamp/certchain - GET
• SLO Medium Burn: Timestamp Authority 99.5% Availability (Prober) : /api/v1/timestamp/certchain - GET
• SLO Slow Burn: Timestamp Authority 99.5% Availability (Prober) : /api/v1/timestamp/certchain - GET
• SLO Fast Burn: Timestamp Authority 99.5% Availability (Server) : /api/v1/timestamp - POST
• SLO Medium Burn: Timestamp Authority 99.5% Availability (Server) : /api/v1/timestamp - POST
• SLO Slow Burn: Timestamp Authority 99.5% Availability (Server) : /api/v1/timestamp - POST

Other alerts:

terraform-modules: gcp/modules/monitoring/timestamp/timestamp_alerts.tf

• Signing Cert Expiration
• Timestamp Authority K8s Unschedulable
• Timestamp Authority K8s Restart Failing Container
• Timestamp Authority uptime alert

terraform-modules: gcp/modules/monitoring/prober/prober_alerts.tf

• [Disabled] API Prober: Timestamp API Endpoint Latency > 750 ms for 5 minutes
  (the other latency alerts are disabled as well)

Summary

I've had a look at each of the alerts and they seem sensible so I don't plan to do anything WRT alerts at this point -- I don't know the story on the API probers but it they are disabled for fulcio and rekor as well.

I'll look at what dashboards might make sense next.

[jku]

Signing Cert Expiration

I might rename this -- it's a little annoying that it does not mention "timestamp" anywhere

[jku]

Dashboards

I've never truly been on-call anywhere so will take suggestions for a useful dashboard... but I've drafted these three widgets now, plan to try to translate these into terraform :

I did not add any incident/alert widgets, can add if that's seen as useful.

[jku]

sigstore/terraform-modules#89