> The truth, of course, is somewhere is the middle.
I agree, as I have stated in other comments, cause based signals should not alert, they should file a ticket, or be displayed along side a symptom based alert if there is a correlation. You should be monitoring as much as possible so you can observe your service in realtime.
> Symptom-based alerting is extremely tricky to get right in the first place.
Depends on the service. If you operating a HTTP API, a simple SLI would just be the error ratio of 500s to total requests. Its not perfect but it is much better than alerting based on CPU percentage or ram usage for a particular machine. Things can get more complex if your API is k8s style or you are operating a data plane that is serving customer traffic.
> What is a “within budget” for a service, can be total outage for a customer.
Now you are getting into SLO territory, which is similar to, but separate from symptom based alerting. Yes, defining good SLOs is very hard and varies greatly by the type of service you are running
> another situation is when problems in background jobs will cause massive outages later.
How do you know what signals to alert with? If you knew what causes/signals to alert, then wouldn't you design your backend jobs to not do the things that would cause an outage? Hindsight is always 20/20. If you are worried that a caused based signal from your backend _could_ be telling you that there _might_ be a problem in the future but your customers are not seeing errors, it is not business critical and you can just file a ticket and someone can look into it later.
I agree, as I have stated in other comments, cause based signals should not alert, they should file a ticket, or be displayed along side a symptom based alert if there is a correlation. You should be monitoring as much as possible so you can observe your service in realtime.
> Symptom-based alerting is extremely tricky to get right in the first place.
Depends on the service. If you operating a HTTP API, a simple SLI would just be the error ratio of 500s to total requests. Its not perfect but it is much better than alerting based on CPU percentage or ram usage for a particular machine. Things can get more complex if your API is k8s style or you are operating a data plane that is serving customer traffic.
> What is a “within budget” for a service, can be total outage for a customer.
Now you are getting into SLO territory, which is similar to, but separate from symptom based alerting. Yes, defining good SLOs is very hard and varies greatly by the type of service you are running
> another situation is when problems in background jobs will cause massive outages later.
How do you know what signals to alert with? If you knew what causes/signals to alert, then wouldn't you design your backend jobs to not do the things that would cause an outage? Hindsight is always 20/20. If you are worried that a caused based signal from your backend _could_ be telling you that there _might_ be a problem in the future but your customers are not seeing errors, it is not business critical and you can just file a ticket and someone can look into it later.