Sounds like you need an alert/monitoring system and not a logging system. Something like nagios where you immediately get an alert if something is past its limits, and where you don't have to rely on logging.

I would add Alertmanager to your stack if you haven't already. It's pretty tightly integrated with prometheus. There's some canned alerting rules based on predicting disk space full in X number of days. We wire Alertmanager to Pagerduty.

The amount of clients that are missing basic events like "you've run out of disk space

For my personal servers, I use Netdata for this. Works pretty well.