1) Go to prometheus graph
2) Select a metric
3) Now go to the graph view
This is showing you what will alert. If there are any lines over whatever time period you have set, these will alarm
Now, tighten the query rules so that it only shows you items in the graph when you really want it to notify you.
My alert is too chatty
1) Go to the
2) Expand the alarm that is too chatty
3) Click on the
expr. This is the query that is used to alarm.
4) Click on the graph view
5) Edit the expression so that it reduces what you dont need
kubernetes | prometheus | alert