Comments on: The Wisdom of Specificity in Monitoring and Alerting http://thenubbyadmin.com/2010/04/30/the-wisdom-of-specificity-in-monitoring-and-alerting/ Just a nublet SysAdmin Sat, 28 Jan 2012 05:18:15 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Wesley.Nonapeptide http://thenubbyadmin.com/2010/04/30/the-wisdom-of-specificity-in-monitoring-and-alerting/comment-page-1/#comment-11 Wesley.Nonapeptide Sun, 02 May 2010 00:13:50 +0000 http://www.thenubbyadmin.com/?p=41#comment-11 @Jason - Good idea about focusing on the collection and less on the interpretation. Don't worry about pitching a product even if it's your own. I'm all for people talking about what they believe in even if they also get paid from it. I like to think of my blog as a free-for-all. If anyone wants to say anything they're welcome to (except if it involves R0l3X watches), but just be sure to back it up and be ready to get some tough questions from other commentors (or myself). =) @Jason – Good idea about focusing on the collection and less on the interpretation. Don’t worry about pitching a product even if it’s your own. I’m all for people talking about what they believe in even if they also get paid from it.

I like to think of my blog as a free-for-all. If anyone wants to say anything they’re welcome to (except if it involves R0l3X watches), but just be sure to back it up and be ready to get some tough questions from other commentors (or myself). =)

]]> By: Jason Dixon http://thenubbyadmin.com/2010/04/30/the-wisdom-of-specificity-in-monitoring-and-alerting/comment-page-1/#comment-10 Jason Dixon Sat, 01 May 2010 14:16:07 +0000 http://www.thenubbyadmin.com/?p=41#comment-10 We're attempting to address this chasm in thought with Circonus (http://circonus.com/). The emphasis should be on metric collection, not some interpretation of what they mean (that is best left to the user). Enable the user to correlate any disparate metrics for trending and root cause analysis... it's a very powerful method (and tool) that to date has been vastly underutilized. P.S. Sorry for the sales-pitch-esque answer, but we're passionate about this stuff. :) We’re attempting to address this chasm in thought with Circonus (http://circonus.com/). The emphasis should be on metric collection, not some interpretation of what they mean (that is best left to the user). Enable the user to correlate any disparate metrics for trending and root cause analysis… it’s a very powerful method (and tool) that to date has been vastly underutilized.

P.S. Sorry for the sales-pitch-esque answer, but we’re passionate about this stuff. :)

]]> By: Wesley.Nonapeptide http://thenubbyadmin.com/2010/04/30/the-wisdom-of-specificity-in-monitoring-and-alerting/comment-page-1/#comment-8 Wesley.Nonapeptide Fri, 30 Apr 2010 21:33:46 +0000 http://www.thenubbyadmin.com/?p=41#comment-8 @Doug: "the key when monitoring is to look at it from the perspective of the service offered to end user" Quoted for truth. What tools do you typically use for that kind of service monitoring? I think the key to Systems Administration / Engineering in general is to look at everything from the perspective of the service being offered to the user. @Doug: “the key when monitoring is to look at it from the perspective of the service offered to end user”

Quoted for truth. What tools do you typically use for that kind of service monitoring?

I think the key to Systems Administration / Engineering in general is to look at everything from the perspective of the service being offered to the user.

]]> By: Doug Luxem http://thenubbyadmin.com/2010/04/30/the-wisdom-of-specificity-in-monitoring-and-alerting/comment-page-1/#comment-7 Doug Luxem Fri, 30 Apr 2010 21:22:53 +0000 http://www.thenubbyadmin.com/?p=41#comment-7 Reiterating what Jason said - the key when monitoring is to look at it from the perspective of the service offered to end user, not the individual components. You could monitor the HTTP port, DNS queries, the Postgre back-end; or instead, have a monitoring platform that can enter a username and password on your login page (or whatever) and make sure it gets a valid response back. The other benefit is that you can monitor and alert on response time of the total system whereas the ICMP round trip is not really useful. Regarding switching web servers via DNS - that sounds like typical hosting extortion :) Reiterating what Jason said – the key when monitoring is to look at it from the perspective of the service offered to end user, not the individual components. You could monitor the HTTP port, DNS queries, the Postgre back-end; or instead, have a monitoring platform that can enter a username and password on your login page (or whatever) and make sure it gets a valid response back.

The other benefit is that you can monitor and alert on response time of the total system whereas the ICMP round trip is not really useful.

Regarding switching web servers via DNS – that sounds like typical hosting extortion :)

]]> By: Wesley.Nonapeptide http://thenubbyadmin.com/2010/04/30/the-wisdom-of-specificity-in-monitoring-and-alerting/comment-page-1/#comment-6 Wesley.Nonapeptide Fri, 30 Apr 2010 20:42:52 +0000 http://www.thenubbyadmin.com/?p=41#comment-6 @Matt: I sympathize concerning the assumption that if I'm not getting alerts I'm okay. I have a backup program that is supposed to email me upon failure. It's poorly implemented and sometimes that email doesn't reach me. For those situations, I create recurring tasks in Outlook to check on various systems. It seems to defeat the purpose of automation if you have to manually check it... but such is life as long as the second law of thermodynamics is in effect. @Jason: Sorry your comment got moderated. Wordpress captures comments that have more than one URL in it. I'll ease up on that. Moving on, you're exactly right about focusing on purpose over availability as well as needing to have insight into the unexpected. It was totally unexpected that my web host would switch the resolution of my domain to one of their other servers upon account suspension. I would have assumed that all resolution for my domain would have been suspended. Doesn't that seem odd to anyone else? @Matt: I sympathize concerning the assumption that if I’m not getting alerts I’m okay. I have a backup program that is supposed to email me upon failure. It’s poorly implemented and sometimes that email doesn’t reach me. For those situations, I create recurring tasks in Outlook to check on various systems. It seems to defeat the purpose of automation if you have to manually check it… but such is life as long as the second law of thermodynamics is in effect.

@Jason: Sorry your comment got moderated. WordPress captures comments that have more than one URL in it. I’ll ease up on that.

Moving on, you’re exactly right about focusing on purpose over availability as well as needing to have insight into the unexpected. It was totally unexpected that my web host would switch the resolution of my domain to one of their other servers upon account suspension. I would have assumed that all resolution for my domain would have been suspended. Doesn’t that seem odd to anyone else?

]]> By: Jason Dixon http://thenubbyadmin.com/2010/04/30/the-wisdom-of-specificity-in-monitoring-and-alerting/comment-page-1/#comment-4 Jason Dixon Fri, 30 Apr 2010 17:50:19 +0000 http://www.thenubbyadmin.com/?p=41#comment-4 The problem with "uptime monitoring" and monitoring in general is that we've been trained to focus on availability, rather than purpose. It's not easy to make good monitors; it takes knowledge of the application, insight into the unexpected and a commitment to perfection (excuse the cheese). Anyhoo, I've written on this issue a couple times already. http://obfuscurity.com/2009/09/Business-Metrics http://omniti.com/seeds/business-metrics-too The problem with “uptime monitoring” and monitoring in general is that we’ve been trained to focus on availability, rather than purpose. It’s not easy to make good monitors; it takes knowledge of the application, insight into the unexpected and a commitment to perfection (excuse the cheese).

Anyhoo, I’ve written on this issue a couple times already.

http://obfuscurity.com/2009/09/Business-Metrics
http://omniti.com/seeds/business-metrics-too

]]> By: Matt Simmons http://thenubbyadmin.com/2010/04/30/the-wisdom-of-specificity-in-monitoring-and-alerting/comment-page-1/#comment-2 Matt Simmons Fri, 30 Apr 2010 17:13:58 +0000 http://www.thenubbyadmin.com/?p=41#comment-2 Oh man....me and assumptions...we go way back :-) DNS is one of those really twitchy things, too, because one small error has untold ramifications. A large problem like the one you experienced wreaks havoc. My biggest assumption when monitoring is always that if I don't get alerts, then things are fine. Before I had two Nagios servers watching each other's backs, if Nagios goes down, how do you know? For that matter, even if you're monitoring it, if your only network connection goes down, how do you find out? Yeah, I've got a loooong history of bad assumptions. I'd assume that I'm getting better, but that's just part of the problem ;-) Oh man….me and assumptions…we go way back :-)

DNS is one of those really twitchy things, too, because one small error has untold ramifications. A large problem like the one you experienced wreaks havoc.

My biggest assumption when monitoring is always that if I don’t get alerts, then things are fine. Before I had two Nagios servers watching each other’s backs, if Nagios goes down, how do you know? For that matter, even if you’re monitoring it, if your only network connection goes down, how do you find out?

Yeah, I’ve got a loooong history of bad assumptions. I’d assume that I’m getting better, but that’s just part of the problem ;-)

]]>