Yep, pretty much. Indeed, when I talk about passive checks in Nagios, I am referring to NSCA. The two key differences are that SNMP is complex to understand out of the box (and requires a lot of wiring to generate MIBs and such) and that you also get “I’m OK” check points (so you know that reporting is taking place).

For instance, you don’t want to poll every N minutes to see if a disk has failed, rather you have an SNMP trap setup to alert you when the disk has failed.

Passive monitoring is possible via something like a Script Plugin to monitor the output of some activity. The script would be executed on a regular frequency (1 min, 5 min, 24 hours, etc)[...]

In the context of the post, passive fault monitoring is defined by two key characteristics:

• it happens asynchronously
• it is intrinsic to the component being monitored

There are, of course, exceptions. “Keep-alives” (regular status OK reports) must be somewhat synchronous so that if something goes wrong and no OK reports take place, the monitoring server can flag it as a problem. Furthermore, there are components (such as appliances) for which it is simply impossible to have [usable] monitoring built-in (some SNMP implementations are rather broken), in which case, a proxy is necessary (asynchronicity is usually lost as well in this case). Finally, there are cases where what we want to monitor for is external behavior (for instance, wher multiple components are involved in a transaction), for which active monitoring is the right tool.

What is, I think, a significant gain in the passive model is that I can save myself the step of watching for, parsing and interpreting output by having the monitoring intelligence built right in, taking place in real-time, and with a richer environment to drive conclussions as to the state of the process and the reasons behind said state, because the monitoring is taking place in the runtime environment. My preference, particularly for home-grown software, is to push as much of the monitoring smarts for a given component to the component itself (and, by extension, to the engineers writing said component) because it is the One True Source. None of this is to say that passive monitoring is the One and Only True Way.

In the realm of operations we write a fair number of tools to aid in running the environment. A lot of these tools perform their work in the background, for instance, moving logs back and forth or pushing changes to systems. If I control those tools, I will definitely make them use passive fault monitoring, and only pester me when necessary. No digestion of output needed. For engineering, where they write large amounts of code to make up the product or service being delivered, passive monitoring is likely the right answer. Take some application server performing database queries to build web pages being consumed by an end user. The best place to know that the running average database query time has slowed by 20% in the last five minutes is the application server itself. Again, no output to grok, and better yet, no polling schedules where swings in the metrics can return to normal.

It is, of course, possible to write these stats somewhere and have the Script Plugin read them. But that’s an extra step I find no need to take. This, of course, requires bringing a lot of moving parts into the same wavelength. A solid definition for monitoring thresholds, procedures, responses and specifications must be in place, well documented and publicized to the appropriate parties. But having this done, contracts can be enacted so that those that are writing code know how they can raise awareness about something going amiss. These definitions are mosrly tool agnostic, but having them allows for levels of automation that are impossible otherwise.

I still believe that the summary of the post is true: Hyperic is not yet a full Nagios replacement as far as passive monitoring is concerned. I don’t simply want it to look like a push model from the outside (fakeable by increasing frequency): I want it to be a push model because I find said model (and I want to stress that I am referring to fault monitoring) to be more accurate, resource friendly and easier to manage.

Thanks for the clarification, I get your points now.

We’ve used this same approach in many, many a Nagios->Hyperic migration.

It’s not a push model, but it looks like it from the outside.

Technically, I never claimed to prefer Nagios over Zenoss. I mentioned Zenoss as an example in the context that there is no mention of it in the Hyperic documentation, whereas Nagios has a section devoted to it especifically geared towards helping in migrating from Nagios to Hyperic (in fact, they have tools to crawl Nagios configuration files and import them into Hyperic). This leads to me to speculate that Hyperic sees current Nagios installations as a good source of potential Hyperic users, whereas Zenoss (for reasons known only to Hyperic) does not seem to be as appetizing. I suspect this has to do with raw numbers and probabilities (i.e., quantity), but it’s all conjecture.

My last direct (and admittedly brief, since I had a local resource far more familiar with Zenoss that I was) contact with Zenoss was a couple of years ago. I mention this because much has probably changed since then, and I am therefore in no position to make any authoritative comparisons in regards to Zenoss. I will say however, that, at the time, we found Zenoss to be very SNMP-centric (in an environment where we had very little of it) and easy to bog down under load with active checks (the system population was fairly large). Having Nagios-based infrastructure that was performing well provided no incentive to migrate to Zenoss, particularly as brittle as it seemed to us.

My key point is that, as much as possible, I am in favor of passive test methodology for fault monitoring because from my point of view, it is inherently highy distributed (given components are keeping track of their own health), takes advantage of levels of introspection not readily available with active checks (i.e., external poking mechanisms), and it’s asynchronous (faults are reported as they happen, not during the next check). This is not to say that everything should be passively monitored: it depends upon the semantics of the test itself, and, in many cases, it is impossible or impractical to do so. Regardless, Hyperic doesn’t seem to do it: the agent model is really an extension of the active check model, where checks still poke some other entity and run on a schedule (but they do so on the client hosts). Zenoss, I believe, doesn’t offer this capility either.

Whether the monitoring software is Python or C it almost irrelevant because I want (and have in many cases) put the onus of keeping track of health on the application itself. Thus, I have simple scripts or system software that is concerned about its own health, and my monitoring platform only has to be concerned about receiving status and acting upon it.

Clearly, none of this is really relevant to performance monitoring (i.e., monitoring that isn’t really about faults but about maintaning historical data about system load and other metrics along those lines). Zenoss did well in that regard, as does Hyperic.

I have been working on a more abstract post about my views on these ideas, but it is a complex subject, and one that, admittedly, is probably somewhat subjective. At the end of the day, I want simplicity, and there is much to be said for having a solid, well-designed template configuration in Nagios that lends itself to simple generating of host configurations from a CMDB that inherit the right properties from the right places. There is no denying that this is possibly doable with both Hyperic and Zenoss as well, but I’m certainly not qualified (yet) to ascertain how difficult it might be (so far, my Hyperic experience has left at lot to be desired, but it takes time to hit the eureka moment when automating becomes an extension of your environment).

I would love to get your thoughts.

Good to hear from you!

I actually like XWiki quite a bit and have been playing with it here and there (time being one of those rather difficult to come by resources). It really comes down to how fast I can get the wiki up and running, and the major dependency has nothing to do with the quality of XWiki or Confluence (I do think they’re both top of the line products) but my familiarity with them. I have been a Confluence (and JIRA) user and admin for quite some time now, which allows me to have Confluence up and running in short order. Then there is the question of support (as in my ability to answer questions or troubleshoot issues, not support from XWiki or Atlassian).

As it turns out, a gracious soul donated a Confluence license, so I’ll get those $5 to work this weekend, but I will continue to explore XWiki as time permits, because the scripting capabilities you mention are certainly quite useful and a huge plus in XWiki’s favor.

And you might be amazed by what XWiki can offer with it’s scripting and structured content capabilities. See http://www.dzone.com/links/jazoon_day_2_xwiki.html

Disclaimer: I’m the XWiki founder