mom: monitoring cpu spikes the right way

NOTE: this script is deprecated. feel free to use it, but you should refer to this post, which actually has a newer, cooler script.

one of the things i can't stand about most monitoring systems nowadays is that they're not really designed to be viewed by an operator. i think we've diluted that term. we don't enable "operators" to really do much of anything. we give them a little console they can stare at and hope that if they see some alert pop up, they'll wake up and dial someone. how does that translate into a successful use of technology? i think we've all been around a phone long enough to know how to dial it. so ... why not take some baby steps and move forward?

here's my baby step. i don't really do things out of my own volition because unless it's making my life easier, it's hard to be inspired. anyway, a fellow coworker received an alert on a cpu spike and asked the obvious question. what's making the condition occur? this raises interesting questions on its own because in order for anyone to answer this, they'd have to be at the machine at the time the problem occurred...

or at least in spirit, proxy, or whatever. then, you've armed your operator with at least a tad more information than what they had before. for mom anyway, the best way to do this is letting the agent handle it.

i wrote up a script that was bastardized out of microsoft windows base operating system state monitoring script. it's the one used to detect cpu spike conditions. that script returns a list of processes utilizing more than 10% of the cpu. so... i took most of the pieces, rearranged them, added a parameter for threshold ... and have added it our environment. aforementioned, it doesn't make sense to use this as a task or anything like that since you'd have to be sitting there glaring at the console, waiting for a cpu spike, and then executing, to get the problem occurance. just add the script as a response to an event or maybe a threshold rule.

it'll create an event so make sure you have an alert that'll pick it up. now, i suppose things that happen over a duration, the information returned may be pointless... since there could be multiple things going on over that duration. oh well... it's a start.

for my sample setup, i created a performance threshold rule that would alert on processor % time utilization. i set it to continously fire just for my test. appended to that, i created a response to run the script to return processes. since the script writes an event, i setup an event rule to grab the event and generate an informational alert. anyway, here's the details:

script properties:

• name: Top Processes
• parameters: Percentage
• value: 5

threshold rule properties:

• rule name: [Test Rule] Processor spike occurring!
• provider: Processor-% Processor Time-_Total-2.0-minutes
• threshold: the sampled value
• match when: always
• response: Top Processes

event rule properties:

• rule name: [Test Rule] Pick up events for top processes.
• source: Top Processes Script
• event id: 40100

i've posted the script to momresources.org and myitforum.com. pete's usually great about getting back to me once the file has been posted so i'm sure it'll happen soon. have fun with it and let me know what you think. it's rough around the edges, but i think you get the idea.