scom: overloading the consolidation module (and how to avoid it)

in a previous post titled using repeat count to detect a problem in a window of time I described a process whereby you can using consolidation settings, you can detect something happening in a window of time.  for example, event id 529 equals "bad password" basically.  if we alerted on every bad password, that'd be problematic.  however, if we looked at every one and then alerted whenever the count of bad passwords for a single user exceeded a threshold, that might be useful.

apparently there's this concept called a "consolidation module".  this module has a limit of 128k.  if you go beyond this limit, you tend to overload the module and cause the event scraping to backlog.  on very active domain controllers, using a large sliding window, it's very easy to overrun this limit.  it results in odd errors like this:

(event id 11105)

The Microsoft Operations Manager Condolidator Module failed to save the state after processing and might loose data. 
Error: 0x80070057 
One or more workflows were affected by this.

 

 

The Windows Event Log Provider monitoring the System Event Log is 317 minutes behind in processing events. This can occur when the provider is restarted after being offline for some time, or there are too many events to be handled by the workflow.

One or more workflows were affected by this. 

before you try to correct me, I copied and pasted that.  I know how to spell "consolidator" and "lose data".  I mentioned fixing this.  there are two ways.  the most obvious is by reducing your sliding window time frame so that you're not collecting as many events in a given period of time.  the second way is to simply set the storestate value to false.  the first one, you should be able to derive quite easily.  modifying the storestate value tells the agent not to store the internal state.  the problem here, albeit a small exchange, is that the state does not survive healthservice restarts.

as for the second method, it's not available in the console (surprise, surprise!) and must be done by editing the xml (surprise, surprise!).  so, you'll need to export your rule, modify your xml, and import it again.  once you have your xml, locate an area of the xml that should look suspiciously like this:

<Consolidator>
  <ConsolidationProperties>
    <PropertyXPathQuery>EventDisplayNumber</PropertyXPathQuery>
    <PropertyXPathQuery>PublisherName</PropertyXPathQuery>
    <PropertyXPathQuery>LoggingComputer</PropertyXPathQuery>
    <PropertyXPathQuery>Params/Param[1]</PropertyXPathQuery>
    <PropertyXPathQuery>Params/Param[2]</PropertyXPathQuery>
  </ConsolidationProperties>
  <TimeControl>
    <WithinTimeSchedule>
      <Interval>1800</Interval>
    </WithinTimeSchedule>
  </TimeControl>
  <CountingCondition>
    <Count>20</Count>
    <CountMode>OnNewItemTestOutputRestart_OnTimerSlideByOne</CountMode>
  </CountingCondition>
</Consolidator>

 

 

okay, now modify it to add this one entry noted below.

 

<Consolidator>
  <ConsolidationProperties>
    <PropertyXPathQuery>EventDisplayNumber</PropertyXPathQuery>
    <PropertyXPathQuery>PublisherName</PropertyXPathQuery>
    <PropertyXPathQuery>LoggingComputer</PropertyXPathQuery>
    <PropertyXPathQuery>Params/Param[1]</PropertyXPathQuery>
    <PropertyXPathQuery>Params/Param[2]</PropertyXPathQuery>
  </ConsolidationProperties>
  <StoreState>false</StoreState>
  <TimeControl>
    <WithinTimeSchedule>
      <Interval>1800</Interval>
    </WithinTimeSchedule>
  </TimeControl>
  <CountingCondition>
    <Count>20</Count>
    <CountMode>OnNewItemTestOutputRestart_OnTimerSlideByOne</CountMode>
  </CountingCondition>
</Consolidator>

after making the modification, import it back in and the problems should go away.