Pavleck.Net

I haven’t seen this solution offered as a way to send more customized alerts, and am fairly excited about it. With some of the previous solutions, they involve using the command shell to create an alert notification. This is fine, except if you open the subscription in the GUI - once you’ve done that, you’ve essentially undid all that work and created a ‘catch all’ that sends an alert on any event. Why? Well, the GUI itself isn’t designed for the custom settings that can be done in Powershell. This makes it fairly difficult to add people or change the alert - not acceptable to me.

After messing around with authoring console and creating classes based on event viewer errors and other equally exotic methods I came upon something that works wonderfully. The catch? You can only create 254 rules this way.

What am I talking about? Some powershell scripts and the alert resolution states!

By default, there are only 2 states defined - 0 for New, and 255 for Closed. They are always there, and can not be deleted. This leaves 1 - 254 as user definable states. We can use these to make one-to-one events.

Let me start off that this isn’t an ideal solution, but it is the most readable and elegant solution for this particular problem. You probably shouldn’t do this on a single rule basis, but target it more at a wildcard match. You do have a naming convention for your rules and monitors, right? If not, this is the perfect reason to get one. I’ll typically use a convention of <Product Type>-<Product>-<Version (If multiple)-<Rule>. So if I had a rule targeted at exchange, I’d have a rule similar to “EMAIL - Exchange - Exchange 2007 - Search for ‘Jeremy is fired’ in execs mail”. Then when I’m using an exotic config to send an alert, such as this one, I can better fine tune alerts.

Remember, the more you move away from the  “Out of box” yfunctionality with OpsMgr, the more you should be documenting. Or even better, a wiki. Just make a reference to the wiki in the description, and people will know exactly what you’re trying to do - that’s for another post though.

Let’s get on with it, shall we?

I’m going to create a situation. I have a custom application which logs to the Application log. There’s one particular event that only one group in the organization cares about - all they want is a notification of this one single event and nothing else. How do we do it?

The Cliff’s Notes version of what we’ll be accomplishing today:

• Create a custom Resolution State
• Define a new rule
• Deploy a PowerShell script to the RMS to update the resolution state of matching alerts
• Create a notification subscription which responds to our particular state

Now, for the complete steps

1. First go to Authoring > Management Pack Objects > Rules - Right click and “Create new rule”
2. Under rule type, select Alert Generating, Event Based, NT Event Log (Alert) and select a management pack to use.
   
3. Enter a rule name that is distinctive enough that no other rules will have that same name.  Then enter a description, rule category and choose a target. You can go with the shotgun approach and pick “All Computers” here if you’d like.
   
   
4. Now walk through the rest of the wizard and configure your event log settings - for this test I’m using the Application log, Event ID of 926 and event source of “Pavleck.NET Test”. But you can put whatever you want here ace, it’s up to you.
5. Configure your alert. It should automatically copy over the rule name as the alert name. The alert name is what we’ll actually be alerting on, so it’s important that you remember what it is, and ensure it’s distinctive enough to not match something that already exists.
   
6. Now we can work on the other parts of this while our rule is propagating across the environment.
7. Go to Administration > Settings > Alerts - this is where we’ll define a new Alert Resolution State to use. Click on “New…” and name your state and choose an ID for it, I used 10 in this example.
   
8. Click Apply, then Ok and now we’re done with part 2.
9. Let’s go ahead and test and see if our rule works, just open a command window and use some EventCreate.exe magic.
   
10. Open up the alert console for whatever machine you ran that on and you should see our new alert in there - yay, we did something!
    
11. Now we’ll add the magic that changes the alert resolution state. It’s a fairly simple script, and it’s meant to be that way. For simplicity’s sake, we’ll be running this script as a timed response from the RMS. Depending on how your particular environment is setup, you could also run it inside of the rule itself, as an additional response to “Create Alert”. But that only works well if you only plan on doinbg this sparingly, otherwise it makes more sense to run this from the RMS and add onto the script as needed.
    First, download SCOM-UpdateResolution.ps1 here (Or view it after the jump) and edit the alert name, resolution state and RMS to what matches your environment.
12. Now we’ll need to go and create a new rule. Rule type is Timed Commands > Execute a command. Give it a name and description. I’ve set the rule category to “Maintenance” as that makes the most sense to me.
    
13. For the schedule, I’ve set mine to run every 2 minutes. This means there will be a delay of that much between alerts and notifications, but that’s acceptable to me. Then hit next.
14. Configure the command line execution settings as shown - remembering to use “&” instead of “&”. I’ve set the timeout to 45 seconds.
    
15. Hit create and that’s almost all of it - all we need to do now is to create the alert subscription. Go to administration, right click on Subscriptions and choose “Create new notification subscription”
16. Step through it like normal, choosing all groups, all classes. When you get to the Alert Criteria page, uncheck “New” and “Closed” and check our new resolution state. If you keep ‘closed’ in there, it will pertain to all alerts that close. That’s one drawback to this method, you won’t get closed alerts.
    
17. Finish it up as you normally would, then lets test it! Create a few more test events, and lets see if it works.

That’s all there is to it. This works, reliably and 100% of the time. It’s extremely flexible and easy to follow for someone just walking into your environment.

By using a single PowerShell script, and targeting the RMS computer group you’ll be making sure that you have only a single simple script to edit and by mirroring the files and directory paths to any other management servers in your environment you maintain this method if you ever need to promote one to an RMS.

Read the rest of this entry »

This is just a refresher course on how to convert an MS Operations Manager 2005 management pack into an OpsMgr 2007 management pack. It’s not tough, but I’m doing it so I figure I might as well document it!

1. Open your MOM 2005 Administration console, click on “Management Packs”, then select “Import/Export Management Packs” in the right hand pane.

2. Go through the wizard, selecting the export option, and choosing the rule group you’re looking for. You can choose to export views and tasks as well, but I recommend against that as it’s easy enough to recreate in OpsMgr and you have a lot more objects to choose from. Give it a file name - I usually use <Rule Group>.MP.2005 to make it obvious where it came from.

3. Next you’ll want to make sure you have the MOM 2005 Resource Kit installed on your management server and go to %PROGRAM FILES%\Microsoft Operations Manager Resource Kit\Tools\Convert Management Packs to XML. Copy the file MP2XML.exe to the same place you exported your management pack to.

4. Now get to that location via the command line, and lets take that binary AKM and make it useful. Converting it is pretty straightforward - just run MP2XML.exe MOM2005ManagementPack.akm MyNewXMLFile.xml.

5. When it’s done, copy that new MOM 2005 XML document into your SCOM directory. In that directory lie 2 tools we need, MPConvert.exe and MPVerify.exe. You can copy these tools somewhere else if you like, but the verification tool needs access to other management packs so make sure it’s in your %PATH% otherwise.

6. Run MPConvert.exe against your XML file. The tool is looking for MPConvert.exe OLD_NAME NEW_NAME. For simplicity’s sake I use the same name, only appending _converted to the end to differentiate the 2.

7. When that’s done, run MPVerify.exe against the new managment pack. What this tool does is ensure that everything has been converted over to the new format properly. It’s as simple as MPVerify.exe New_Management_Pack.xml

8. From there, it’s up to you. You can open the new unsealed management pack in the Authoring Console to adjust things, or you can just import it in as it is - the choice is yours! That’s all there is to it!

In MOM 2005, virtually everything was a rule. A rule looked for an even in the event viewer, a line in a log file, a return code from a script, etc and fired off an alert (Or did another action). It was essentially ‘dumb’, because it had no idea whether or not if an even it raised was ever fixed. It just fired them off every time it saw it.

Enter OpsMgr 2007. It introduced us to an old concept of the ‘monitor’. The monitor is a multi-state event. It watches for multiple items; something will set a particular item into a failed or degraded mode, and there is a corresponding event that marked it as being healthy again. This is wonderful, as it helps minimize the amount of open alerts sitting in your system at any given time. Less open alerts means we have more relevant information to look at.

When it comes to core Windows monitors, it works beautifully and 100% of the time. If you cross a memory threshold, an event is created and an alert goes out (If you’ve set it up to alert). When the memory drops below this threshold, then the monitor marks that particular object as being in a Healthy state again and, if you’ve allowed it to, it auto-closes the alert.

When this doesn’t work beautifully and 100% of the time is when you need to rely on 3rd party agents and management packs. I’ll use the HP Management Packs as an example, because that’s what I’ve been facing recently.

The way OpsMgr knows about hardware events that happen on an HP machine is because the HP agents themselves will place an event in the Event Viewer and/or send an SNMP trap about it. Works flawlessly to create an event in SCOM about an unhealthy object. What doesn’t work perfectly is the corresponding event that marks that system as being healthy again.

The reason for this seems to depend on the exact configuration of a server, the version of the HP agents, and the actual event itself. If there is an event, such as a power supply failing, the log is populated and SCOM creates an event saying “Power Supply #1 degraded.”. When that power supply is replaced, it won’t necessarily auto-resolve the event, because instead of seeing “Power Supply #1 Healthy”, the HP agents might instead log “Power Supply (Serial number: FD30401104-P) Inserted into Bay #0″. The monitor isn’t looking for that, and so it isn’t aware that that is the corresponding ‘good’ event, and the event stays open.

So theoretically you could replace a failing piece of hardware, such as a Power Supply, which doesn’t auto-resolve and then in the future have that same PSU die, which won’t cause a new alert and literally leave you ‘powerless’ to know what is going on.

Now, in a normal deployment of OpsMgr this isn’t to large of a concern. There are always eyes on the console or emails being sent. Someone will see it, fix it, then ensure the event is closed.

The current situation I’m in, however, doesn’t work this way. SCOM is being used consoleless to monitor a group of monitoring tools. Essentially it’s here to keep ‘them’ honest, and to ensure there’s another level of defense to protect us and let us know when a failure has occurred.

Because of this, those slight discrepancies in the HP agents and the HP management pack aren’t acceptable. But OpsMgr really doesn’t have a way of being run without anyone paying attention to it - or does it?

It actually does. What I’ve setup at this site is a PowerShell script which runs every 4 hours and resolves all the open HP alerts.The HP Agents themselves will run a self-check every hour or so, and log that “Power Supply #1″ is still failed. Because we’ve already cleared that alert, SCOM will pick it up again and re-fire the event, the alert, and all that jazz. In essence, we’ve created a ‘nag’ feature in SCOM.

This is beneficial in our case, because the current setup of OpsMgr where I’m at is mainly there to watch the other monitoring tools. This ‘nag’ lets us know that the problem was either not taken care of, or was not alerted on - thus ‘keeping them honest’.

How we do all this is very simple - the OpsMgr Command Shell has almost everything we need.

We’ll use Get-Alert to bring back a list of all open HP events, and Resolve-Alert to close them, adding a comment that we automated this.

To find the HP alerts, we need to match against the MonitoringObjectFullName property inside the alert. Through trial and error, I noticed that every single HP object began with “HewlettPackard”. So we’ll match against that, picking all alerts that don’t have a resolution state of 255 (Closed).

From there, we cycle through the alert array, passing each one to Resolve-Alert, along with a -comment - in my case I used “Closed by Powershell - see (link) for more details” with a link to the internal Wiki.

And that’s really all that there is to it. Mind you, I’ve done a lot more in the script, as you’ll see below. It measures how long it took to bring up the alerts, counts how many were per severity, the repeat count, etc then creates a PropertyBag and submits all the information to OpsMgr for reporting. It then also logs it to the eventviewer.

Download SCOM-Resolve-HardwareAlerts.ps1

This script is best setup to run every 4 hours or so. It’s setup as a generic ‘timed script’ inside of SCOM. If you’d like more info on setting up SCOM to work with Powershell more properly, see Brian Wren’s post here.

Here’s the script:

Ian Blyth of the System Center Technologies blog has put together a quite extensive list of MOM & SCOM bloggers. I encourage you to check em out and add their feeds to your feed reader - and to make it even easier on you, I’ve created 4 OPML feeds for you to choose from. They’ve been lumped into 3 groups; MS Blogs, Community Blogs, and Useful Feeds. Most newsreaders allow you to choose which links you want to import from a feed, and they are in folders, so choose the grand list first and if that doesn’t work, try one of the other ones.

SCOM & MOM - All Feeds.opml

SCOM & MOM - MS “Official” Feeds.opml

SCOM & MOM - Community Feeds.opml

SCOM & MOM - Useful Feeds.opml

Remember in MOM 2005 the ease at which you could find out exactly what rules were running on what agent? It was as simple as going to Administration > Computers > Agent-managed computers > RIght-click on the server you’re interested in, hit properties then the “Rules” tab and bam:

In OpsMgr, it’s not quite so east. Sure, there are a ton of powershell commandlets available to help you figure it out - I think. I haven’t quite managed to get the correct order of Get-Agent, Get-MonitoringObject, Get-MonitoringClass, Get-Monitor, Get-Rule to produce a successful listing of what runs on a particular agent. I’m fairly positive it has to be there, I just haven’t found it¹. So it was a tad difficult to find out what was running on a SCOM2007 agent.

Well, until Service Pack 1 RTM came around, that is. Located in UpdateCDImage\SupportTools folder is a management pack called “Microsoft.SystemCenter.Internal.Tasks.mp”. Go ahead and import it into your OpsMgr installation. I’ll wait right here.

Finished? Excellent. Now go to the Monitoring tab of the OpsMgr console and select “Computers” under Monitoring. Under the Windows Computer Tasks on the right side, you should see 4 new ones.

The new tasks are

• Resubmit local cache state change events
• Show Failed Rules and Monitors for this Object
• Show Local Cache for State Change Events
• Show Running Rules and Monitors for this Object

What we’re most interested in is the “Show Running Rules and Monitors for this Object” task.

Click on it, submit the task, wait for it to crunch for a little bit and then you’re presented with an excellent little screen like below:

Pretty nice, eh? A complete, valid XML document listing the rules & monitors (Called ‘Workflows’ in this case) running on the agent.

Now, let’s PowerShell that up a bit - you can pull out all that information from within the Command Shell, in a nice little function.

Oh also, while we’re on the subject, I’ve found the super super easy way to determine which agent a HealthService ID belongs to. I know my previous result used raw queries to the SQL database and all that jazz, but not this one. Ready for it? If you blink, you might miss it. Here it is!

Yep, it’s that easy. The wonders of powershell, eh?

Anyway, to run this task on an agent from within powershell, we have to do a little more work, but it’s really not all that bad.

Function Get-ActiveRules has 2 arguments: -server and -location. Server is self-explanatory, and it’s written to not require a FQDN; if your server is MYSERVER01.midwest.dc02.company.com, you just need to use MYSERVER01. The second argument is a location and filename for the output XML file. If left blank, it defaults to making C:\$server-rules.xml.

The is virtually no serious error checking, and it just dumps the Task OutPut field to an XML file instead of doing any magic with it, so it’s nothing fancy - take it as it is.

Enjoy! Download Get-ActiveRules.ps1

1. If you happen to know, please tell me! [↩]

I’m essentially mirroring this from the original one on my blog, but since I’ve decided to transition to a tech blog and a non-tech blog, it seems like a good idea.

I’ve taken the listing of SQL queries from Kevin Holman’s original post, cleaned them up a bit, and made it into one file which you can download and load into SQL Studio Manager, instead of copying and pasting individual queries. I’ll also be showing you how to automate a lot of these with PoSh, but right now I’m swamped and haven’t had the time to clean things up proper - so you’ll just have to wait.

Download Useful SCOM 2007 SQL Queries.sql

Read on to see what’s in the actual file

Read the rest of this entry »

Every now and then you’ll run into them - the ubiquitous online contest. And maybe you run into one that isn’t a scam, and put on by some big name companies - Marlboro, Microsoft, AMD, Best Buy - companies that you think ‘Hey, I probably have a good chance of actually receiving the item’.

Marlboro has one going on right now called “Flavor Chase“. They’re giving away one prize per minute from June 1st until July 31st. The prizes range from the low end (Mini Maglites ((217 given away per day)) ), to the middle area (Logitech Pure-Fi Anywhere iPod Speakers¹ ), to the high end (46″ Sony Bravia XBR LCD TV², Dell XPS m1530 laptop³, Washburn acoustic guitar⁴ and whole lot more in between. At the end of it all, they have 3 grand prizes - a Harley Davidson Night Rod Special, Dodge Challenger SRT8 or a Jeep Wrangler Rubicon.

You can enter once per day - and they’ll happily send you text messages and emails to remind you. That’s all fine and well - but what if you aren’t by your PC or phone for the day or even just plain forget?

SCOM can help.

How? It’s easy - SCOM has a management pack template called “Web Applications”. It is essentially a synthetic transaction recorder. Like it sounds, it will loop through a set of actions you do at timed intervals - specifically to gather performance information, but you can also use it to enter for you. Give it a shot!

It’s a fairly easy thing to setup, too. Go to Authoring, right click on Management Pack Templates, then click on “Add Monitoring WIzard”.

Now I’ll be writing this tutorial not using the Marlboro contest website. I’m in the contest, and just to be on the safe side I’ll do it the old fashioned way. Instead, I’ll setup a synthetic transaction that will go through a few google searches.

Read the rest of this entry »

1. 50 given away per day [↩]
2. One per day [↩]
3. One per day [↩]
4. 3 per day [↩]

One of the nice things about SCOM2007 is the ease in which you can monitor and manage non-Windows devices (Even better with the dhjdhdhdhd addon). Being able to add your switches, routers and Unix devices makes for a more complete overview of the health of your system.

There is a downfall though, and that would be that although you can use Get-RemotelyManagedDevice in the Command shell to list the, umm, remotely managed devices, all it will return is the IP address.

Here’s a SCOM command shell one-liner that will use some .Net-Fu to reverse the IP for you, on the fly:

If you want, you can go a step further and add it to your profile (Or more preferably, a SCOM snippet file that I’ll be writing about later.) and pull it up anytime you want.

The Display-NetAgentsByHostName function has one parameter, -short. Run the default ‘long’ way, it will reverse the IP, list the client that is managing it, the management group and the health state.
With the -short switch, it simply outputs health state, host name and IP address.

It’s not perfect, and coult use some formatting, but it works as is, so I leave it up as an exercise to the reader to polish it some more
Page formatting messing it up? I’ll fix it some day - until then, download Display-NetAgentsByHostName.ps1

Welcome to the new Pavleck.Net, the technical side of Jeremy Pavleck’s blog. I need to do a LOT of work on this now, but at least I know what I’m doing. So stay tuned.

Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!