Warning: I hope your brain needs a slap in the neurons.  This is a little deep on technical stuff.  Enjoy.

In the past month, one of my customers ran into an interesting issue with all of their Windows 7 and Windows 8.x clients on the network.  The issue was that the security descriptor registry value for the BITS and WUAUSERV services were corrupted.  This causes Windows to skip over trying to even load them.  The symptoms are easy to misread at first:  Windows Update stops working.  Configuration Manager clients stop reporting and no longer consume policy updates.  Many will attempt reboot, repair (scm), and even WMI repair, since it’s a common knee-jerk reaction.

The first clue is missing services.  The “Windows Update” service will be missing in the Services console (see below).  The BITS (“Background Intelligent Transfer Service”) service may also be missing.  Windows can’t load them because the security descriptor blocks the kernel session context from reading the entry.  It’s just MIA.  BITS and Windows Update are very closely tied together, so if one breaks, they’re usually both out of action.

svcbad

The next clue has to do with Windows Event log errors.  Most will be related to DCOM and indicate launch and activation failures.  You could easily lose your mind chasing the ProgID and AppID class relations while someone else is yapping in your ear about something totally meaningless.  Ignore them.  This is more important!

Root Cause Evidence

The crosshairs will focus when you compare the Security descriptor registry value.  To do this, find a machine that is working fine, and one that is not (with regards to BITS and/or WUAUSERV).  Use the REG COMPARE command and point the two arguments to the same key/value path on both machines.  The key path is HKLM\SYSTEM\CurrentControlSet\Services\(service_name)\Security.  In this example, replace “(service name)” with either “BITS” or “WUAUSERV” and let her rip.

If they match, it should show something like the following…

cmd3When they don’t match, it will look something like the following…

cmd2Notice the difference?

The Fix?  Believe it or not, it’s as simple as exporting a good registry value and importing it on the other client device(s).  Then you have to restart in order for the service to load properly.  Big deal.  Simple.  High five and walk away.

But what about doing this on 5,000 or 10,000 computers in a given night?

You might think: “Use Configuration Manager and deploy a…”.   …Nope.  The clients don’t work because BITS is broken.  Group policy won’t work either.  You could do a startup script or walk around (roll around on skates maybe), but there’s a better way:  An Admin Script.

You can do this with PowerShell, but I was dealing with machines which have a wide variety of .NET and PowerShell versions.  WSH and VBScript are ubiquitous and stable across most platforms, particularly since XP days.  I like “ubiquitous”.  It’s a neat word.

Oh, there’s another catch…

The registry imports only work if you run them under the local SYSTEM context.  That’s because the broken security descriptor doesn’t allow any other account context to modify the value.  Not even local Administrators or the local Administrator account.  So this means I need to use Sysinternals’ PsExec and invoke it using -s on each client.  Not a problem.  One engineer suggested component corruption, but SCM won’t identify anything related to this since it’s actually a registry value that’s impaired.

Let’s get to the meat and potatoes…

The Ingredients

This approach uses a set of scripts, two (2) .reg files, and a client-side script which does the .reg file imports.  The names and types are shown below.

folder

The runscript.bat file is the starting point.  It runs the regfix.vbs script with a particular set of input options (parameters, switches, whatever).  It calls psexec.exe from a remote share to invoke the clientfix.bat script on each client that’s fed in from the command line or from a text file.

Script 1 – The kick-off script.

The runscript.bat file kicks off the VBscript (regfix.vbs) with the specified command arguments to load the computer names from a text file.  There are options to also specify wildcard patterns, numeric ranges and load from Active Directory, with or without name constraints.

As each client is updated, it creates a log file under the C:\WINDOWS\TEMP folder, allowing you to collect inventory later.  This also helps this script identify machines which have already been processed.  The /x parameter tells it to ignore that and process machines regardless of having been processed earlier. The /v argument is “verbose” for detailed status output.  The /r argument causes each computer to do a force reboot with a 30 second delay.  There’s a few more options (see script 3 below)

runscript_bat

Script 2 – The Client Script

This is what gets launched on each remote computer using PsExec.  It simply runs REG IMPORT and pulls both .REG files from a designated UNC location.

clientfix_batScript 3 – The Workhorse Script

The VBScript file is what does the heavy lifting.  It’s what parses the inputs and does the iterative headbanging grunt stuff.  Rather than try to capture a PNG of it, I just added a link to the .TXT version of that here.  You will need to edit the Const declarations at the top of the script to suit your environment.

More notes about this script:

It will only run when invoked from cscript (cscript.exe), and will warn you otherwise.  Typing “cscript regfix.vbs /?” will display the optional parameters you can use (see below).  The /w parameter runs test mode with verbose output.  I suggest using that before taking it to the next step.

cmdhelp

Putting it to Use

Enough talk!  The easiest way to set this up is as follows:

  1. Create a shared folder where all the remote computers will be able to read from it  (grant permissions on the share and NTFS to Authenticated Users and “Domain Computers” for read access)
  2. Copy the files into that folder, including psexec.exe
  3. Edit regfix.vbs to suit the path settings and domain LDAP situation.
  4. Export a clean set of .REG files for BITS and WUAUSERV service security settings.
  5. Run it.

Tips

If you use the /r option to force remote reboots (you can adjust the delay setting inside the regfix.vbs script), you can monitor their status using ping -t computername and see how they respond.  If you are unable to perform tasks against the remote computers such as “REG COMPARE” or “REG QUERY”, you may need to enable and start the RemoteRegistry service on the remote computer(s).

Buenos Nachos!

Advertisements

2 thoughts on “Brain Surgery: BITS, SCCM, Registry, Services, Corruption and Disruption

  1. Awesome… Never had issues like that, but I liked the way you solved it. I would like to have a job like this. I’m troubleshooting many issues here at work, but never went so deep as you did.
    Nice job. 😉

    1. Thank you! It is one of the perks of being a consultant. But it also depends on the customer and how they value a non-standard approach to challenges. Not all customers are open to such things.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s