Master Plan – Automating an SCCM Site Installation

I was thinking, “Man, (or woman), it would sure be coolio if I could push one button and *presto!* I have a fully-built SCCM site server, without having to install anything but Windows itself“.  There’s some really shiny stuff out there already, like hydration kits and prerequisite installers.  But then, I thought “That’s not enough!  I need it to be ‘real’.  I need more!!” I want to build the site itself, and configure EVERYTHING to be JUST LIKE A REAL site out in the real world.  So, here’s how I sketched it out…

  1. Install Windows Server
    1. Do NOT patch it
  2. Create all the folders in all the right places
  3. Install Windows Server roles and features
  4. Install ADK for Windows 10
  5. Install MDT
  6. Install SQL Server with file auto-growth settings, Domain Service accounts, and register the SPNs
  7. Configure SQL Server memory limits
  8. Install the WSUS role
  9. Run the WSUS post-install configuration step
  10. Create file NO_SMS_ON_DRIVE.SMS on C:\
  11. Install Configuration Manager 1511 (not the latest baseline) with a Site Code, Site Name, Roles
  12. Configure Discovery Methods, including Network Discovery, where each one runs every day
  13. Initiate Discovery Methods
  14. Create Site Boundaries and Boundary Groups
  15. Configure Client Settings
  16. Configure Client Push Installation
    1. Configure automatic client installation and client upgrades
  17. Create and Configure Query-based Collections
  18. Configure all Query Collections to update every hour
  19. Create and Configure a set of Applications
  20. Deploy Applications to Collections
  21. Add everyone in IT to the “Full Administrator” RBAC role in SCCM
  22. Add everyone in IT to the SQL Server server admins role
  23. Install Symantec Antivirus, McAfee Antivirus, Cylance Agent and anything else on the same host
  24. Turn on the Domain Firewall
  25. Create an OSD configuration
    1. Windows 10 Image captured from a 5-year old machine with EVERY conceivable application installed.  If it’s not at least 10 GB, we have to repeat until it does.
    2. Boot Images – add every component and every driver for every model we have
    3. Drivers – everything.  all of it.
    4. Applications – only those product names which begin with 0 – 9 or A – Z.
    5. Task Sequences – create a dozen with names that mean nothing to anyone unless they’re on drugs
    6. Deploy each TS  to “All Systems” via PXE without a password
  26. Set a scheduled task to reboot the site server every night at the same time the backups and SQL jobs are supposed to run
  27. Add “Domain Users” to the local Administrators group
  28. Turn off Site Maintenance tasks: backups, reindex.
  29. Do not install Ola’s tools – or Steve’s guidelines for SQL – they just make it run too well.
  30. Install Google Chrome, MS Office, Adobe Reader and any freeware applications I can find on the primary site server
  31. Change the NIC to use DHCP
  32. Install additional Web Site applications within IIS
  33. Verify no PKI exists and then configure all clients to use PKI only.
  34. Take a snapshot/checkpoint every day and once a month revert to a previous snapshot from a week earlier.
  35. Post job openings to all the job search sites insisting the candidate ONLY know Ghost or Acronis and has never touched or even heard of SCCM or MDT, ever.
  36. Configure Azure with O365 and and EMS tenant
  37. Integrate SCCM and EMS
  38. Enroll devices in EMS/Intune
  39. Deploy SCCM clients to the Intune devices
  40. Automatically open support tickets with Microsoft on why the clients stop working in EMS/Intune

That should just about cover it.

GPODoc – PowerShell Module

I just posted my first PowerShell module to the PowerShell Gallery, even though it’s been available on my GitHub site for a little while longer.  It’s a small project, with only two (2) functions, so far.  Get-GPOComment and Export-GPOCommentReport.

Get-GPOComment is purely CLI and helps query GPO’s to return embedded descriptions, comments and so on.

Export-GPOCommentReport is a CLI function as well, but produces an HTML report, which is technically GUI output, of embedded GPO comments and descriptions.

The module is simple to load by way of the Install-Module and Import-Module cmdlets (see below).  The only requirement is to have the GroupPolicy PowerShell module installed first.  This is normally installed on all AD domain controllers, as well as other servers and workstations which have the Windows 10 RSAT package installed (tip: You can also use Chocolatey to install RSAT:  choco install rsat <or> cinst rsat)

Loading The PowerShell Module

Install-Module GPODoc

Trying it out: Get-GPOComment

Get-GPOComment -GPOName '*' -PolicyGroup Policy

get-gpocomment1

Get-GPOComment -GPOName '*' -PolicyGroup Preferences

get-gpocomment2You can easily modify the output layout since the output is in hash table format…

Get-GPOComment -GPOName '*' -PolicyGroup Preferences | Format-Table

If you want more flexibility in selecting the inputs, you can use the Get-GPO cmdlet and channel the output through a pipeline.  For example…

Get-GPO -All | Where-Object {$_.DisplayName -like "U *"} | 
  Select-Object -ExpandProperty DisplayName | 
    Foreach-Object {Get-GPOComment -GPOName $_ -PolicyGroup Policy}

get-gpocomment3

The parameters for this function are as follows:

  • GPOName
    • [string] single or array of GPO names
  • PolicyGroup
    • [string] (select list) = “Policy”, “Preferences” or “Settings”
    • Policy = query the GPO descriptions only
    • Settings = query the comments attached to GPO internal settings only
    • Preferences = query the comments attached to GPO Preferences internal settings only

This also respects -Verbose if you want to see more background details.

Export-GPOCommentReport

This function only takes two (2) inputs:

  • GPOName
    • [string] single or array of GPO names
  • ReportFile
    • [string] the path/filename for the HTML report file to create.

It invokes Get-GPOComment, combining “policy”,”preferences” and “settings” output, and writes it all to an HTML report file of your choosing.  The CSS formatting is hard-coded for now, but I’m open to feedback if you want more options. You can specify the GPO name(s) directly, or feed them via the pipeline.

export-gpocommentreport1

Example web report…

export-gpocommentreport2

If you used 1.0.1 (posted a few days earlier) scrap that and get 1.0.2.  The older version had some major bugs in the Get-GPOComment function, and the Export-GPOCommentReport function wasn’t available until 1.0.2.

Feedback and suggestions are welcome.  Even if it’s “hey man, your shit sucks. Consider a dishwashing job, or something else.”  Any feedback is better than no feedback.

Thank you!

 

Documenting Your IT Environment (The Easy Way)

One of the most common, and tragically problematic, aspects of today’s IT world is the lack of sufficient documentation. In fact, ANY documentation. It’s not associated with any one area either. It crosses everything from help desk tickets, to change requests, to Active Directory objects, to Group Policy, to SQL Server, to System Center to Azure.

UPDATED 8/8/17 – added SQL and SCCM examples at very bottom.

When and Why Does It Matter?

It matters when things go wrong (or “sideways” as many of my colleagues would say).  There’s the “oh shit!” moments where you or your staff find the problem first.  Then there’s the “oh holy shit!!” moments when your boss, his/her boss, and their bosses find the problem first and come down hard on you and your team.  When shit goes wrong/sideways, the less time you waste on finding out the what/where/why/who/when questions for the pieces involved with the problem, the better.  To that end, you can document things in other places, like Word documents, spreadsheets, help desk and ITIL systems, post-it notes, whiteboards, etc.  Or you can attach comments DIRECTLY to the things which are changed.  This isn’t possible in all situations, but it is most definitely possibly for many of the things that will straight-up shove your day into shit storm, and I have no idea what that really means, but it just sounds cool.  Anyhow…

The best part of this approach is that you can leverage PowerShell and other (free and retail) utilities to access and search comments you nest throughout the environment.  For example, you can collect and query Group Policy Object comments…

Need to Get the Comment buried in a particular GPO or GPPref setting?  Here’s one example for querying a GPO under User Configuration / Preferences / Control Panel Settings / Folder Options…

#requires -modules GroupPolicy
<#
.SYNOPSIS
  Get-GPPrefFolderOptionsComments.ps1
#>
param (
  [parameter(Mandatory=$True)]
  [ValidateNotNullOrEmpty()]
  [string] $GPOName
)
try {
  $policy = Get-GPO -Name $GPOName -ErrorAction Stop
}
catch {
  Write-Error "Group Policy $GPOName not found"
  break
}
$policyID = $policy.ID
$policyDomain = $policy.DomainName
$policyName  = $policy.DisplayName
$policyVpath = "\\$($policyDomain)\SYSVOL\$($policyDomain)\Policies\{$($policyID)}\User\Preferences\FolderOptions\FolderOptions.xml"
Write-Verbose "loading: $policyVpath"

if (Test-Path $policyVpath) {
  [xml]$SettingXML = Get-Content $policyVpath
  $result = $SettingXML.FolderOptions.GlobalFolderOptionsVista.desc
}
else {
  Write-Error "unable to load xml data"
}
Write-Output $result

Example…

So, What Now?

So, in the interest of avoiding more bad Mondays, start documenting your environment now.  Need some examples of where to start?  Here you go…

 

Thank you for your support.

60 seconds of IT Advice

1. DO NOT get comfortable on old technology.  Keep moving.  Keep current.  It hurts a LOT less to upgrade now, than to wait for later.

2. Patch, Patch, Patch the fuck out of your environment until there are no more patches left to be applied.  I apologize for the expletives, but it’s really that fucking serious.

3. Test EVERYTHING in an isolated environment.

4. Document EVERYTHING. Use internal commenting wherever possible (Group Policy, AD, SQL Agent Jobs, scripts, etc.)

5. DO NOT EVER grant admin rights to anyone using their standard login. EVER.

6. Log files – Learn them. Know them. Live them.

7. Find a hobby. Anything. Get your mind into a separate creative channel after hours and on weekends.

8. Ask “Why?” as often as possible. Especially to yourself.

9. Respect the coffee.  All hail the coffee.

10. Take it seriously, but not too seriously.  (tip: envision describing your worst IT problems to the kids pictured above)

11. Have you finished patching yet?!?!

Poor Man’s IT Chain Reactions

wpid-chinese-take-out.jpg

Challenge:

Make sure every machine in the enterprise (connected to LAN or always-on VPN) has the latest version of psexec.exe on the local C: drive.

Why?

Why not?  That’s why.

Option 1:

AKA – the semi-automated, safety switch turned off, fully-loaded, drunk guy holding the trigger option.

  1. Download psexec.exe from live.sysinternals.com (or direct: https://live.sysinternals.com/psexec.exe) and place into AD domain SYSVOL scripts folder (e.g. \\contoso.com\netlogon)
    Example…

    $WebClient = New-Object System.Net.WebClient
    $WebClient.DownloadFile("https://live.sysinternals.com/psexec.exe","\\contoso.com\netlogon\psexec.exe"
  2. Create Group Policy Object (GPO) with Computer Preferences setting to copy psexec.exe from the SYSVOL share to a location on the local C: drive. Configure to “update” so that future version updates will be passed down to the clients.
  3. Create a Scheduled Task to keep the SYSVOL copy up to date with the latest version.

Pros

  • Cheap (free)
  • Fairly automated (just add water, makes it’s own sauce / set it and forget it)

Cons

  • Smells like duct tape and coat hanger wire

Option 2:

AKA – The “I have a budget, so kiss my butt” option.

  1. SCCM package or application deployment

Pros

  • You look cool pulling it off, but not as geeky as option 1.

Cons

  • More moving parts under the hood.
  • May require additional steps to maintain a consistent current version across all devices.

Option 3:

AKA – The “I don’t have a budget, so kiss my butt” option.

  1. Include within image configuration (MDT, SCCM, Ghost, Acronis, etc.)

Pros

  • Easy

Cons

  • Difficult to maintain a consistent and current version across the enterprise

Option 4:

AKA – the “most fun to laugh about during the next beer-meeting” option

  1. Send the new guy around with a USB thumb drive

Pros

  • Great fun in the office

Cons

  • Do I really need to spell this out?

 

IT Security Methods by Industry

After years (okay, decades,… okay, okay, centuries…..  damn it… alright! alright already, eons… are you happy now?  yes.  I’m THAT freaking old.  I still remember coal-fired computers and horse-drawn airplanes and shit.  My birthday cake is a slice of tree trunk of matching rings, but the table can’t hold the weight anymore.  sheesh!)

What was I saying?  …. (eyes wandering left and right…. … . . .          …  .         …. . .      .   .  )

oh yeah!  I’ve amassed a data set that accurately summarizes the predominant security practices or strategic “methods” leveraged by each major US industry. I warn you: this is highly scientific information.  It may require additional consumption of various questionable substances just to remain conscious while trying to read it all. Here goes.

Idiocracy-LB-1

Banking

Method: Place sufficient restrictions on the adoption of new technologies, so as to (A) mitigate unknown vulnerabilities and exploits, (B) insure that those with knowledge of older, proven exploits have died from old age, and (C) keep certain aging consultants employed (because they’re married into your family).  And besides, what’s wrong with COBOL?

Insurance

Method:  Never leave important IT decisions up to any one person, ever.  In fact, the more people involved, the greater insurance that the decision will eventually be reliable, maybe.  Larger companies focus on perfecting multi-role hyper-proliferated subterfuge logic branching and coalescing processes.  In layman’s terms: they foster greater variety among responses to decision inquiries.  Many have invested heavily in processes which depend entirely on custom hand-stitched, stone-carved, natural leather encased software, usually written by someone who left or died long ago.

Defense Manufacturing

Method: Implement dozens of stop-gap procedures to insure every motion of IT is slowed to the lowest possible, almost un-measurable, velocity.  Think of a Japanese rock garden, only slower.  Where the sand is executive processes and the stones are IT staff, now simply add quick-set cement to the sand mix and sprinkle some water on it.  This insures that even the bad stuff will take forever to make headway, and by that time, the entire system will have been eventually decommissioned.  Forget penetration attempts, even social engineering-based, because they’re often project-oriented, not departmental, so most people have no clue what that next cube is working on.  In fact, they probably don’t use the same network, computers or operating systems.

Legal

Method: Relegate “IT” to whomever answers the Craig’s List ad for an “IT Expert”.  Critical skills include: printer management, thumb drives, recovering lost files and emails, and using Excel databases” (that’s not a typo).  Must also have experience with Macs and Windows XP, particularly with kids games.

If they have any in-house “IT” capacity at all, it’s often enough shock to send a consultant into cardiac arrest.  Due to possible legal implications, it’s best to never change passwords for critical user accounts and never, I mean NEVER, delete anything.  Keep everything forever, or as long as you can afford somewhere to store it.

Travel

Method:  Agents need to be flexible and mobile.  Everything is done on laptops.  Everything remains on laptops.  No time for that silly, trendy, cloud stuff.  No backups, no cloud sync, but OMFG do NOT let anything happen to that precious data on those roaming laptops!  Thumb drives are forgotten like Matt Damon in Interstellar, waiting for someone to give them a hug, only to have their face shield cracked open and their chip tossed away.  Shit.  Did I give away the plot?

Advertising / Marketing

Method: Hire someone quick, and get back to the conference before the food runs out.

Transportation

If it’s airlines, use railroad standards.  If railroads, use airlines standards.  Either way, the older the technology the better.  It’s like a cast-iron frying pan, after years of seasoning, or a vintage wine.

 

Municipal

Method: Deny all requests for pay increases for five (5) years, reduce promotions from once every five (5) years to once every ten (10) years, discontinue any training programs, and for God’s sake: deny all requests for stupid things like newer software and hardware  It worked in 1995, so it should still work!  Hire a consultant to blame internal staff for every deficiency, terminate and reassign to avoid audit trails and blame the contractor afterwards.

Federal Agencies

Method: Same as municipal, but on a much larger scale.  Every four (4) years, change direction from in-sourcing to out-sourcing, and blame the opposite for any failures that remain.  If conservatives win, out-source to private contractors, where expertise and trust are premium values, after all, when has anyone ever heard of a private contractor doing something wrong in a government position?  Then blame liberals.  If liberals win, open up the job requisition flood gates and hire at will.  However, keep GS-rating pay scales at 1995 levels to avoid asking for tax increases.  This helps insure only the highest-quality employees are onboarded from their previous positions as private contractors or foreign exchange students.  Then blame conservatives for any failures.  Think of it as seasonable employment.

Medical/Dental Practices

Method: Hire the first contracting IT firm that actually shows up.  If they wear those spiffy-looking polo shirts with a slick company logo, they might be too expensive.  Ask if your cousin’s friend graduated tech school yet.  You know, the one who puked all over your sofa when he brought her to crash in your apartment while you were out of town.  That one.  If she’s not available, what about that kid that asked you about spark plugs while you were trying to inflate your car tires that day.

 

Summary

See if you can guess which of these most closely matches the photo above.

(Seriously) 5 Most Common SCCM Issues

Joking aside (for a few minutes anyway)…

teamamerica3

The five (5) most common root causes for SCCM site issues that I’ve seen over the past year, working as a consultant.

  • Site scale:  (smallest) 500, (largest) 180,000
  • Site types: CAS (5%), Primary alone (85%), Primary with Secondaries (5%), None (5%) aka “new install”
  • Avg staffing: (IT dept) 12-24 (SCCM admin) 1
  • Avg coffee consumption: 1 cup per 30 minutes
  • Avg sleep: 5.2 hours

1 – Lack of planning before installing the environment

In the past year alone, I’ve run across almost a dozen sites which had a CAS and didn’t need one, or Secondary sites, and didn’t need them, and so on.  Some didn’t have a FSP and could’ve used one.  Some weren’t using the appropriate credentials for client installations, network access and so on.  And lately, many seem to have pinned their plans on outdated platforms, such as Windows Server 2008 R2 or SQL Server 2012.  At least keep them patched (e.g. SQL 2012 SP3 CU9)

2 – Lack of monitoring and following-up on warnings/errors

Of the last 24 customer engagements I’ve been involved with, roughly 60% do not keep a daily watch over site issues (sites, components, clients, content distribution, deployments, etc.).  Of those that do monitor, about half ignore lingering warnings which impact site performance.

3 – Lack of cohesive management

This varies by scale/size of the organization (at least in my world).   Often it’s a matter of job roles and organizational divisions.  For example, DBA’s controlling the SQL Server environment without allowing SCCM admins any direct access (very bad).  Or AD admins who drag their feet (or push back) on requests for schema extensions, keeping AD accounts “clean” and so on.  Or Network Admins who fight back against using PXE, no matter what the rationale.  In many cases, it rolls up to team managers who don’t work well together, so resolving conflicts and barriers is difficult, especially when the CTO or CIO prefer to avoid dealing with it.  My advise: deal with it!  The good of the company outweighs your stupid personal disagreements.

4 – Lack of keeping up on updates

Whether it’s the Windows Server, SQL Server, ADK, MDT or Configuration Manager itself, all of these require persistent support and oversight. Keep them patched.  But more importantly, READ THE PATCH details first.  Understand what’s being “fixed” or “modified” (or deprecated) as well as “known issues”.  You can save yourself a shit-ton (that’s a scientific measurement, by the way) of headaches and support costs by not blindly installing without understanding.  However, do not avoid patching simply because of fear and doubt.  You work in IT, which means “change” is inevitable and continuous.  It’s why the “soft” in “software” exists (trust me, Babbage wasn’t kidding around).

5 – Inefficient use of features

This one alone could be broken out into sub-categories actually, and now that I mentioned it, I will…

a – Ignoring features which are not fully understood (not doing research)

b – Continuing to use outdated methods (disk imaging, for one, like Acronis or Ghost)

c – Ignoring other System Center capabilities (SCOM, Orchestrator, etc.)

d – Not following “best practices” (excessive permissions on common accounts, incorrect client installation settings

e – Paying for 3rd-party products which SCCM (or other System Center) capabilities could provide (depends upon the individual requirements of course)

f – Ignoring 3rd-party products out of fear of the unknown (FUD)

g – Ignoring new features added with each build (current branch), such as Azure, OMS, UA, and mobile device features

h – [my peeve] Inefficient mapping of tools to processes.  Such as ignoring Group Policy in favor of doing everything in SCCM or via scripts. Continuing to use familiar solutions even when newer and better (cheaper, faster, more efficient, more reliable) solutions are available.

i – Insufficient use of Internet search tools (Google, Bing, etc.)

Did I miss anything?