Not long ago, while consulting with a particular municipal customer, I was tasked with developing a process to validate IT asset inventory. It sounded simple enough. They already had Microsoft System Center Configuration Manager (SCCM) in place, and they had a single-domain AD forest. Checking off the usual list of suspects, they were fortunate enough to have all of the five (5) basic inventory systems in place:
- Active Directory (AD)
- System Center Configuration Manager (SCCM)
- Purchase Orders (POs)
- Maintenance Records (MRs)
- Disposal Records (DRs)
In human terms:
- A birth record repository
- A driver’s license repository
- A Social Security card repository
- A medical record repository
- A death certificate repository
Case Study: The Usual Problems
Pulling information from all of their repositories and comparing them wasn’t very difficult. I created a SQL database, some custom tables, views, and procedures (or TVF’s), and built a handy web interface to make the results colorful and pretty. Easy on the eyes of suit-clad manager types.
Within a few hours, the results were in. Of the roughly 6,000 desktops and laptops in their environment during that time, approximately 15% or 900 were “out of alignment“. This is one way to say their associated “records of existence” did not agree. After further investigation, we determined that around 5% (or 300) were outright invalid. This is just computer devices. I’m not including peripherals and ancillary assets, such as docking stations, monitors, printers, IP phones, and all that.
But, what does any of this really mean?
When I say “out of alignment“, I’m talking about one tracking system not agreeing with another tracking system. For example, AD and Purchase Orders, or SCCM and Disposal Records. When I say “outright invalid“, it means that the actual asset (computer) did not exist in their operating environment. It had been removed, disposed, lost or stolen, vaporized, eaten, whatever. Regardless of how or why, it was effectively gone.
So, I built a SQL database and a simple web interface to analyze the sources and display the results for direct comparison. I threw in some nifty colors and graphs to wow the suits, just to be safe. The data was “live”, meaning it was being rendered pulled directly from the respective sources as the web page was rendered, and each time they pressed F5. Nothing incredible, very 101 stuff.
After ten minutes of blank stares, finally the CTO blurts out, “I don’t like this! What am I looking at exactly?!” So I started over my explanation bombing run and when I ended the second round, I was again surprised to see his dissatisfaction. “Which one is wrong?!”, he demanded (very grumpy guy). “Well, sir, all of them, or none of them. Take your pick.” He did not like that at all. After fifteen minutes of mindless ranting, he issued a final order, “I want all of them to concur with the PO list.” The faces in the room pretty much looked like we were watching a cat eat an elephant right in front of us.
Why are there five (5) data sources in the first place? Let me explain…
- The Purchase Order marks the start of an asset. It’s birth record. It shows who bought it, when they bought it, for whom it was bought and so on.
- The Active Directory account shows that the asset was at one time allowed to play in the production sandbox with all the other computers, users and resources. It’s like having a Driver’s License. Now we know the device was born, and was able to obtain a permit to drive.
- The SCCM resource record shows that it was detected during a Discovery Cycle. If there’s inventory data, then it shows that the asset (device) was “real” enough to have an SCCM agent installed and reporting back inventory data. It’s like having a Social Security card. Now we know the device was born, obtained a permit, and was actually seen driving on the roads.
- In addition to being born, having a driver’s license, and a Social Security card, the Maintenance Records show sequential events in the life of the asset. Events such as user assignments, department assignments, component upgrades and replacements, software license upgrades, and so on. This is like a medical records history.
- The Disposal Record is the death certificate. Not much more to say than that.
So, if we have a PO, an AD account, and an SCCM inventory data set, and no Disposal Record, the asset (device) should in fact exist in the production environment, somewhere. Right? Not so fast.
What happens when you’re holding a matched set of PO, MR, AD, SCCM and DR and you’re standing in front of a person using the device, connected to the business network?
If you think, “oh boy, that’s crazy”, you haven’t begun to consider the ramifications:
- A production computer carries an associated support fee (per month). That fee is based on “valid” inventory records. This device was getting free support.
- A separate software license inventory was based on valid hardware inventory. The licenses on this device weren’t being accurately counted. Even with SCCM, the report used for financial planning was only loosely derived from the SCCM data.
- The AD account was disabled, but never deleted. The computer had been re-imaged with a new name and joined to the domain again. The account was placed under the wrong department-based OU and not added to the appropriate department-based security groups.
This would be small concern for a single instance. As it turned out, there were many others like it.
Some of the most common root causes of “variance” (discrepancies) come from the following:
- Forgetting to remove AD accounts for disposed computers
- Forgetting to update Maintenance Records when assets are reassigned
- Improper OS deployment practices resulting in duplicate names in SCCM (or new names for same device, but old name is never removed)
- PO was never created because device was purchased through uncommon process
- Device never “touches” the central network environment (i.e. “off grid”)
- Device is placed in storage and forgotten
- Device is collected for disposal, but after DR is created, the device is reintroduced into the environment elsewhere.
The more you review the list above, you should start to see a pattern. The real cause is HUMAN ERROR. The tragedy is that too-often, upper management assumes it’s SINGLE HUMAN ERROR and someone gets to be the fall-guy. The real problems is that it’s an INSTITUTIONAL ERROR. In other words: a broken process.
For example, when any of the following appears on the root causes list, there’s a clear indication that the business needs to rethink how they operate:
- Decentralized, independent operations management
- No inventory tracking system (e.g. SCCM)
- No telemetry or maintenance tracking
- No life-cycle or disposal process
- Incomplete records keeping
In most of the tragic cases I’ve witnessed, it’s the software licensing beast that bites them in the end. Someone gets wind that the business isn’t managing their software licenses effectively, and then the BSA shows up.
The fact is, even in the realm of businesses who place inventory as their “number one” priority, things get out of alignment. They’re just better at identifying them, diagnosing them and resolving them. That’s still not an excuse to slack off.
Some people ask why the cloud is becoming so popular and why IT vendors are pushing it so aggressively. Part of the answer is ‘control’. Control over the resources, the utilization, the access, and the derivative values such as content, and behavioral patterns. Another part of the answer is ‘control’ over licensing. To put it bluntly, vendors have grown tired of waiting for customers to own up to proper licensing.
The overhead placed on the vendors is costly. Putting applications into a cloud hosting paradigm removes a ton of the logical overhead (licensing, upgrades, assignments, maintenance), and replaces it with resource overhead (data centers, resource allocation, bandwidth, security). Instead of selling a $5000 license of a product with a single-use key, and worrying about you installing it 5000 times, they control the installation targets: the cloud hosts or virtual guests. Now they know for certain at any time, who is using which products, which versions and how often.
In my opinion, everywhere you can find inventory problems in the IT world you can expect vendors are looking for ways to either reduce profit loss, or gain new revenue streams from changing how it’s managed.
During my time with that customer, I also built several process-automation applications aimed at cleaning up these discrepancies. The powers that be decided they didn’t like attention focused on that, and insisted those be disabled and removed. After leaving, I received some inquiries about how to maintain or restore some of those processes, but without serious management buy-in, such efforts are doomed. It really comes down to the business culture, and the top-level people that drive it. You can build the coolest, prettiest, fastest car in the world, but if you let a drunk Stevie Wonder get behind the wheel…