Lately a good friend of mine, Cyril Azoulay, ran into an interesting issue while in-place upgrading a relatively recent SCOM 2019 setup to 2022 version. Since his blogs are mainly in French, he allowed me to write down his case over here in my own words (since his blog is not out yet). Merci Cyril!
One of the errors during the SCOM in-place upgrade looked something like this (from the OpsMgrSetupWizard.log):
Always: :FirstManagementServer: CreateConnector in preparation for Networking MP import. Error:CreateConnector failed
We couldn’t recognize this either at the time. Meanwhile the upgrade was at a point of the old dll’s being removed and so on, so the roll-back did not give a well working system anymore either.
In fact, this error left the SCOM environment in an unrecoverable state and was totally broken. It required a snapshot rollback of the relevant machines or restore from backup to recover! Again, a reminder to always have backups and sometimes also snapshots before you start doing such major updates, just in case something goes wrong and you are left in a state where you cannot recover otherwise.
Cause
Turns out there was a root cause of this. Of course!
The issue was caused by references in a management pack of the currently imported management packs in the environment.
It used to be that Aliases in the references section of a management pack were case sensitive. So, you could have two aliases, which are the same, except for a capitalized character for example. Watch this example:
These aliases got created by the SCOM console itself while making overrides and such.
Since SCOM 2022, this is not supported anymore. And thus, while you upgrade the management group it would find a problem with this and cut off the installation.
Turns out Microsoft support was aware of this and there is a script to check if you have this “issue” with duplicate aliases (with different capitalization), which you can run to identify them and take action before trying to in-place upgrade your management group.
$mps = Get-ChildItem -Path “C:\Program Files\Microsoft System Center\Operations Manager\Server\Health Service State\Management Packs”
foreach ($mp in $mps) {
$mpText = Get-Content $mp.PSPath
$references = @()
foreach ($line in $mpText) {
if ($line -like “*Reference Alias*”) {
$references += $line.ToLower()
}
}
$ht = @{}
$references | foreach { $ht[“$_”] += 1}
$ht.keys | where {$ht[“$_”] -gt 1} | foreach {write-host “Duplicate element found $_”; write-host $mp.PSPath}
}
I was checking around and testing this all on my own demo systems.
Created a dummy pack and adjusted an alias Windows to WinDows, while the other one still existed as well.
This test pack imported fine on an old 1807 SCOM demo environment we have meant for testing our Health Check scripts and such. It refused to import on my SCOM 2019 though.
Where did this change?
After asking around, it turns out that also SCOM 2019 UR4 included some change to enforce this behavior. And SCOM 2022 of course as well.
I have not yet heard about any UR3 to UR4 upgrade going wrong because of this issue, but I can imagine it could happen. I do not expect this issue to occur in that many environments, but there is always a possibility.
We will try to build this into our SCOM Health Check scripts for a future build. We have had several customers run our check to confirm there are no problems right before an upgrade to a next version. And this sounds like just another reason to include such an item on our list! Check out SCOM Health Check | TopQore for more information on that service we offer.
While manually creating aliases, I hope nobody does it this way, and always makes sure the alias is different, by adding a number or by making sure its simply not the same word. This just happened to have been due to an automatically generated alias. I am sure this does not happen all the time, and was an exception.
Good luck to you all!
Bob Cornelissen