What if a SCOM Update Rollup is very slow

Had an interesting case today where we were implementing Update Rollup 8 for SCOM 2016 on a management server and it took very long. It took so long that we decided to kill it and start it again. And again it seemed to hang.

 
So reboot and try again. We had difficulty finding the log file that the UR was creating, so we decided to start an elevated command prompt and go to the place where we stored the patch file and run the command from there and specifying the log location.
msiexec.exe /lv install.log /p KB4514877-AMD64-Server.msp
Next we took the Configuration Manager Trace Log Tool to open it, because it will follow whatever gets written there in real time.

Now it was becoming more clear what it was doing. It was going through the list of agents to try and put them in Pending Management (for agent update). Check in SCOM:

And indeed it was slowly going through the list and placing the agents belonging to this management server into maintenance mode. This server alone has about 1200 agents. So this took a while. Now we knew what it was doing we could let it finish.
We would question though why it took so long. We never seemed to have had this problem before on this system.
Also what strikes me is that I expect this workflow to go through the agents list, for the agents belonging to this server as primary server, and determine if the agent is set to remotely manageable. And if it is, to check the agent version and if it lower than the new UR version of the management server to put it in pending management for Agent Update. In this case though it seemed to add all agents to the list, and also the agents are already at a higher version than the server (we have SCOM 2019 UR1 agents there). Maybe that pending agents process is not as it should be. Anyway, it took a long time. The first 300 agents appeared in the list within a few seconds, but the rest took much longer.
If you run into the same, remain calm and check the SCOM console pending management and wait a minute to see if the number in pending management increases.
We still have a few management servers to go for this environment and will plan accordingly that it does not take 5 minutes per server, but a bit longer in this case. 2.5 hours actually for this one machine. Thanks to Luis for his remark that I could better spend the waiting time to write a blog post about it.
Good luck! Bob Cornelissen