If the division of the Microsoft Corporation that handles the development and release of Windows Updates for desktop and server operating systems and services, such as MS Exchange, were a living and breathing person I’d be seriously concerned that they’re suffering from early onset Alzheimer’s. Over the last few years I’ve witnessed and read about how these updates we’re supposed to trust to be vetted and approved safe to be installed are breaking systems rather than enhancing them or fixing previously uncovered problems. In many ways these updates that are wreaking havoc on production systems are very much like the malware being released by nefarious groups or individuals for the sole purpose of making computing life miserable for the rest of us., hence the title of this post.
The title, as you can plainly see, is a not so veiled accusation of Microsoft’s inability to make sure updates aren’t going to break more than they fix, or more preferably not break anything at all! That is, after all, the reason updates are released in the first place: to fix something that has the potential to cause problems. However, more and more lately that doesn’t appear to be the case. So, you might be wondering what I’m raving about? Let me state it plainly then.
Most MSP’s (Manged Services Providers) – myself included – rely on systems like Kaseya or Solar Winds RMM to manage a number of systems remotely for our customers. One of those services we provide is to push out updates available for installation to those remote system to make sure that the systems we care for are up to date and fully patched. Some MSP’s have their systems configure with a policy in place to automatically push out all updates marked critical or important, while some – such as myself – take a more hands on approach and do it semi-manually. I’ve been bitten a few times in the past and I like to at least know what the patch is for before i approve it for installation. Especially on Exchange servers that always seem to be so sensitive to changes. Either way, which ever method is used we are placing trust in the vendor – in this instance Microsoft – that the patches/updates are safe to be installed; especially when they’re marked critical or important. (generally anything released that is marked as a security update it considered important enough for the appropriate action to be taken). As I’m sure you’ve guessed by now I got bit again. Thankfully, it wasn’t one of my clients that took the hit but one of my servers. More specifically, my Exchange 2016 server.
I know… some may be asking, “Why were you installing an update on a live system rather than installing on a test system?” Well, to be honest I don’t maintain test systems for such things, but more to the point my customers certainly don’t. Certain things like Exchange CU installs I always perform on my own server first to make sure they work before doing the updates for clients and having a virtual environment helps with that immensely, however I wasn’t expecting a mere security update to do the kind of damage it did when the installation failed.
Again, thankfully the Exchange server is a VM running on a Linux platform using Oracle’s Virtual Box and as I’m writing this post I’m restoring an image of the VM from yesterday before the update. But, just the fact that I have to do this to bring my Exchange server back online is an unnecessary burden placed on myself and so many others by Microsoft and their Malware-like patches. The patch in question is Security Update For Exchange Server 2016 CU10 (KB4459266). The general description for this patch is:
This security update resolves vulnerabilities in Microsoft Exchange. To learn more about these vulnerabilities, see the following Common Vulnerabilities and Exposures (CVE). Further information can be found here if you’re interested: https://support.microsoft.com/en-us/help/4459266/description-of-the-security-update-for-microsoft-exchange-server-2013
Well, bless their hearts… While the update was attempting to resolve said vulnerabilities the installation failed and completely hosed my Exchange Server rendering it completely inoperable. Thank you very much for that Microsoft. I did have important things to get accomplished today, but instead I’m sitting here waiting for the VM image to be restored so I can bring Exchange services for my company back online. I figure till it’s all said and done I’ll have about 5 hours into the process. Perhaps I should invoice them for my time. It’s a thought but I’m fairly certain that nothing would come of it. So, I’ll just add one more post to the already hundreds of thousands that already exist on the internet about such problems network admins around the world are experiencing with this very same issue. The damned patches are breaking the very things they’re supposed to be fixing!
Normally, when I see a failed patch on a server or workstation it’s that too big of a deal since 9 times out of 10 I simply download the patch directly from the MS Update Catalog and install it manually on the affected machine. However, in this instance that wasn’t an option because, as I mentioned earlier, that failed patch installation completely nuked the Exchange server and there wasn’t much left to recover, so a restore was the only option left for me.
So, here I sit with 1 hour and 17 minutes remaining until the VM recovery process completes from the backup media and I get to find out if the rest of the process is going to be smooth and full of bumps and bruises for the remainder of the day. I had read recently that there was discussion at Microsoft about slowing down and curtailing for a time the release of updates because so many were causing problems. I can see how doing this would be a potential problem. And, in fact, that’s exactly what they did when the processor vulnerabilities were discovered earlier in the year because those patches were really causing a lot of problems.
Strange… I keep hearing Pink Floyd’s song Money playing in my head… Perhaps I should go to YouTube and locate Comfortably Numb and get it playing instead.