Back in May 2016, Microsoft issued a blog entry on TechNet giving the world insight into its new patching strategy. The concept of a monthly “rollup” patch or what a lot of people are calling a “mega-patch”. In August another blog entry was posted that further detailed this strategy and explained that from October 2016 going forward, this is how Microsoft would patch Windows.
But there is even more to it. For WSUS and SCCM users, security patches will be separated from the Monthly Rollup in their own Security mega-patch. The idea behind separating the security patches into their own mega-patch is to allow organizations to at least stay current on security. However there is a twist on this approach as well. Organizations such as small business that do not use WSUS or SCCM will only get a single mega-patch through Windows or Microsoft Update that will contain the Monthly Rollup and Security mega-patches in one mega-patch.
So what could go wrong you might be asking?
The biggest drawback to this scheme is that, should you have any issue with a mega-patch, you must back out the whole patch, not just the item that is creating the issue. That means instead of having just one potential issue to mitigate, you could have as many issues to mitigate as the patch contains. From a PCI compliance perspective, that could mean lots of missing patches in your Windows systems if your systems run into an issue with a mega-patch. This can get doubly bad for organizations not using WSUS or SCCM because they will be backing out security patches as well as application patches.
But it can get even worse. These mega-patches are cumulative meaning that every month Microsoft adds the previous mega-patch to the new month’s mega-patch. For example, say one month the mega-patches cannot be applied for compatibility reasons. For example, you apply the monthly mega-patch and your point of sale (POS) application fails to work with the mega-patch and you must back it out. If that issue continues because of your vendor, you will not be able to patch your POS systems until that compatibility issue is resolved because month after month the mega-patches are cumulative. So until the compatibility issue is resolved, you will not be able to patch your systems.
But I foresee small businesses running into the worst issue with this new approach. Since small organizations likely will not be using WSUS or SCCM, they will not get a separate Security mega-patch, they will only get a single mega-patch that combines the Monthly Rollup and Security into one mega-patch. If any issue occurs with that single mega-patch, the small businesses will not even get their security patches. That will create a situation where the organization must figure out how to mitigate their inability to secure their systems. In addition, that could mean months of security issues until the original compatibility issue can be resolved.
But to add insult to injury, I can also see situations where a vendor has issues resolving a compatibility problem with a mega-patch and finally gets it fixed only to encounter a new compatibility issue with the latest mega-patch. Based on how Microsoft is running these mega-patches, there appears to be no way to go back to a compatible and useable mega-patch. This could result in organizations being unable to patch at all due to ongoing compatibility issues.
At a minimum, I think Microsoft will need to make the Security mega-patch separate from the Monthly Rollup for all organizations, not just those using WSUS or SCCM. At least then, all organizations can apply security patches independent of the Monthly Rollup which would be more likely to be the one that would create compatibility issues.
It will be interesting to see how this new patching strategy plays out. Hopefully it does not create even more risk for uses of Windows. If it does, I would not be surprised if the PCI SSC invokes significant new controls on Windows-based solutions. That could be the final straw in using Windows for a lot of merchants. Time will tell.
Since Microsoft has NEVER EVER gotten its Update system to work properly – with a lot of people having updates fail virtually every month – or worse, bricking systems so they won’t boot – this is going to be a major disaster. Microsoft seems to have mastered the art of shooting themselves in the foot.
While I see your point and agree with you that this is going to create a ton of headache for SMBs, especially in the nonprofit sector, it’s also worth noting that WSUS is a free addition for organizations running Windows Server 2008+, and is included as a role in Windows Server 2012+. So you might be able to get away with adding that to your DC, and then using it to manage patches.
Is it foolproof or ‘easy’? No – it does require a little bit of elbow grease and research and work. It’s a burden, for sure, but I’d rather orgs deal with the headache of having to stay on top of patches than ignoring them entirely.
Did not know that WSUS could be a role add-on, so will have to try that out. But you are right, WSUS is not an easy tool, so SMBs will likely have their hands full.
Sure you can add WSUS as a role add-on, but be very careful as it will eat up storage in large gulps. Domain Controllers don’t like running out of disk space. I’d recommend a secondary drive is used, and if it’s just for WSUS heck a USB/eSATA external drive would suffice as redundancy or fault tolerance may not be a concern. I’ll also point out that if you want to serve updates to Windows 10 PC’s, you’ll want to use WSUS on Windows Server 2012 R2 or later – I understand that new Windows Server 2016 has a better WSUS (but I haven’t actually set one up yet).
I’m not holding my breath on this new patching. I sort of understand their position of attempting to make the systems consistent, but their track record on patches has been less than optimal. I for one will have to go back to the days of testing on a small number of systems and likely make WSUS updates a manual approval.
Good advice and comments about WSUS. It is definitely not for the faint of heart or uninitiated which is why I am concerned about neophytes trying to go that route just to gain back some control. I also agree with your last remark to a degree. Manual approval will mean even more non-current systems in the inventory. However, at least with WSUS, you will have an idea as to where those systems are located.
“Sure you can add WSUS as a role add-on, but be very careful as it will eat up storage in large gulps.”
Yes, this is definitely true. The idea of storing WSUS updates on a secondary drive/external drive is a really good one.
It’s also a good reasons for NPOs to standardize on the same OS/Office software/etc. If an org. is patching two different Microsoft OSs AND two different versions of Office AND other products, you will end up sacrificing all kinds of space. Some NPOs I have seen don’t take that seriously and I am sure it’s a massive headache for their IT MSPs.