Patching is the process of updating software. The task is particularly important for updating operating systems and their services. Operating systems and many software packages have their own update processes. However, network security blocks those routines. So, while home users can leave their systems to update themselves, business users have to create new procedures to substitute for that lost utility.
RMM packages provide automated systems to monitor an entire IT system and that includes its software. Patch management is classified as an endpoint management task. The system isn’t straightforward because it requires quite a lot of preparation. However, these preparatory routines can be included in RMM packages as additional system management features.
Software management in an RMM
The remote management functions of an RMM start with the documentation of existing assets. Network monitoring services start with a discovery cycle. This feeds into the asset management functions of an RMM because it creates a hardware inventory. That list of equipment includes all the endpoints connected to the network.
The software management services in an RMM rely on the knowledge of what endpoints are on the network. This includes both workstations and servers. Once all devices have been documented, the RMM can scan each endpoint for software.
The endpoint scan creates a software inventory for each computer. This list is all entered into a single database for all computers. Thus, information on the software present on the site can be filtered per device or summarized per software package. This enables RMMs to provide software license management very easily. The system administrator needs to enter into the system the number of licenses available for each software package. The software inventory can then show which software still has spare licenses available, and if any have been installed in more companies than are allowed.
Software versions
The software inventory includes a record of the operating system installed on each endpoint. While there are few options for operating systems, identifying each computer’s installation is important because not every operating system will be up to the same version. Ultimately, the intention of automated patch management is to ensure that each software package and operating system installation is up to the latest version. However, when an RMM is first installed, it is likely that every installation has a slightly different version.
Software versions are a lot more complicated than the average person realizes. You aren’t just dealing with version 1, 2, or 3, or even 1.1, 1.2, or 1.3. Version numbers are a lot more informative. For example, take a look at the About page of your Chrome browser – you can get to it by clicking on the Hamburger menu and looking under the Help menu entry. You will see a version number like 123.0.6312.106.
The version number of each piece of software and also of each operating system installation identifies its patch status. When the software gets updated, that version number will increase.
The purpose of patching
You might wonder why every operating system and piece of software needs to be updated so frequently. It seems as though software houses are releasing products that are full of bugs. Can they really be so full of faults that they need to be updated? What’s more, it seems that as soon as they have fixed one error, they discover another.
Most of the patches that get issued aren’t because of coding errors that get discovered after the software is released. There are some updates that are released for that reason, but they tend to emerge early on in the software’s service history. This could be because the software performs differently once it is running than it did in the enclosed system of the testing environment.
The main reason for the release of patches is to close down security weaknesses. A software house can produce a product that has been created with knowledge of all potential threats, and then test it with the most up-to-date security testing systems available. The producer will probably scan the software for vulnerabilities and observe activity during its beta release. However, new exploits can emerge later.
A new threat is known as a zero-day attack. That refers to a new attack strategy that no one has tried before. As it is an innovation, there is no way that the software house could have tested for vulnerability to that technique at the time that the software was first released.
Software companies keep track of the activity of their software and will also receive complaints and warnings from buyers. As soon as a new vulnerability has been discovered, the software house needs to work on an update. The improvement will be integrated into the product so that new buyers will be protected against this new attack. However, every copy of the software that has already been sold needs to be updated to account for the problem. This update is implemented as a patch.
The importance of patching
Many private computer users turn off updates wherever possible because the process can slow down the computer, freeze the software, and cause the computer to reboot unexpectedly. However, that is a risky strategy because in many cases, the patch is the only way the software house can ensure that their products are as protected as possible.
Updates present a security risk because there is an obvious mechanism that hackers could use to bypass firewalls and get into a system. Therefore, firewalls generally block update mechanisms. Therefore, there needs to be an authorized method to update software and that is where patching comes in.
As an RMM system is supposed to supply all the major monitoring and management tasks needed to run an IT system, the package would be seriously lacking if it didn’t include a patch manager. The assessment of installed software for patch statuses is usually regarded as part of a vulnerability manager. However, RMMs typically include that function as well as an IT asset management service.
Device configuration security assessments often also form part of a vulnerability manager. However, that function isn’t frequently included in RMM packages. RMM packages are expected to monitor system performance and not system security.
Patch gathering
Surprisingly, very few RMM producers explain in detail exactly how their patching services operate. Without exception, the basis for all patching systems is to maintain a software inventory. The next step is to check the latest patch that is available. That stage is obscure.
The only RMM producer that explicitly states its patch availability process is ManageEngine. That company gathers all patches from operating system creators and a list of software producers. So, each instance of the ManageEngine patch manager just has to check the ManageEngine cloud server for patch availability.
The patching system of ManageEngine is available in several products:
- Vulnerability Manager Plus
- Patch Manager Plus
- Patch Connect Plus
- Endpoint Central
- Endpoint Central MSP
- RMM Central
The patch manager in each of these systems is actually the same tool. It gathers patches for Windows, macOS, and Linux. Each of those operating systems has multiple versions that are current. For example, Windows 7, 8, 10, and 11 are still in circulation. The Linux distros that the system will update are Ubuntu, CentOS, RHEL, and SUSE. The tool also stores updates for more than 850 software packages.
Each installation of the patch manager, bundled into the various ManageEngine modules, will periodically check on the server, getting a list of version numbers, which it compares to the version numbers in its software inventory. If the patch availability report number is higher than the version number of the installed instance of a particular software package, the patch manager downloads the installer for the most recent update.
Problems with patch managers
ManageEngine’s website is very helpful because it goes into great detail about its patching system. Other RMM producers are less clear about their patching process, and that’s understandable. ManageEngine, by being open about its patching service, has probably lost itself some potential customers.
Any company that runs Debian Linux is not going to buy any of the ManageEngine products that include its patch manager because it doesn’t cater to them. There are many other Linux distros that ManageEngine doesn’t patch.
Buyers would also have to run down that list of 850 software packages that ManageEngine will patch. What if one of the essential systems that they use isn’t on the list? Do they buy that patch manager that covers most of their software and make other arrangements for that one, unpatchable package?
With other patching RMM systems, the list of software that will be patched is not available before purchase. So, a company could get stuck with a patch manager that won’t patch all of their software. Another issue is that a buyer could implement extensive research to ensure that all of their software will be covered by a new patch manager. Only to find that a subsequent software purchase could be outside of the capabilities of that patcher. Selecting a patch manager will limit your choice of software in the future.
Patch files
There is no standard format for a patch file. A patch file for a scripting language program, such as Perl or PHP is a text-based file that lists the lines of code that need to be changed. This seems a strange way to go about updating software because it is prone to error. Surely, it would just be easier to present a new copy of the program so it could just be used to replace the existing file. However, the program file can be updated with the new lines by a short command line utility.
Windows utilities are updated by MSU files. This file format has four sections and its own special syntax, so it can’t be run as an executable. It is, in fact, an input for the Wusa.exe program. So, the update is implemented by copying the MSU into a temporary directory and then running Wusa.exe, giving the file name with its path as a parameter.
It is possible that there will be several Windows patches available at the same time. It is also common that a Windows patch file will update many programs. So, one patching session can update many files. Not all updates will change programs; some update data files and some alter environment settings or registry entries.
These two examples show the range of patch files that run from very simple text files to complicated scripts that update many files through a specific update program. Each software house can create its own update file formats and patching processes. So, a patch manager needs to be able to apply patches in many ways and know the way to run each update for each software package.
Patch scheduling
RMM patch managers are schedulers. The functions performed by the patch managers are set up as scripts and then set up to run at a specific frequency in the scheduler. Some RMM packages, such as Atera, make that scheduling facility available for other processes. The administrator can load a script into the patch manager screen and set it up to run at a specific time or regularity. This can be used for tasks that have nothing to do with patching.
The patch manager of all RMMs needs to be given execution times. That is, the patch manager should be authorized to check for the availability of patches. It should also be given a calendar that defines when it can apply patches.
The two periods in the scheduler – the patch availability check and the patch rollout, do not need to be synchronized. The patch availability check will copy down the patch installer and queue the patch. The patch execution service will run automatically when the next valid period arrives. All patches that are in the queue will be applied.
The presence of the patch queue provides opportunities. While a patch is in the queue, it can be dealt with manually. For example, most patch managers allow an administrator to delete a patch or put one on hold. If a patch is left in the queue but paused, the patch manager will run all the other patches and leave that one in the queue.
Patch validation
The delay between identifying patches and applying them provides a period during which patches can be checked. This process involves running the patch in a sandbox. The idea behind this secure test is that it will update a copy of the software, run it, and test it for security weaknesses or unintended consequences. For example, you would need to ensure that the updated program won’t delete all of your data.
One problem with the sandboxing concept is that it won’t entirely model your current system. It doesn’t take into account that a specific piece of software will interact with other systems. That factor is particularly important when updating operating system components. No commercially available patch manager fully accounts for these potential problems.
Some patch management systems pre-validate patches. ManageEngine does this. As soon as the central patch server acquires a new patch file, it tests it. The service will only store the patch into its library to make it available for customers if it passes that sandbox test.
Patch priorities
Some software producers categorize their patches in terms of their importance. Microsoft does this. The category levels refer to how essential the patch is. The levels used by Microsoft are broadly accepted as an industry standard. The levels are:
- Critical
- High
- Medium
- Low
You would expect your patch manager to display this priority level. The exact details in the patch queue vary from system to system, so it isn’t guaranteed to include the priority.
Patch dependencies
Some patches require that other patches have already been applied before they will run. This is usually a bigger issue with systems that are just coming into a formalized patch regime. It should be assumed that a system that has been operated by an RMM with a patch manager should be up-to-date with all patches. Therefore, any new patch will already have all of its pre-conditions present before it is listed for rollout.
There are some cases where patch dependencies are not autonomous. That is, they don’t refer to the requirement for all previous patches of the same package to have been applied. This can occur when a piece of software relies on supporting frameworks or APIs. Take, for example, a piece of software that relies on the presence of the Java Runtime Environment. It could be that the update for the dependent software requires an upgrade to be made to JRE before the new version of the system will run.
Dependencies aren’t always automatically taken care of by the patch manager because they can be folded into an installer without any mention of the spawned installation program in the patch documentation. These integrated third-party patches can be annoying because some of them will prompt the user for permission and then download the extra update from another source. This scenario would cause the patch installation process to hang because technicians expect a patch manager to run independently, unattended, out of office hours.
Issues such as unaccounted-for patch dependencies can cause a single patch to abort or even cause the entire patch queue to be held up. Some patch managers have a timeout mechanism that will kill a patch if it is inactive for too long; others will just hold up until a human gets to the patch manager console and kills the paused patch manually.
Completion statuses
Every patch in the queue will run in sequence. Once one patch has been completed, the patch manager will activate the next patch in the list. Patch managers don’t consolidate patches, so if several patches are being applied to each computer, the entire process could involve numerous reboots before the entire list is completed.
One patch might fail and cause the entire patch run to fall over. A good quality patch manager should be able to mark a failed patch as being in error and then move on to the next patch in the list. Some patch managers show all patches in a batch after the full list has been applied. Others will remove each completed patch from the list, leaving only those that have been put on hold or that have been run and failed.
In all cases, the patch manager will write the patching activity to a log file. This will include a record of the completion status of the patch installation. The administrator should also be able to see what supplementary activities took place, such as alteration to the environment, additional installations, or reboots.
The patch manager should remove the patch installer from its temporary storage upon completion. Some will leave those patch installers that didn’t complete successfully.
Manual intervention
The patch queue gives the administrator an opportunity to intervene and possibly hold off or remove a specific patch. Some patch managers allow administrators to specify that only patches above a given priority level should be applied.
The manual decision over withholding patches can be a reason why patch dependency requirements become a problem at a later date. One patch that was marked as relatively unimportant could end up becoming a prerequisite for a later patch – it could have grown in importance over time.
Patch managers should allow the administrator to command a patch to run immediately, rather than waiting for the next maintenance window. This would be necessary if an important patch failed and the company operates a relatively infrequent patch schedule, such as once a month. In this scenario, the administrator wouldn’t want the system to go without that important update for an entire month.
Choosing an RMM
Some RMMs don’t have a patch manager; others will only patch Windows. It is conceivable that you will choose to buy an RMM and then buy a separate patch manager. For example, N-able N-sight and N-able N-central both include a patch manager that only updates Windows. If that isn’t good enough, then you are going to have to buy a patch manager that has wider capabilities. If you have a standalone patch manager, it would be better to let that cover your Windows systems as well as all other patching requirements. For this reason, the patch manager in N-able bundles can be turned off.
Ensure that your chosen RMM is able to patch all of your software, and be aware that the list of patching capabilities of your RMM will influence all of your future software purchases.
It is difficult to get a list of all the systems that a given patch manager will update before you buy. That is why it is very important to select an RMM that has a free trial or a refund period. You should be able to test your patch manager to ensure that it patches all of your software. If it doesn’t consider trialing a different RMM.
