Everything you need to know about patch management

What is patch management?

Patch management ensures the availability, security and performance across all devices or endpoints in the enterprise.

Patch management ensures the availability, security and performance across all devices or endpoints in the enterprise.

Patch Management is a basic requirement for a robust operations practice. It involves distributing updates to operating systems and software across a wide variety of computing environments. While a core portion of patch management involves updating software so new features and functionality can be made available to a user base, a critical aspect of patch management is the ability to deliver patches rapidly as new security vulnerabilities are discovered and addressed. This aspect of patch management addresses a basic need for the Security Operations (SecOps) practice. 

IT patch management is needed across operating systems on computers (servers, desktops, laptops) and network gear, as well as for applications running on these devices. For MSPs, patch management can be a critical component for optimal endpoint device operation. 

IT patch management can be an intensive, time-consuming activity when attempted manually, so many organizations are now moving to automated or centralized patch management applications, ensuring they can touch every endpoint effectively.

Where do patches originate?

Patches are often bits of software or operating code developed by operating system developers, vendors of hardware or by a software application's vendor. They are designed to address vulnerabilities, known errors or bugs and release new features. These vendors create the patches to bring these changes to market more quickly, and in the case of vulnerabilities, some of them are urgently needed to protect systems and services. The IT patch management process ensures they reach the designated endpoints in a timely manner. It also must wade through hundreds of patches made available to ensure the most critical fixes are applied first.

IT patch management should leverage policies to address the urgency levels of different types of patches and prioritize them, including:

• New features/functionality: Routine patches with new functionality that is in high demand. While not critical to protect the enterprise, patch management programs will distribute these to endpoints, often to desktops and laptops, so that the business may leverage them. As some of these may include innovative features that offer a competitive advantage to the business, they should be deployed promptly even though they are not as critical as other patches.

• Security vulnerability patches: These are the most critical patches to be distributed as they fix weaknesses in software and operating systems that could be exploited by hackers to gain access to systems or otherwise interrupt service. A good patch management practice is able to address these rapidly to avoid loss of availability. Every organization needs a way to know of new vulnerabilities that have been discovered and whether they affect systems within the organization's digital environment. A good IT patch management program will interface with Security Operations (SecOps) practices to ensure that any vulnerabilities cataloged by NIST are addressed as quickly as practical.

• Bug fixes: When vendors have repaired a reported software defect or bundle of defects, they will release patches that can be applied to address these bugs. Prioritization of these can be cross-referenced against an internal known error database to ensure that those already affecting end users are prioritized appropriately.

Patch management ensures that patches can be prioritized and delivered where and when needed on a timely basis, regardless of the type of end-point. Without patch management software, it is difficult to manage the delivery of the right patches to the right environment and ensure every endpoint meets the requirements needed to operate the patched software or OS.

The importance of patch management

Patch management is critical to ensuring systems and services remain available and organizations remain compliant within their industry. Security and compliance have driven up the need for automated patch management as they require developers and consumers to give immediate attention to new vulnerabilities discovered in operating systems and applications. Organizations that report compliance to regulatory authorities also need to demonstrate that all endpoints are in compliance and have patched all known vulnerabilities, helping to ensure data is secured.

The vulnerability management aspect of IT patch management also prevents downtime. With the ability to distribute security fixes promptly, patch management prevents hackers from gaining access to the environment, stopping ransomware and other cyber attacks before they can bring down critical systems.

Patch management is also needed to ensure the business functionality of software and applications is kept updated so users and clients can take advantage of them, enhancing the end-user experience. Additionally, as some patches include bug fixes, patch management ensures software and application issues are addressed as promptly.

Why central patch management is important

There are two aspects to IT patch management that make it challenging in a medium to large enterprise or MSP:

• Ability to identify the assets that are impacted by a patch and deploy the patch only to those assets once they have been successfully tested

• Ensuring that all affected assets are updated by auditing the environment for successful application of the patch

Centralized patch management enables an organization to use automation to enable testing and distribution of patches to the appropriate end-points and then to check end-points for missing patches, so each and every device on a network is properly maintained. Centralized patch management uses workflow, policies, and rules to manage the patch management process, involving the appropriate personnel for any required approvals to address failures and ensure the process is adequately completed.

By centralizing the process, the organization ensures that endpoints are being properly maintained and those patch management policies are applied as documented and scripted across the enterprise as soon as possible after they become available. Without central patch management, organizations may miss critical patches or be unsuccessful in deploying them completely.

While ad hoc methods of performing patch management may have been acceptable in the past, the size, complexity and vulnerability of the digital environment now demand centralized patch management to be considered a requirement of a robust operations practice. According to the Ponemon Institute State of Vulnerability Response survey, approximately 60% of security breaches are the result of a known vulnerability that was not patched. The likelihood of missing critical patches is high with only a manual process.

Understanding the patch management process 

There are several steps to building a patch management process that come from understanding patch management, the origin and priority of patches, and the due diligence that must be performed before applying a patch. The more resilient the environment, the more quickly patches can be applied. For example, in a virtual server environment, after testing, patches can be applied to a group of VMs, observed and then completed so the affected service maintains availability even if the patch fails or otherwise disrupts service.

The first phase of building a patch management process is to understand and prepare the environment:

• Discover a baseline inventory of assets, including operating systems and software. This will require some form of asset management or discovery as part of the baseline needed for patch management.

• Standardize "gold standard" configurations. This includes selecting the best software version and OS levels and ensuring all systems are consistently configured wherever possible. Any exceptions need to be documented and considered in the patch management plan so they can be skipped automatically during deployment.

• Prioritize applications and systems by criticality to the business and document the order in which specific environments will be patched.

The second phase is developing the process that will be used to deploy the patches as part of a centralized patch management process. This phase includes:

• Creating a patch test program to ensure patching doesn't destabilize systems and applications when applied. This may require building a test environment to be used if one does not already exist. The goal is to be able to test patches as quickly as practical and to accommodate as many patches as possible in a change cycle.

• Creating/implementing a vulnerability management process to find known vulnerabilities and patches associated with them. The organization's SecOps teams can assist with this.

• Creating a mechanism to catalog and prioritize routine patches. This includes understanding vendor patch and release schedules, as these will be done routinely.

• Creating a roll-back plan in the event of patch failure.

With centralized patch management and patch management automation, scripts and policies can be used to automate the patching process, but to do so, the activities must first be documented as part of a patch management plan.

The patch management process is a lifecycle that an organization will use for their program, as shown below:

1. Discovery/Asset inventory: Build an inventory of assets, categorize them by type, and standardize the software and OS versions for each.
2. Discover and document patches: All vulnerability patches and routine vendor patches must be cataloged and categorized for the environments in which they will be deployed (continually adding to the list as new patches are released).
3. Prioritize patches: Patches and the environments to which they are deployed need to be prioritized for testing and rollout.
4. Test patches: Before deployment, all patches need to be tested to ensure they don't introduce issues that cause downtime or other failures.
5. Test results should be documented, enabling the automation of their deployment. Those requiring approval (by policy) should be sent to environment owners for approval.
6. On approval, deploy the patches.
7. Audit the results to ensure patches have been applied as expected to all devices within an environment and to document the results of any failures for improvement opportunities.

The advantage of documenting the patch management process as a lifecycle and developing a set of rules and procedures for approvals is that these can then be built into a centralized patch management application and automated. Operations personnel spend their time on value-added activities like reviewing and approving patches and managing exceptions. In a commercial or MSP environment, the IT patch management lifecycle needs to be performed for each client as part of their onboarding process. The patch management process should also be re-evaluated and/or re-executed anytime there is a significant change to services being supported.

Designing your patch management plan

The phases and activities of the patch management process should be documented and integrated into an overarching patch management plan. The patch management plan should include documentation on how each phase and activity will occur. Specifically, the patch management plan should document how patches and vulnerabilities will be documented, prioritized, tested, and applied. It is helpful to have a patch management application that helps manage this as patches are released, as the ability to manage these manually becomes an immense body of work in all but small IT organizations. 

To ensure the prompt installation of patches, the patch management plan should include agreements for turnaround times for testing and applying patches for each environment and for restoring service if a patch fails or causes downtime even after successful testing.

Given the possibility that certain patches could be incompatible with some environments, a formal exception policy and process needs to be documented and followed when conflicts occur, and the business must accept the risk.

A good patch management plan will ensure that the activities can be automated and that appropriate policy and or approval gates are in place. For example, once testing passes, does the patch need to be approved before it is automatically scheduled and applied? One thing to consider here is the criticality of the service being delivered by the environment being patched and the risk of applying the patch. Combining a criticality and risk profile to an impact score for each patch enables low and medium-impact patches to be automatically applied, while high-impact patches may require approval. It would be standard practice to align this with an organization's change management process.

Once this is in place, the patch management program should be piloted by selecting a single environment or group of environments and configuring the centralized patch management application, then testing the program. Each environment should be piloted as part of the implementation of the automated patch management process and application.

This can be a significant body of work by itself in large organizations, so it's important to engage the administrators and managers of the affected environments in the planning and implementation process, moving from a small pilot to full-scale implementation with the involvement of the right groups. Each group should understand how patches will be applied and their responsibilities in helping to manage the process, including assistance in validating services and/or rolling back patches in the event of a failure.

Continual improvement should be part of the patch management plan. When there are failures during patching, they should be analyzed to see which step in the process failed and why adjusting system policies and rules to fine-tune the centralized patch management application.

Business benefits of patch management

-  Improved system and service availability

-  Faster deployment of new features and bug fixes

- Ability to meet compliance needs

- Satisfied customers

The digital business environment relies on system stability now more than ever before, and rapid, automated patching makes this possible. Business benefits of patch management include:

Improved system and service availability: Preventing cyber-attacks and applying fixes for known issues addresses two of the largest areas for system instability. Vulnerability patching addresses a large risk for the organization, given the previously stated rate of attack for known vulnerabilities with patches available.

Faster deployment of new features and bug fixes: As vendors release new features or fixes for known errors, faster deployment of their patches ensures these reach end users more quickly.

Ability to meet compliance needs: Depending on the industry, organizations may need to prove customer data is secured and that the organization is doing all it can to protect sensitive information. The ability to deploy patches and document the results for audits can help prevent fines due to compliance breaches.

Satisfied customers: Patch management is critical to ensuring the business stays operational and can get the most out of its digital environment. Rapid deployment of new features and prevention of downtime enhances the customer experience, leading to satisfied customers.