3 ways to apply security patches in Linux

When shopping for something where all the choices look similar, the feeling of choice paralysis creeps over you. This can happen when shopping for electronics, computer parts, or a pair of shoes. Choice paralysis occurs when a person finds it very difficult and stressful to make a decision with the available information. Applying security patches in Linux is a relaxing and rewarding experience when you have confidence in your process. There are several ways to apply security patches in Linux, and choosing the best approach does not have to be confusing or complicated.  I’ll walk you through 3 ways to apply security patches in Linux, and itemise the pros and cons of each approach. In several minutes you will know exactly how you want to apply security patches in Linux, and you can spend more time thinking about which gizmos, gadgets, and shoes you want to buy.

This blog post assumes you are familiar with my talk about security patching to maximise uptime. If you missed the broadcast live, you can still register on LinkedIn and play it on demand. I spoke about:

•  the top 3 challenges associated with security patching,
• 6 core strategies for successful security patching,
• 3 risk-based considerations your security patching strategy should take into account, 
• and mitigating risks in your Day 0 to Day 2 journey. 

At a high level, your choices for applying security patches in Linux are: manual patching, patching via package managers, and automatic patching.

Manual patching

Manually patching security vulnerabilities is an interactive process, and requires an alert and experienced system administrator. Our vulnerability and remediation-focused patching approaches both use the Pro Client, which is available in the Ubuntu Advantage Tools package.

To install Pro Client, run:

sudo apt-get update && sudo apt-get install ubuntu-advantage-tools

Vulnerability focused patching

sudo pro fix <CVE#>

Common Vulnerabilities and Exposures (CVEs) are reports that contain information about security issues in a specific version of a software package. If you know the CVE number of the security vulnerability you wish to resolve, you can use Pro Client to fix that specific vulnerability. For example, if your system was vulnerable to CVE-2021-3583, you could run this command:

$ sudo pro fix CVE-2021-3583

CVE-2021-3583: 
A flaw was found in Ansible, where a user's controller is vulnerable to template injection. This issue can occur through facts used in the template if the user is trying to put templates in multi-line YAML strings and the facts being handled do not routinely include special template characters.
This flaw allows attackers to perform command injection, which discloses sensitive information. The highest threat from this vulnerability is to confidentiality and integrity.
https://myasnchisdf.eu.org/security/CVE-2021-3583
1 affected source package is installed: ansible
(1/1) ansible:
A fix is available in Ubuntu Pro with esm-apps enabled.
The update is not installed because this system is not attached to a subscription.
Choose: [S]ubscribe at myasnchisdf.eu.org [A]ttach existing token [C]ancel

If your machine does not have Ubuntu Pro enabled, follow the prompts and resolve your issue. You can follow the Get started with Ubuntu Pro tutorial if you need a more detailed walkthrough of this step.

In situations where you know what vulnerabilities you are looking to patch, this is an excellent way of patching one-off CVEs.

What happens if your production workloads are impacted by a regression defect introduced by a security patch?  Bug fixes that resolve defects introduced in the security patching cycle are not tracked via CVEs. To track bug fixes for security patches, track the associated Ubuntu Security Notices (USNs). USNs are patch notices published by Canonical when a security patch is available for software installed on Ubuntu.

Remediation focused patching

sudo pro fix <USN#>

USNs will track the security patches, and the follow-on remediation bug fixes, if any bugs were introduced by the patch.

$ pro fix USN-5315-1

USN-5315-1: Ansible vulnerabilities
 Found CVEs:
 https://myasnchisdf.eu.org/security/CVE-2020-10744
 https://myasnchisdf.eu.org/security/CVE-2021-3583
 https://myasnchisdf.eu.org/security/CVE-2020-1733
 https://myasnchisdf.eu.org/security/CVE-2021-3620
 No affected source packages are installed.
 ✔ USN-5315-1 does not affect your system.

Pro Client can identify if the patch needs to be applied, and takes no action if an action is not necessary. All the benefits of vulnerability-focused patching apply for remediation-focused patching. There is a chicken and egg problem with both vulnerability based patching and remediation focused patching: how do you know what security vulnerabilities  are impacting your system, and which security patches are available?

Package focused patching

The first step towards knowing what CVEs impact your system, and what USNs are available, is having an inventory of installed software. A list of software names and versions when formatted in a machine readable way can be referred to as a software manifest. This software manifest can be analysed against an Open Vulnerability and Assessment Language (OVAL) data feed that tracks vulnerabilities in software. The results of this analysis can be exported to an HTML report, or a machine readable XML file, using a command line tool called oscap, which is available in the libopenscap8 software package:

sudo apt install -y libopenscap8

wget https://security-metadata.canonical.com/oval/com.ubuntu.$(lsb_release -cs).usn.oval.xml.bz2

bunzip2 com.ubuntu.$(lsb_release -cs).usn.oval.xml.bz2

oscap oval eval --results $(lsb_release -cs).xml --report $(lsb_release -cs).html com.ubuntu.$(lsb_release -cs).usn.oval.xml

oscap is useful when managing a single machine or image, but people typically need to manage more Ubuntu machines. There is a simpler way to obtain a list of CVEs and USNs that impact all your machines, and it works easily for any number of machines at enterprise scale: Landscape. Landscape is Canonical’s systems management dashboard, and is available in a self-hosted edition, or in a software-as-a-service model.

Landscape provides a listing of packages for any cross section of machines in your Ubuntu estate, and identifies which grouping of machines have similar upgrade journeys. If any machines have CVEs which can be resolved via USNs, the CVE numbers and USN numbers are listed on the package page associated with your selection of machines.

Package managers

It is considered best practice to install packages through package managers when possible. On Ubuntu, 2 popular package managers are apt and snap. Interestingly, it is possible to use a management tool like Landscape to manage your package managers at scale.

Apt

sudo apt update && sudo apt upgrade -y

Canonical publishes tens of thousands of open source software packages across several repositories: main, universe, and multiverse.

Running apt update and apt upgrade will download the latest software packages from the repositories. Repositories are divided into pockets, and all the security updates exist within a security pocket. It is possible to identify which pocket the software update is coming from by looking at the output of:

apt-get cache policy <package-name>

The following command will install only the security updates:

sudo apt-get -s dist-upgrade | grep "^Inst" | grep -i securi | awk -F " " {'print $2'} | xargs sudo apt-get install -y

For organisations that mirror repositories internally, it’s worth cautioning that relying on this approach requires an up to date repository. If you are using the default repository configurations provided in Ubuntu, and they are accessible over the Internet on your machine, your package managers will always have the latest security information. If your Ubuntu machines look at a repository mirror you manage, the package manager will only be as up to date as your repository mirrors.

Snap

sudo snap refresh

Canonical also distributes software packages as Snaps, which are distributed as tamper-proof, GPG signed, compressed, and read-only filesystem images. Software packages delivered as Snaps are very secure because they run in a sandboxed environment, and system access is denied by default. The System Administrator controls what a snap package is allowed to access, through pre-defined snap interfaces. Snaps are attractive from a software distribution standpoint because they use less disk space, by virtue of being file system images, they don’t need to be unzipped for installation because they are mounted. Snaps use less bandwidth when applying security patches, only the bits that have changed need to be downloaded, and Snaps have the ability to rollback automatically if the software upgrade fails.

Running snap refresh will manually trigger updates for all your installed snaps.

Landscape

While Landscape is not a package manager, its capabilities encompass managing package managers across selections of multiple machines. Beyond this functionality, Landscape also provides web based systems management controls, with an interface to manage Inventory, Policies, Profiles, Processes, Automation, Alerts, Monitoring, and Reporting. These management functions are available through an API, accessible via HTTP and a command line utility, with human and machine readable outputs.

Automatic updates

Security Professionals looking to strike a balance between security, usability, and availability, must leverage automation. Canonical provides a turnkey security patching solution that works in even the most restrictive environments, through Snaps, Livepatch, and Landscape. Examples of security patching automation include self-updating Snap packages, Canonical’s kernel Livepatch service for high and critical updates, and Upgrade Policies in Landscape. All 3 of these automations reduce the time required for software patching maintenance windows. Patching automation can be done intelligently, by dividing the Ubuntu estate into groups which can be updated independently of each other. Landscape is your tool for segmenting your estate and managing your automation effectively.

Livepatch

sudo pro enable livepatch

Livepatch is Canonical’s kernel security patching service which eliminates the need for unplanned maintenance windows for addressing high and critical severity kernel vulnerabilities by patching the Linux kernel while the system is still running. Once Livepatch is enabled, the livepatch client will receive and apply security patches after they are made available. The livepatch client will replace vulnerable code in memory, while the machine is running. It will provide this automatic patching capability for 10 years on Ubuntu LTS with an Ubuntu Pro subscription.

Snap

Snaps can be pinned to a specific version and refreshed manually. By default, Snaps are not held at a specific version for a duration of time. Instead, the publisher is in control of your upgrade timelines, and when they publish a new version, it is visible to machines everywhere in the world. This is ideal for high risk software where security updates need to be rolled out in a time sensitive manner, such as in web browsers and email clients. The Mozilla Foundation publishes the Firefox snap, which is the default browser on Ubuntu. Canonical also publishes a Chromium snap, along with many other Snaps which are relevant in desktop and datacenter use cases alike.

unattended-upgrades

The unattended-upgrades package is over a decade old, and is enabled by default on all currently supported versions of Ubuntu. It is intended to install security updates automatically, by default, so that machines that are not proactively maintained do not become a liability.

Landscape’s “Upgrade Profiles”

Landscape’s Upgrade Profiles functionality is similar to unattended-upgrades, but it provides more granular controls, and works very well at scale. Some of the additional features Landscape’s Upgrade Profiles offer are scheduling, reporting, and staggered delivery windows. It’s also possible to associate different Upgrade Profiles to different selections of machines, based on a cross section of their defined Access Group and assigned Tags.

In conclusion

There are 3 approaches to applying security patches in Linux:

1. Tracking each security vulnerability, and the corresponding patch notice. Once the vulnerability is identified, apply the appropriate software patches through tooling that is aware of CVE and USN numbers, such as Pro Client.
2. Tracking and applying available software updates through package managers. This requires connectivity between your machine and Canonical’s repositories, for instance, or configuring your machines to reach an up to date mirror of Canonical’s repositories.
3. Lastly, automated updates are another means of applying security patches. Utilities like Livepatch and Snap ensure your updates are applied on your behalf. These tools can be configured to update your systems in a staggered manner, instead of all at once. While unattended-upgrades cannot be scheduled, Landscape provides an equivalent alternative that scales incredibly well, when security patches have to be applied in an automated manner across many machines.

If you need security patching that can be automated at scale, and audited on the fly with on-demand reports, Landscape has your bases covered.