تاریخچه نحوه بروز رسانی کرنل لینوکس بدون ریستارت

Installing the latest Linux kernel used to mean a reboot, until the development of ‘rebootless kernel updating’, a method that patches servers without restarting them. With the technique now just over 10 years old, this article takes a brief look at its origins and current state.

2001–2010: The Patent Trail

If you trawl through the patent archives with keywords like hot patching, or live system updating, you’ll dredge up many applications and rejections showing that the idea of updating a computer system without stopping it is nothing new. The significant dates, tracing the idea from general to specific, are as follows:

• 2001: Hewlett Packard patents a method for dynamically updating software to circumvent missing hardware functionality.
• 2002: Microsoft joins the game with an approach to updating a system (Windows) without interrupting it. (Their initial application is rejected on the grounds of HP’s ‘prior art’.)
• 2008: Jeff Arnold announces Ksplice, software for updating (patching) a Linux kernel without interruption (i.e. without rebooting¹).
• 2010: Microsoft’s patent finally gets granted on appeal.

The interesting point about these is that they share the aspiration to rectify, with a software update, a fault in a system’s core software or hardware without affecting the continued running of that system and without altering the hardware. Sounds familiar? (Clues: Meltdown, Spectre.)

2009: The Birth of Rebootless

Jeff Arnold was an MIT student looking after one of their servers². It needed a security patch, but he delayed it because a reboot would inconvenience his users. Before the system could be updated, it was hacked. The disgrace and (ironically) inconvenience suffered inspired Jeff to find the topic for his Master’s thesis in the problem of performing a system update without delay and without rebooting it. The story may be apocryphal, but it reminds us that live patch techniques sprang from a concern not for convenience but for security, and it is in that role in which they should be appreciated.

Jeff Arnold teamed up with three student colleagues to study the problem of how to update a Linux server’s kernel, without delay and without interrupting the system’s processes. The solution came in the form of software called Ksplice, the technical foundations of which were laid out in a 2009 academic paper. The paper’s title included the word rebootless, now familiar Linux shorthand for ‘uninterrupted updating’, but first coined by Microsoft in 2005 to apply to Windows driver updates.

After graduating, Jeff and his MIT colleagues started Ksplice Inc., and in May 2009 they won the MIT $100K Entrepreneurship Competition prize. The company launched a commercial service in 2010; things were going well.

2011–2016: Oracle and the New Wave

On 21st July 2011, Oracle acquired Ksplice, Inc., integrating the software into their own brand of Linux, itself a derivative of Red Hat³. Despite that heritage, Oracle stopped supporting Red Hat. The acquisition of Ksplice by Oracle kicked off a surge of activity among other key Linux vendors left in the lurch.

Between 2011 and 2014, SUSE and Red Hat worked in isolation (and ignorant of each other’s goals) to release their own live kernel updating solutions, which they did in Kgraft and Kpatch respectively. (Despite their slight head start, SUSE’s Kgraft was only made GA (i.e. suitable for production systems) in 2016.)

Red Hat shared their Kpatch code with the community and integrated it as a supported feature of Red Hat Enterprise Linux.

The difference between the two incarnations can be inferred from the message emblazoned on the open-source version’s project page:

WARNING: Use with caution!
Kernel crashes, spontaneous reboots,
and data loss may occur!

Throughout the same period, and in parallel with the SUSE and Red Hat efforts, basic ABI foundations for supporting live patching were being integrated into the Linux kernel version 4.0 source code. The idea was to take the best ideas from both Kpatch and Kgraft and…patch and graft them into a common approach for the mainline. This was called livepatch, and in October 2016, Canonical announced they were introducing their own commercial kernel updating service based upon it, predictably called the Canonical Livepatch Service. First only available for Ubuntu 16.04 LTS, it was later extended to cover 14.04 LTS as well. In Ubuntu 18.04 LTS Livepatch is an install option and can be configured from the built-in software management tool, a sign of its growing importance in the standard software distribution.

2014: New Kid on the Block

As the major vendors scrabbled to be the first to launch viable live patching solutions, CloudLinux, a major player in Linux-based web hosting operating systems, launched KernelCare in May of 2014, after a successful beta in March.

They surprised the market by offering the widest feature set across the most number of Linux platforms, backing it up with a strong reputation in Linux kernel development and customer support. Another shock was the affordability, appealing to website hosters who find KernelCare’s per-server costs more manageable and scalable than their main competitor’s per-site costs.

More recently, the bundling of KernelCare with Imunify360 has made it show up on the radar of a new squadron of high-flying, security-minded system admins.

Conclusion: The Core Issue in 2019

As the world moves towards automated security, you’ll see automatic live kernel patch management software being integrated into popular Linux distributions ever more tightly. There are currently only five distinct vendors on the market. A feature comparison table lists their major selling points. In the further reading section, you’ll find documentation sources and background articles.

Tinkering with an active kernel can be messy. It’s not something an enterprise, or anyone running servers, wants to trust to untested and unsupported software. When done in the name of security, it’s one of a number of applications in Linux worth paying for, one of the few that absolutely must be done right.

¹ Because of Microsoft’s 2002 patent application, discussions at the time reveal concern over the long-term viability of the technology. See lkml.org April 2008 and lwn.net July 2011.

² MIT News: “Bringing the world reboot-less updates” (2014).

³ Since January 2016, Ksplice has been available only as part of Oracle’s UEK and Oracle Linux 6 & 7 products. In November of that year, they removed Red Hat’s upstream Kpatch code.

Appendix

Linux kernel live updating services: Feature comparison table

See Comparison Notes for details.

Further reading

General Articles

• Livepatch: Linux kernel updates without rebooting (27 June 2018) linux-audit.com
• Live Patching Meltdown–SUSE Engineer’s research project (Part 1) (2 May 2018) suse.com
• An update on live kernel patching (27 September 2017) lwn.net
• A Guide to kpatch on Red Hat Enterprise Linux 7.2 and Later (10 November 2016) redhat.com
• Hotfix Your Ubuntu Kernels with the Canonical Livepatch Service! (18 October 2016) blog.dustinkirkland.com
• Linux vs. Unix Hot Patching–Have We Reached The Tipping Point? (20 May 2016) forrester.com
• A rough patch for live patching (25 February 2015) lwn.net
• Live Kernel Update Tools (September 2014) admin-magazine.com
• KernelCare: New no-reboot Linux patching system (6 May 2014) zdnet.com

Documentation

Comparison Notes

Easy Install/Config

Rollback

Firewall friendly

Offline updates

Patch access control

Fully automated

Management GUI

Free 24/7 Support

No. of Platforms

• Ksplice Supported Kernels (Red Hat Enterprise Linux, Oracle Linux)
• Kgraft Data Sheet (SUSE Linux Enterprise Server 12/15)
• Kpatch (RedHat) Scope of Support (Red Hat Enterprise Linux)
• Kpatch (github.com) (Debian, CentOS, Ubuntu, Gentoo)
• Canonical Livepatch Service Data Sheet (PDF) (Ubuntu 14.04 LTS, 16.04 LTS)
• KernelCare Patch Server (Ubuntu, RHEL, CentOS, CloudLinux OS, Debian, Oracle Linux, Proxmox VE, Virt-SIG/Xen4CentOS, Virtuozzo/OpenVZ)

Free Trial

API (REST)

Non-latest override