For much of the Linux operating system’s history, patching a kernel has been a process that has typically involved downtime. In 2014, that’s no longer the case as there are now at least three distinct efforts that offer the promise of zero-downtime kernel patching to Linux servers.
The most recent entrant into the zero-downtime patching parade is Red Hat with the kpatch effort. Red Hat first revealed its efforts on Feburary 26th in a blog post detailing how kpatch works. Red Hat was unable to directly comment about any aspect of kpatch to ServerWatch for this article.
Red Hat is a relative late-comer to the dynamic patching party. Oracle has been in the space the longest, thanks to its 2011 acquisition of dynamic-kernel patching vendor Ksplice.
Michele Casey, Director of Product Management at Oracle, told ServerWatch there are several factors that come into play when considering providing kernel patches with zero-downtime.
Oracle’s Ksplice Has First-Mover Advantage
“Ksplice has been providing zero-downtime kernel updates to thousands of production systems for a number of years,” Casey said. “In fact, there have been over one million Ksplice patches released over the lifetime of the technology.”
Casey explained that in order to resolve security issues or bugs with a patch that can be applied without a system restart, a vendor needs to account for all the various function calls and touch points a given piece of code has to the kernel.
She added that it’s crucial to have the necessary infrastructure components to provide the checks needed for ensuring the updates are stable and ready for production deployments.
“Ksplice offers the tools and the expertise to monitor patches to determine which should be converted while providing multiple methods for installing and managing these updates, such as Ksplice’s web interface and offline client,” Casey said. ” In addition, scalability has been created within Ksplice’s infrastructure and tools to help ensure all kernel errata released can be patched by Ksplice.”
Oracle also doesn’t have a particularly enthusiastic view of rival dynamic kernel-patching technologies from Red Hat and SUSE. Casey said that from what Oracle has seen so far of the two other technologies, they are proof-of-concepts at best.
“kGraft (SUSE) is very restricted and cumbersome, requiring a lot of code duplication and code editing,” Casey said. “Neither technology Proof of Concepts have shown the ability to do anything more than creating a simple patch. There is no build infrastructure to handle large-scale patch building for a large variety of kernels.”
Vojtech Pavlik, director of SUSE Labs at SUSE, explained to ServerWatch that kGraft doesn’t ever need to stop the kernel to do the patching and is even able to patch kernel functions that are being actively executed by the OS.
“kGraft is intended to be merged into the upstream Linux kernel and to become a living open-source project,” Pavlik said. “It builds on and improves existing Linux infrastructure to fit seamlessly into the Linux kernel.”
SUSE Takes a Different Upstream Kernel Approach with kGraft
Pavlik noted that a key difference between Ksplice and kGraft is the upstream kernel approach. Pavlik noted that Ksplice tried – and failed – to get upstream acceptance in 2008, primarily because of the complexity of the changes required.
“Our goal is upstream acceptance, and thus we decided for a fresh start, and that allowed kGraft to be smaller, simpler and leaner and so give it a better chance to be accepted into Linus’s upstream kernel tree,” Pavlik said. “After our initial announcement of kGraft, it became apparent that we weren’t the only ones to follow that line of thinking — Kpatch (Red Hat) was most likely born the same way.”
In terms of upstream Linux kernel timing, Pavlik said that SUSE’s expectation is to open the code before the end of March and submit it for comments to the Linux Kernel Mailing List shortly thereafter. There is also the possibility of some form of collaboration between the Red Hat Kpatch effort and SUSE’s kGraft.
“We welcome this work on the same topic and are looking forward to working together with the Kpatch team and the kernel developer community in general on a common upstream solution for live kernel patching that may in the end turn out to be quite unlike either of the two technologies as they stand today,” Pavlik said.