Further_Reading.md

Further Reading

For the book "Linux Kernel Programming", 2nd Ed, Kaiwan N Billimoria, Packt (2024).

Book GitHub repo: https://github.com/PacktPublishing/Linux-Kernel-Programming_2E

Here, we present, in a chapter-wise format, additional online as well as book references. We feel that these will help you, the reader, in digging deeper into topics, or gaining additional views/information, or both. We definitely recommend you browse through the provided references. Also, you will find that some resources are repeated across chapters; we leave it this way so that you can gain all required resources in one place (and chapter-wise). (Obviously, these reference articles/blogs/etc are provided "as-is"; we cannot vouch for their correctness or validity).

For your convenience, below are chapter-wise links to the Further Reading notes:

• Generic online and book resources: miscellaneous, very useful

Section 1: The Basics

• Chapter 1, Kernel workspace setup
• Chapter 2, Building the 6.x Linux kernel from source - Part 1
• Chapter 3, Building the 6.x Linux kernel from source - Part 2
• Chapter 4, Writing your first kernel module - Part 1
• Chapter 5, Writing your first kernel module - Part 2

Section 2: Understanding and Working with the Kernel

• Chapter 6, Kernel Internals Essentials - Processes and Threads
• Chapter 7, Memory Management Internals Essentials
• Chapter 8, Kernel Memory Allocation for Module Authors - Part 1
• Chapter 9, Kernel Memory Allocation for Module Authors - Part 2
• Chapter 10, The CPU Scheduler, Part 1
• Chapter 11, The CPU Scheduler, Part 2

Section 3: Delving Deeper

• Chapter 12, Kernel Synchronization - Part 1
• Chapter 13, Kernel Synchronization - Part 2

Friendly Reminder !
This book's companion volume 'Linux Kernel Programming (Part 2): Writing character device drivers: Learn to work with user-kernel interfaces, handle peripheral I/O & hardware interrupts' is available as well.
And, hey, the ebook's free!
The LKP-Part2 book's GitHub repo is here.

Besides the Further Reading resource specified chapter-wise, be sure to refer to the "special" section below entitled Generic online and book resources : miscellaneous, very useful! These links and books are considered to be useful throughout your journey through this book and after.

Generic online and book resources : miscellaneous, very useful

• The Linux kernel documentation
• Very useful! LWN (Linux weekly news) Kernel index
• Deprecated things: DON'T use 'em! Just say No; Deprecated Interfaces, Language Features, Attributes, and Conventions

A few excellent books that come to mind (specific books on the Linux kernel and writing Linux device drivers are mentioned in the appropriate chapter's Further reading section):

• The Mythical Man Month, Frederick P Brooks

• The Art of Unix Programming, Eric S Raymond (AW)

• Programming Pearls, Jon Bentley

• The Practice of Programming, Brian Kernighan and Rob Pike

• The Pragmatic Programmer: From Journeyman to Master, Hunt and Thomas

• The Joy of Linux, Hall and Proffitt

• Linux Kernel Debugging - Leverage proven tools and advanced techniques to effectively debug Linux kernels and kernel modules, Kaiwan N Billimoria, Packt

• Linux Kernel Programming - A comprehensive guide to kernel internals, writing kernel modules, and kernel synchronization, Kaiwan N Billimoria, Packt

• Linux Kernel Programming, Part 2 - Writing character device drivers: Learn to work with user-kernel interfaces, handle peripheral I/O & hardware interrupts, Kaiwan N Billimoria, Packt

• Hands-On System Programming with Linux, Kaiwan N Billimoria, Packt

• Unix Network Programming, W. Richard Stevens

• Unix Systems Programming: Communication, Concurrency and Threads, Robbins and Robbins

• Secure Programming for Linux and Unix HOWTO, David Wheeler

• Excellent newsletters / blogs (biased towards embedded but useful to all IMO)

  • The Embedded Muse (TEM) blog/newsletter by Jack Ganssle
  • Embedded Artistry
  • In general an excellent in-depth technical article: SO YOU WANT TO BUILD AN EMBEDDED LINUX SYSTEM?, Jay Carlson, Oct 2020 ; it covers an introduction to the whys and why-nots of using Linux on an embedded system, hardware and software design workflow, as well as a pretty detailed rundown on 10 CPUs/boards that run embedded Linux (with benchmarks)
  • Julia Evans' blog and zines:
    • 'You can be a kernel hacker!', Julia Evans
    • Blog
    • Zines

• Ideas / articles on Software Design (biased towards embedded)

  • Collection of excellent articles from Embedded Artistry
  • Practical Decoupling Techniques Applied to a C-based Radio Driver, PHILLIP JOHNSTON, Embedded Artistry, Aug 2019
  • LOW-LEVEL SOFTWARE DESIGN, Kaiwan NB, Jan 2017

• Lowlevel programming University, gurugio

• ADVICE TO A YOUNG FIRMWARE DEVELOPER – BY JACK GANSSLE; AND ASSEMBLY, Kaiwan NB, Nov 2018

• Tip (reprodcued from Chapter 5): "Is there a really practical hands-on tip, an almost guaranteed way to become a kernel hacker? Of course, keep reading this book! Ha ha, yes, besides, do partake in the simply awesome Eudyptula Challenge. Hang on, it's - very unfortuantely, and as of this writing - closed down. Fear not, here's a site with all the challenges posted (and the solutions, don't cheat!); do check it out and try the challenges."

• Create Your Own Operating System: Build, deploy, and test your very own operating systems for the Internet of Things and other devices, Lucus Darnell, Aug 2016; Kindle:

[Top]

Chapter 1, Kernel Development Workspace Setup - Further Reading

Resources on setting up a Linux guest using VirtualBox on a Windows host:

• A very clearly written tutorial entitled Install Linux Inside Windows Using VirtualBox, Abhishek Prakash (It's FOSS!, August 2019)

• An alternate, similarly excellent resource: Install Ubuntu on Oracle VirtualBox

• How to Install VirtualBox Guest Additions in Ubuntu

• Related – Easy Ways to Check If Your Processor Supports Virtualization

• 'Share Folders between Host and Guest OS in VirtualBox', S Butler, April 2020

• Emulating Raspberry Pi on Linux

Good resources on toolchains:

• Cross-compilation toolchains for Linux, Bootlin Very user-friendly – simply select and download the toolchain of choice.

• Toolchains on eLinux

• A wiki page (from the SEALS project I maintain, see the following) with instructions on downloading and installing the latest ARM Linaro toolchain (a popular one) is available here

• Learn how to install many apps on Ubuntu/CentOS/Debian/etc via the Installi web guide

Some useful tooling, drivers, and so on, created by myself:

• The procmap utility - view any process's virtual address space fully (both user+kernel VAS): procmap
• Scripts, tools, utils, code : usefulsnips
• SEALS : Simple Embedded Arm Linux System
• device-memory-readwrite : Read/write pretty much any memory location (RAM or hardware IO memory) on a device or PC

Miscellaneous:

• Emulating a Raspberry Pi:

  • Emulating a Raspberry Pi on Linux
  • qemu-rpi-kernel, GitHub

• Code browsers

  • Ctags Tutorial
  • The Vim/Cscope tutorial

• Ubuntu releases

• Ubuntu kernel mapping to mainline

[Top]

Chapter 2, Building the Linux Kernel From Source, Part 1 - Further Reading

• Official Kernel Documentation : the modern Linux kernel documentation is very well written and presented here

  • "2. How the development process works"
  • The Linux kernel build system - kbuild - infrastructure
  • Kernel Sysctl Explorer - content from official docs in a searchable web page

• Git

  • Firstly, A Gentle Introduction to Version Control
  • Resources to Learn Git
  • The “official” git website
  • Try Git in your browser – interactive tutorial!
  • Most commonly used git tips and tricks on github
  • The "Git Magic" book, by Ben Lynn (PDF)
  • Aha! Moments when learning Git
  • How not to be afraid of Git anymore
  • Getting Git Right, Atlassian.com
  • ... and many more

• Linux Kernel Release Model, Greg Kroah-Hartman, Feb 2018

• Civil Infrastructure Platform Announces New Super Long Term Support Kernel that Advances Automation, Machine Learning and Artificial Intelligence, LF, Feb 2019

• LTSI Project update - Long Term Support Initiative – slide deck, Open Source Summit Japan, June 2018

  • Working with the kernel development community
  • Blog artcile: 'Kernel Patch Submission tutorial', Saurabh Singh Sengar
  • Working with the bleeding-edge, the linux-next tree:
    • Working with linux-next
    • How to contribute (was Re: Kernelnewbies Digest, Vol 77, Issue 7

• Linux kernel versions with detailed notes on new features, kernelnewbies

• Fun: Occurences of words in the Linux kernel source code over time

• Licensing

  • The differences between the GPL, LGPL and the BSD
  • Proprietary loadable kernel modules, LWN
  • Choose an open source license
  • Open source licensing for supervillains, Mossberg, Jan 2021
  • Legal Risks of Open Source – GPL/Linux Loadable Kernel Modules
  • Linux kernel licensing rules
  • To circumvent the EXPORT_SYMBOL_GPL(), people have been known to use a GPL "shim" kernel module whose routines are invoked from a proprietary module; Greg K Hartman 'Re: EXPORT_SYMBOL_GPL recursive for shim and/or wrapper
  • MIT vs. BSD vs. Dual License
  • Recent, off-beat
    • What comes after open source? Bruce Perens is working on it, The Register, Dec 2023
    • The eternal struggle between open source and proprietary software, TechCrunch, Dec 2023

• io_uring

  • What is io_uring?, unixism.net, 2020
  • The next deep magic Linux program to change the world? Io_uring, Steven J. Vaughan-Nichols, The Register, Sept 2022
  • Why you should use io_uring for network I/O, D Hunter, RedHat Developer, April 2023
    • This article includes more resource links.

• Rust in the Linux kernel !

  • Rust in the Linux Kernel, Steven J. Vaughan-Nichols, thenewstack, Oct 2022
  • Article on How to create a kernel module using Rust, Jack Clayton, 2022?
    • Based on this video presentation: Mentorship Session: Writing Linux Kernel Modules in Rust, The Linux Foundation
  • Official kernel doc: Rust

• Configuring the kernel

• Kconfig/Kbuild system

  • Detailed article: Exploring the Linux kernel: The secrets of Kconfig/kbuild
  • Slide presentation (PDF): A Dive Into Kbuild, Cao Jin, Fujitsu, Aug 2018

• Article: How to compile a Linux kernel in the 21st century, S Kenlon, Aug 2019

• Boot config: official kernel doc

• GCC plugins: official kernel doc

• Interesting: Meet the contributor of the 1-millionth commit: Ricardo Neri, Swapnil Bharatiya, Sept 2020

[Top]

Chapter 3, Building the Linux Kernel From Source, Part 2 - Further Reading

• Initramfs (initrd):

  • Opening and modifying the initrd
  • Initramfs (on Ubuntu wiki)
  • Informative: debian bug report: intel-microcode: breaks initrd for newer kernels, Sept 2014
  • distri: 20x faster initramfs (initrd) from scratch, Jan 2020
  • Encrypting disks with LUKS and cryptsetup
    • 'How To: Linux Hard Disk Encryption With LUKS (cryptsetup Command), Sept 2017, nixCraft
    • 'Basic Guide To Encrypting Linux Partitions With LUKS', Oct 2016, Nick Congleton
    • RedHat-specific: 'What is LUKS disk encryption and how can it be implemented?', Oct 2019, RedHat

• Article: How to Build Linux Kernel From Scratch {Step-By-Step Guide}, Nov 2020

• GRUB: How do I change the default boot kernel

• How to Boot into Single User Mode in CentOS/RHEL 7, Kili, Tecmint, August 2017

• Article: Linux Kernel Cross Compilation

• SEALS - Simple Embedded ARM Linux System - project

  • • Wiki pages

• Building the (usual 32-bit) kernel for the Raspberry Pi device

  • Raspberry Pi kernel build: official doc
  • Building a 64-bit kernel for the Raspberry Pi
    • blog article: Cross compiling for arm or aarch64 on Debian or Ubuntu, Jensd, Jan 2021
    • blog article: 'BUILD A 64-BIT KERNEL FOR YOUR RASPBERRY PI 3', Dec 2016
    • blog article: 'Tutorial: How (and why!) to set up a 64-bit kernel, 32-bit Raspbian host OS, 64-bit nspawn Debian guest OS RPi3 system', Jan 2019
    • On GitHub: 'Bootable RPi3 image with 64-bit kernel, 32-bit Raspbian Stretch host OS, 64-bit Debian Buster guest OS in nspawn container', Sakaki, Apr 2019
    • Interesting, an Ubuntu 18.04 server on Aarch64 Raspberry Pi; simplest way to get a 64-bit Raspberry Pi kernel and rootfs in a single image which can be burned onto the SD card.

[Top]

Chapter 4, Writing your first Kernel Module-LKMs Part 1 - Further Reading

• Official Kernel Documentation : kernel parameters

• 'How to get printk format specifiers right', Linux kernel doc

• 'What are the __init* and __exit* macros ?', kernelnewbies

• Systemd

  • Article: 'Systemd Essentials: Working with Services, Units, and the Journal', DigitalOcean
  • Article: 'Understanding Systemd Units and Unit Files', DigitalOcean
  • How to create a systemd service in Linux, Patel, Sept 2022
  • Understanding and administering systemd, Christopher Engelhard, Kamil Páral, Caleb McKee, Fedora docs, Aug 2020
  • Systemd Documentation
  • Systemd FAQs

• Linux console (on Wikipedia)

• Debugging by printing, eLinux

• LWN kernel index on 'Kernel messages' (printk)

• printk() indexing, Jon Corbet, LWN, May 2021

• Makefiles : introductory stuff

  • A Simple Makefile Tutorial
  • Practical Makefiles, by example
  • Learn Makefiles; With the tastiest examples

• The Linux Kernel Module Programming Guide, by Peter Jay Salzman, Michael Burian, Ori Pomerantz, Bob Mottram, Jim Huang, July 5, 2023

• Blog article: "MAKEFILES TUTORIAL", Liran B.H.

• Blog article: A Standalone Linux Kernel Module, Levinzon, Medium, May 2020

• BusyBox: The Swiss Army Knife of Embedded Linux

[Top]

Chapter 5, Writing your first Kernel Module-LKMs Part 2 - Further Reading

• Blog article: WORKING ON THE CONSOLE WITH THE RASPBERRY PI, kaiwanTECH

• Cross compiling a kernel module, embear

• Clang, a modern alternative to GCC

  • Cross compiling made easy, using Clang and LLVM, Feb 2021
  • Official Clang docs: Cross-compilation using Clang

• StackOverflow Q&A: "How to prevent “error: 'symbol' undeclared here” despite EXPORT_SYMBOL in a Linux kernel module?"

• Article: 'Improving EXPORT_SYMBOL()', Jon Corbet, Feb 2016, LWN

• Linux kernel static analysis

  • Blog article: "Static analysis on the Linux kernel"
  • PDF: "Static code checking In the Linux kernel", Arnd Bergmann, Linaro
  • Thesis paper: "Automatic Bug-finding Techniques for Large Software Projects", Jiri Slaby
  • A book on static analysis with the opensource SonarQube tool: ['SonarQube in Action', G. Ann Campbell and Patroklos P. Papapetrou, Oct 2013] https://www.manning.com/books/sonarqube-in-action#toc

• Licensing

  • Being honest with MODULE_LICENSE, Jon Corbet, LWN, Apr 2004
  • Multi-licensing
  • Dual Licensing: Having Your Cake and Eating It Too, LinuxInsider
  • GPL License FAQs
  • Selecting a license
    • Open Source Licenses Comparison -Guide, Leroux, Nov 2019
    • Choose an open source license
      • Appendix; a convenient comparison table of OSS licenses

• Blog article: "LINUX KERNEL DEVELOPMENT – KERNEL MODULE PARAMETERS", Liran B.H.

• Useful Arch Linux Wiki on kernel module - gathering info, auto-loading, aliasing, blacklisting, etc: "Kernel module"

• StackOverflow Q&A: "Where does modprobe load a driver that udev requests?"

• DKMS: Dynamic Kernel Module Support

  • AskUbuntu: "What does DKMS do? How do I use it?"
  • dkms(8) man page](https://linux.die.net/man/8/dkms)

• A few Linux FastBoot! resources

  • Fast boot with Raspberry Pi, Himesh’s blog, Aug 2018
  • Presentation by Jan Altenberg on booting linux in less than 1 second
    • Powerpoint here
    • Youtube link
  • powerpoint on boot time optimization using a beaglebone as a prototype
  • powerpoint on speeding up raspberry pi boot time here
  • Blog article Reducing Boot Time with systemd, Murray, Feb 2022
  • Good stackoverflow question on using systemd to figure boot time...
  • A Challenge called boot time, witekio blog

• Linux Kernel Security

  • The Kernel Self Protection Project (KSPP, Kees Cook)
  • Regular updates on the state of kernel security: codeblog, by Kees Cook
  • YouTube Linux Foundation channel: 'Mitigating Hackers with Hardening on Linux – An Overview for Developers, Focus on BoF', Kaiwan Billimoria (Embedded IOT Summit, Prague, June 2023)
  • Arch Linux wiki: "Security / Kernel hardening"
  • "The status of kernel hardening", LWN, Nov 2016
  • Blog article: "Effectively bypassing kptr_restrict on Android"
  • Quora: "What are some of the best resources for Kernel exploitation on Linux?"
  • Linux Kernel Exploitation GitHub blog - a long list of kernel exploitation resources
  • "Hardened GNU/Linux - Linux kernel mitigation checklist"
  • "Loading signed kernel modules", LWN, Dec 2011
  • "Is My IoT Device Secure? 7 Questions You Should Be Asking Yourself Today", Jan 2020, DZone
  • Kernel Lockdown LSM
    • Linux Kernel Finally Gets Its Lockdown, Oct 2019
    • kernel_lockdown(7) — Linux manual page
    • Securing the Linux kernel with lockdown mode, 2020

• Arithmetic Overflow

  • An excellent must-read whitepaper: 'Understanding Integer Overflow in C/C++', Dietz, et al, June 2012
  • 'Catch and compute overflow during multiplication of two large integers', StackOverflow
  • An example of checking for arithmetic overflow in 'real-world' code can be found in the git codebase; see the wrapper.c code for usage examples

• The Eudyptula Challenge: "The Eudyptula Challenge was a series of programming exercises for the Linux kernel, that started from a very basic "Hello world" kernel module, moving on up in complexity to getting patches accepted into the main Linux kernel source tree." It's really cool but NOT active as of this writing though.

[Top]

Chapter 6, Kernel Internals Essentials - Processes and Threads - Further Reading

• Book: Hands-On System Programming with Linux", Kaiwan N Billimoria, Packt. Prerequisites to this chapter (essential reading, really):

  • Ch 1 : Linux System Architecture
  • Ch 2 : Virtual Memory

• Process Virtual address space (VAS) on Wikipedia

• CPU ABI, a blog article : APPLICATION BINARY INTERFACE (ABI) DOCS AND THEIR MEANING

• LWN kernel index on kernel stacks

• Blog article: Some Tricks used by the Linux kernel

• [un]likely() macros:

  • likely()/unlikely() macros in the Linux kernel - how do they work? What's their benefit?
  • Why do we use __builtin_expect when a straightforward way is to use if-else

• [e]BPF – The modern approach to system/app tracing and performance analysis:

  • Linux Extended BPF (eBPF) Tracing Tools, Brendan Gregg
  • Learn eBPF Tracing: Tutorial and Examples (B Gregg)
  • How eBPF Turns Linux into a Programmable Kernel, Jackson, October 2020
  • (Kernel-level) A thorough introduction to eBPF, Matt Fleming, LWN, December 2017
  • How io_uring and eBPF Will Revolutionize Programming in Linux, Glauber Costa, April 2020
  • BPF Performance Tools, by Brendan Gregg – official repo

• Flame Graphs

• Kernel lockdown:

  • Linux Kernel Finally Gets Its Lockdown, J Wallen, October 2019
  • Lockdown as a security module, Jonathan Corbet, LWN, June 2019
  • Running an eBPF program may require lifting the kernel lockdown, Gehrcke, September 2019

• System Call internals

  • Internal glibc-and-kernel level implementation of system calls, a good blog article: LINUX SYSTEM CALLS UNDER THE HOOD, Julien Sobczak, Aug 2021
  • Official kernel doc: Adding a New System Call

[Top]

Chapter 7, Memory Management Internals Essentials - Further Reading

• Books on Linux kernel memory management internals:

  • 'Professional Linux Kernel Architecture', Mauerer, Wrox Press
  • 'Understanding the Linux Virtual Memory Manager', Mel Gorman (Bruce Perens Open Source)

• x86 Paging Tutorial, C Santilli

• Deep detail on MM and paging on the Intel processors can be found in their excellent manuals here (Ch 4, Paging): Intel® 64 and IA-32 Architectures Software Developer’s Manual. Volume 3 (3A, 3B & 3C): System Programming Guide

  • All Intel 64 and IA-32 architecture and programming environment manuals

• Excellent blog article: Linux Internals: How /proc/self/mem writes to unwritable memory, 'offlinemark', May 2021

• pahole, BTF

  • 'Poke-a-hole and friends', Rodrigues, LWN, June 2009
  • How to generate BTF information ?
  • Official kernel doc: BPF Type Format (BTF)

• How the ARM32 kernel starts, Linus Walleij, Aug 2020

• On vsyscall, vdso and VVAR

  • "On vsyscalls and the vDSO", LWN, Jon Corbet, June 2011
  • "Are system calls the only way to interact with the Linux kernel from user land?", StackExchange, Apr 2014

• Using the pmap and smem utilities

  • pmap(1) man page

  • 'How to analyze a Linux process' memory map with pmap', Rossi, RedHat, Dec 2022

  • How to Use the pmap Linux Command to Check Memory Usage, D Kariuki, June 2022

  • smem(8) - Linux man page

  • 'Get memory use statistics with this Linux command-line tool', T Waraksa, Oct 2021

  • smem memory reporting tool, Selenic

  • 'Linux Terminal: Check who uses all your memory with smem', Linuxaria, Oct 2013

• Kernel memory optimization techniques

  • THP (Transparent Huge Pages)
  • KSM (Kernel Samepage Merging)

• 'Virtual Memory and Linux', PDF, Matt Porter, Alan Ott

• Linux (physical) memory models

  • Official kernel doc: Physical Memory Model
  • Memory: the flat, the discontiguous, and the sparse, LWN, M Rapoport, May 2019
  • struct page, the Linux physical page frame data structure, Mathhew Wilcox, Oct 2020

• Blog article: In defence of swap: common misconceptions, Chris Down, Jan 2018

[Top]

Chapter 8, Kernel Memory Allocation for Module Authors, Part 1 - Further Reading

• Book: 'Hands-On System Programming with Linux', Kaiwan N Billimoria, Packt. Prerequisites to this chapter (essential reading, really):

  • Ch 1 : Linux System Architecture
  • Ch 2 : Virtual Memory

• Linux kernel documentation on memory management

• A pair of memory-allocation improvements in 5.13, Jon Corbet, LWN, May 2021

• Information regarding GFP flags internal usage, etc: 'Memory management when failure is not an option', LWN, Mar 2015

• smem: smem memory reporting tool

• CPU Caching

  • Why software developers should care about CPU caches, EventHelix, Medium, July 2017
  • Cache coherence
  • "How L1 and L2 CPU Caches Work, and Why They’re an Essential Part of Modern Chips", Joel Hruska, Aug 2018
  • An example diagram of CPU (cache) hierarchy: "Memory hierarchy of an AMD Bulldozer server"

• Slab layer - performance, etc

  • Linux SLUB Allocator Internals and Debugging, Part 1 of 4, Imran Khan, Dec 2022
  • 'Toward a more efficient slab allocator', LWN, Jon Corbet, Jan 2015
  • Blog article 'INTERESTING NUMBERS'; see the section on Networking for some information on how the network subsystem has time critical code paths (and the resource links that follow)

• GNUplot

  • Must-See Demos for gnuplot version 5.4 (pngcairo terminal)
  • "GNUPLOT 4.2 - A Brief Manual and Tutorial", Duke University
  • gnuplot Quick Reference

• 'Object-oriented design patterns in the kernel, part 1', Neil Brown, LWN, June 2011

[Top]

Chapter 9, Kernel Memory Allocation for Module Authors, Part 2 - Further Reading

• LWN kernel index on memory management

• Better tools for out-of-memory debugging, LWN, Jon Corbet, May 2022

• 'kvmalloc()', Jon Corbet, LWN, Jan 2017

• A pair of memory-allocation improvements in 5.13, Jon Corbet, LWN, May 2021

• 'The "too small to fail" memory-allocation rule', Jon Corbet, LWN, Dec 2014

• Blog article: 'linux slab poisoning 101', Mar 2009

• 'The slab and protected-memory allocators', LWN, May 2018

• Kernel memory debugging tools and techniques:

  • KASAN – Kernel Address Sanitizer
  • Kernel SLUB debug
  • kmemleak

• DMA - Direct Memory Access

  • Books: 'Essential Linux Device Drivers', S Venkateswaran, and 'Linux Device Drivers', Rubini, Corbet & Hartman
  • 'DMAEngine documentation', Linux kernel
  • 'A deep dive into CMA', LWN, Mar 2012
  • A quite simple and interesting example of a “DMA test application” device driver is from Xilinx (for it's Zynq PL330 DMA controller); wiki page here
  • Q&A on SO: Linux DMA: Using the DMAengine for scatter-gather transactions, May 2016

• Reclaim: Official kernel documentation

• Multi Generational LRU (MGLRU); merged in 6.1

  • Official kernel docs:
    • Multi-Gen LRU: provides documentation on quick start, enabling / disabling it, various tunables
    • MGLRU design doc
  • LWN:
    • Merging the multi-generational LRU, Jon Corbet, May 2022
    • Multi-generational LRU: the next generation, Jon Corbet, May 2022
    • The multi-generational LRU, Jon Corbet, April 2022
  • Related:
    • Enable or disable multigenerational LRU. Practical..
    • mm: multi-gen LRU: per-process heatmaps
    • Linux kernel page replacement algorithms, SoByte, Jan 2022
    • Linux to Adopt New Multi-Generation LRU Page Reclaim Policy, Sergio De Simone, Jan 2022
    • Early commit: mm: multi-gen LRU: minimal implementation
    • OpenWrt / MIPS benchmark with MGLRU, Yu Zhao, Aug 2022

• DAMON Data Access Monitor

  • Official kernel doc: DAMON
    • Section 'Visualizing Recorded Patterns'
  • D A M O N: Data Access MONitoring Framework for Fun and Memory Management Optimizations, SeongJae Park, Kernel Summit Aug 2020
  • DAMON: Data Access Monitor, SeongJae Park, May 2023
  • The damo front-end utility on GitHub; Shows some nice usage examples!
  • The masim memory access simulator on GitHub
  • An example of DAMON usage for profiling, SeongJae Park, April 2023
  • Using DAMON for proactive reclaim, Jon Corbet, LWN, July 2021
  • Memory-management optimization with DAMON, Jon Corbet, LWN, Feb 2020

• OOM and VM Overcommit

  • Better tools for out-of-memory debugging, Jon Corbet, May 2022, LWN
  • Per-file OOM badness, Jon Corbet, June 2022, LWN
  • Chris Siebenmann's excellent blog articles; (search for 'overcommit' within them)
  • Toward more predictable and reliable out-of-memory handling, LWN, Jon Corbet, December 2015
  • Quora: What are the disadvantages of disabling memory overcommit in Linux?
  • Relevant Linux kernel documentation
  • Blog article: 'Virtual memory settings in Linux - The Problem with Overcommit'
  • RHEL specific but useful: 'Interpreting /proc/meminfo and free output for Red Hat Enterprise Linux 5, 6 and 7'
  • Blog article: Surviving the Linux OOM Killer, October 2018
  • User space OOM killer systemd-oomd
    • Man page on systemd-oomd.service(8)
    • Older - March 2014: User-space out-of-memory handling, David Rientjes, LWN
  • How to diagnose OOM errors on Linux systems, Paul Gottschling, Datadog, Dec 2023

• StackOverflow: How to mmap a Linux kernel buffer to userspace?

[Top]

Chapter 10, The CPU Scheduler, Part 1 - Further Reading

• Detailed PDF: 'A complete guide to Linux process scheduling', Nikita Ishkov, Feb 2015

• (CFS) Autogroups

  • Why nice levels are a placebo and have been for a very long time, and no one seems to have noticed, Reddit, 2020
  • RFC: documentation of the autogroup feature [v2], Nov 2016
  • man 7 sched

• Scheduler tuning:

  • A short and useful PDF doc: Completely Fair Scheduler and its tuning, Jacek Kobus and Rafał Szklarski, 2009
  • Tuning the Task Scheduler, SuSe documentation
  • A real-world, useful case: Reducing latency spikes by tuning the CPU scheduler, T Gabriec, June 2016, Scylla

• perf

  • Linux perf examples, Brendan Gregg
  • The perf wiki page
  • perf tutorial
  • Tutorial: Playing around with perf
  • Memory Access Patterns Are Important

• Video tutorial: talk recorded for the Linux Foundation’s OSSNA (Open Source Summit, North America), June 2023: 'Leveraging the OS CPU Scheduler to Write Real-Time MT Apps', Kaiwan N Billimoria, kaiwanTECH

• Ftrace

  • 'ftrace: trace your kernel functions!', Julia Evans, Mar 2017

• Linux kernel preemption and the latency-throughput tradeoff, Dec 2019

• An excellent detailed code-level article on how, specifically, the context switching switch_to() code path has evolved on the x86[_64] arch: Evolution of the x86 context switch in Linux, Maizure, September 2018

[Top]

Chapter 11, The CPU Scheduler, Part 2 - Further Reading

• Using the kernel's cgroups v2 cpusets: kernel doc

• Control Groups (cgroups)

  • The must-read: official Linux kernel doc: Control Group v2
  • Four-part series on cgroups and systemd, Red Hat, Steve Ovens
    • A Linux sysadmin's introduction to cgroups, Steve Ovens, Sept 2020
    • How to manage cgroups with CPUShares, Steve Ovens, Oct 2020
    • Managing cgroups the hard way-manually, Steve Ovens, Oct 2020
    • Managing cgroups with systemd, Steve Ovens, Oct 2020
  • How I think you set up fair share scheduling under systemd, C Siebenmann, May 2016
  • The basics of Linux fair share CPU scheduling in cgroup v2 ('unified cgroups'), C Siebenmann, May 2022
  • Understanding cgroups, Grant, Nov 2018
  • Good introductory article (focussed on cgroups1): 'Control Groups in Linux', Feb 2017
  • 'Understanding the new control groups API', Rami Rosen, LWN, Mar 2016
  • man page (informational): 'cgroups - Linux control groups' (man 7 cgroups)
  • Managing resources with cgroups in systemd, David Both, Oct 2020
  • RedHat System Design Guide (RHEL 8); focus on cgroups v1: 'SETTING LIMITS FOR APPLICATIONS'
  • FB: Includes an interesting case study on how Facebook uses Linux's cgroups2 to perform equitable resource distribution on (some of) it's servers: 'Maximizing Resource Utilization with cgroup2'
    • 'Creating and organizing cgroups'
  • Blog article, examples of using cgroups v1 for CPU and memory bandwidth control: 'Linux Virtualization : Resource throttling using cgroups'

• Kernel namespaces

  • The 7 most used Linux namespaces, Oens, Red Hat, Jan 2021
  • A Practical Introduction to Container Terminology, Feb 2018, McCarty, Red Hat Developer
  • namespaces(7) — Linux manual page

• Hard Real-Time and Linux as an RTOS

  • 'Inside Real-Time Linux', Feb 2017
  • 'The Road to Real Time Linux', Steven Rostedt, Mar 2017
  • VDC Survey on real-time response : The Embedded Muse, 341, 03 Jan 2018, Jack Ganssle
  • Blog article: 'Algorithm time complexity and big O notation'
  • Book: Automate This: How Algorithms Came to Rule Our World, Christopher Steiner
  • Older wiki site on preempt_rt; it's in the process of being migrated to the new documentation for RTL; it still has very useful HOWTO guides, articles, etc: 'Real-Time Linux Wiki'

• RTL – Real-Time Linux, using Linux as an RTOS:

  • The RTL Collaborative Project
  • RTL Real-Time documentation
  • HOWTO: RTOS and RT Applications
  • Interesting: "... compares the load results of the Ubuntu 18.04 LTS generic kernel 4.15.0-22-generic, the Ubuntu low-latency kernel 4.15.0-22-lowlatency and an Ubuntu Linux kernel patched to be fully preemptive 4.16.0-rt4-PREEMPT_RT_FULL as well as preemptive with low-latency 4.16.12-rt4+REEMPT_LL" : 'Low latency and real-time kernels for telco and NFV', Ubuntu, Oct 2018
  • 'Real-time Linux communications - An evaluation of the Linux communication stack for real-time robotic applications', Sept 2018
  • 'The real-time linux kernel: A survey on Preempt_RT', Feb 2019
  • Detailed slides on cyclictest, good for understanding latency and it's measurement: 'Using and Understanding the Real-Time Cyclictest Benchmark', Rowand, Oct 2013
  • 'Intro to Real-Time Linux for Embedded Developers', an interview with Steven Rostedt

• The still amazing and very relevant book on software engineering: 'The Mythical Man Month: Essays on Software Engineering', Frederick P Brooks, 1975, 1995

  • On Amazon (Anniversary Edition)
  • Quotes

• New(er) articles on LWN regarding scheduling-related topics:

  • The new 6.6 EEVDF (Earliest Eligible Virtual Deadline First) CPU/task scheduler:

    • An EEVDF CPU scheduler for Linux, Jon Corbet, Mar 2023
    • EEVDF Scheduler Merged For Linux 6.6, Intel Hybrid Cluster Scheduling Re-Introduced, M Larabel, Aug 2023
    • KernelNewbies: Linux 6.6

  • Fixing a corner case in asymmetric CPU packing (January 7, 2022)

  • Hybrid scheduling gets more complicated (September 30, 2022)

  • Core scheduling lands in 5.14 (July 1, 2021)

  • Improved response times with latency nice (March 17, 2022)

  • Evaluating vendor changes to the scheduler (May 19, 2020)

  • Removing the scheduler's energy-margin heuristic (July 1, 2022)

  • Cgroups

    • The burstable CFS bandwidth controller (February 8, 2021)
    • A "kill" button for control groups (May 3, 2021)
    • Cleaning up dying control groups, 2022 edition (May 19, 2022)

• Google's research project: ghOSt OS

  • 'ghOSt: Fast & Flexible User-Space Delegation of Linux Scheduling', J T Humphries et al, ACM Digital Library, Oct 2021
    • Paper
    • Video talk
  • ghOSt user space repo on GitHub
  • ghOSt kernel-space repo on GitHub

[Top]

Chapter 12, Kernel Synchronization, Part 1 - Further Reading

• The official kernel documentation on Locking: Locking

• In-depth, execellent: Is Parallel Programming Hard, And, If So, What Can You Do About It?, Paul E. McKenney, 2023 (maintained)

• Deeper details on the LKMM: Explanation of the Linux-Kernel Memory Model

• Who's afraid of a big bad optimizing compiler? Article contributed by Jade Alglave, Will Deacon, Boqun Feng, David Howells, Daniel Lustig, Luc Maranget, Paul E. McKenney, Andrea Parri, Nicholas Piggin, Alan Stern, Akira Yokosawa, and Peter Zijlstra, LWN, July 2019

• Why kernel code should use READ_ONCE and WRITE_ONCE for shared memory accesses, Google wiki. CAUTION! Module/driver devs, don't blindly do this!

• Daniel Vetter’s excellent blog posts here:

  • Locking Engineering Principles, Daniel Vetter, July 2022 (In this post, he says, among other stuff: “… 1. Make it Dumb: Since this is the key principle …”. IOW, keep it simple)
  • Locking Engineering Hierarchy, Daniel Vetter, August 2022 : A ‘pile’ of locking patterns and designs, from the easiest to hardest
  • Here's Paul E. McKenney’s follow-up post: Stupid SMP Tricks: A Review of Locking Engineering Principles and Hierarchy, Aug 2022

• A good read: 'What every systems programmer should know about concurrency', M Kline, May 2018

• Chip Overclock blog: Peeking under the Hood, Sloan, May 2012

• Measuring context switching and memory overheads for Linux threads, Eli Bendersky, Sept 2018

• Compiler Explorer website, Matt Godbolt; very useful: See the live disassembly of your code on a large choice of compilers and compiler flags

• This presentation covers deadlock scenarios: 'Lockdep: how to read it's cryptic output', Steve Rostedt, Linux Plumbers Conf 2011

• 'Mutexes and Semaphores Demystified', Micheal Barr, May 2016

• Mutex internal implementation

  • Official kernel doc: Implementation
  • Mutex - Linux inside - 0xax

• Spinlock internal implementation

  • 'Peeking Under the Hood', John Sloan, Chip Overclock blog, May 2012
  • 'spin_lock implementation in ARM linux', linuxforthenew blog, Apr 2013
  • Synchronization primitives in the Linux kernel. Part 1, linux-insides online book

• The Mars Pathfinder mission and Priority Inversion

  • 'What really happened on Mars ?' Glenn Reeves
  • A must-read (PDF): What the Media Couldn't Tell You About Mars Pathfinder
  • Detailed paper (PDF): Mars Pathfinder: Priority Inversion Problem, R.M. Pathan
  • More recently (May 2015), with respect to the NASA Curiosity Rover: 13 engineering truths proved by NASA's Curiosity Rover, EDN

• OSFY magazine: 'How to Avoid Priority Inversion and Enable Priority Inheritance in Linux Kernel Programming', Thangaraju & Warade, Apr 2019

• The w/w - wait/wound - mutex:

  • ww-mutex-design.txt, Jon Corbet, LWN, April 2013
  • Official kernel doc (based upon the above one, actually) Wound/Wait Deadlock-Proof Mutex Design

[Top]

Chapter 13, Kernel Synchronization, Part 2 - Further Reading

• LWN Kernel Index on Reference counting

• Official kernel doc: Reference counting (interfaces/APIs)

• Reference Counting API: kernsec: a summary

• CPU Caches, coherency and False Sharing

  • Cache coherency protocol examples, Wikipedia
  • Cache Coherence Problem & Cache Coherency Protocols, Neso, YouTube video
  • MOESI protocol; used on AMD64 processors
  • 'Avoiding and Identifying False Sharing Among Threads', Intel, Nov 2011
  • Blog article: 'Understanding False Sharing', Mar 2017

• Lock-free (or lockless) programming:

  • What every systems programmer should know about concurrency, Matt Kline, May 2018
  • Excellent "lock-free 101" articles:
    • Fear and Loathing in Lock-Free Programming, T. Neely, Medium
    • An Introduction to Lock-Free Programming, Preshing on Programming blog, June 2012
  • Good articles, series: 'Introduction to Lock-free Algorithms'
    • Presentation: 'Lock-Free Programming', Geoff Langdale
    • Blog article: 'Some notes on lock-free and wait-free algorithms'
    • 'CppCon 2014: Herb Sutter "Lock-Free Programming (or, Juggling Razor Blades), Part I', Herb Sutter, YouTube video
    • 1024cores: Dmitry Vyukov's blog articles etc

• Per-CPU variables:

  • Blog article: A brief introduction to per-cpu variables, Chen, May 2014
  • How are percpu pointers implemented in the Linux kernel?, Stack Overflow, June 2013

• RCU - Read-Copy-Update - in the Linux kernel

  • Official kernel doc (includes FAQs): RCU Concepts
  • Excellent (and up-to-date!) collated documentation on RCU, Paul E. McKenney: Read-Copy Update (RCU) Publications
  • RCU articles on LWN kernel index, from 2004 onward
  • A superb playlist of YouTube video tutorials on RCU (by S.T. Chang)! RCU
    • Do see the first one first 12 what is RCU, Paul E. McKenney, presentation at IISc Bangalore, 2013
    • Modern-ish take: Kernel Recipes 2019 - RCU in 2019, Joel Fernandes, Google, Oct 2019
  • My First Kernel Module: A Debugging Nightmare, Ryan Eberhardt. Nov 2020; see the section entitled 'RCU: Read, Copy, Update', excellent and clear!
  • A 3 series article to learn RCU: 'What is RCU, Fundamentally?', LWN, Paul E. McKenney, Dec 2007, Part 1 of 3
  • 'RCU Usage In the Linux Kernel: One Decade Later', Paul E. McKenney et al (PDF), circa 2013
  • Code-level examples of RCU usage:
    • A nice simple 'book library' example: rcu_example, Jinbum Park
    • Another covering rw-lock and (S)RCU: Linux RCU Usage and internals, Feb 2021
  • Linux RCU, Lizhou Shan, Oct 2021
  • Sleepable RCU, Paul E. McKenney, LWN, Oct 2006
  • 'Make any algorithm lock-free with this one crazy trick', May 2016
  • 'The lockless page cache', LWN, Jon Corbet, July 2008 (an interesting article)

• Lockdep

  • 'Runtime locking correctness validator', kernel documentation
  • 'The kernel lock validator', LWN, J Corbet, May 2006
  • 'Lockdep: how to read it's cryptic output', Steve Rostedt, Linux Plumbers Conf 2011
  • 'LOCKDEP, AN INSIDE OUT PERSPECTIVE', Nahim El Atmani, Nov 2016
  • Lockdep False Positives, some stories about, David Airlie, Aug 2020

• Memory barriers:

  • PDF: 'Memory Barriers: a Hardware View for Software Hackers', Paul E. McKenney, IBM LTC, June 2010
  • Memory Barriers Are Like Source Control Operations, Preshing on Programming blog, July 2012

• kdump and crash:

  • Kernel documentation for kdump: The kexec-based Crash Dumping Solution
  • Oops! Debugging Kernel Panics, Linux Journal, August 2019
  • Excellent resource for learning and using the crash app: White paper: Red Hat Crash Utility, David Anderson
  • Analyzing Linux kernel crash dumps with crash - The one tutorial that has it all, Dedoimedo, June 2010

[Top]

[End Doc]