SONiC Kernel Update Process and Testing Procedures

[littlespace]

Hello!

Could someone please clarify the process for upgrading the kernel in SONiC? Specifically:

• What types of tests are typically run before opening PRs, like this one?
• How are CVEs related to kernel updates handled?
• Are there any best practices for addressing minor kernel updates (as opposed to major ones)?
• How do we determine which files need to be updated during the kernel upgrade process for example going to kernel version 6.1.123?

Thanks in advance!

@saiarcot895 @qiluo-msft @bingwang-ms @xumia @lguohan

[bingwang-ms]

@saiarcot895 Can you help share some doc for kernel update?

[saiarcot895]

Sorry for the delay here.

• Typically, just bootup tests and BGP session establishments tests are run (along with making sure interfaces come up, port channels come up, etc.). Specific platforms may have more testing done.
• There's no special work done for CVE fixes that are coming in from the kernel updates.
• Because minor kernel updates usually (but not always) don't break something, and the number of API changes is smaller or none, there's less work needed to bring in a minor kernel update than a major kernel update. Major ones may need support from the vendor due to breaking changes.
• There are many files in sonic-buildimage that have hardcoded references to the kernel version (because of the kernel image file names). This is updated with something like git grep -l 6.1.0-22 | xargs sed -i 's,6.1.0-22,6.1.0-29,'. Besides that, if there are any build failures in kernel modules, then those are updated as necessary.

See also this for some details on the kernel (and other applications from Debian) update.

[littlespace]

Thanks for the detailed explanation! That gives us a good understanding of the process.

[aidan-gallagher]

Hi @saiarcot895,

We are interested in using an to date Linux Kernel in SONiC so we get CVE fixes. Below I have documented the process I used to bring in a later version of the kernel. Is there anything you would add?

PRs for some previous Linux kernel upgrades

• [202405] Update to Linux 6.1.94 #19885
• [202311] Update to Linux kernel 5.10.218 #19886
• [202405] Update to Linux 6.1.94 #19885

Locally Build SONiC Kernel

These steps let you build just the SONiC Linux kernel. This lets you quickly verify the kernel build succeeds before you go onto build the rest of sonic-buildimage.

Check rules/linux-kernel.mk to find KVERSION_SHORT (e.g. 6.1.0-22-2), KERNEL_VERSION (e.g. 6.1.94) and KERNEL_SUBVERSION (e.g 1).

With this information you can determine what the Linux kernel Debian package target is called by substituting those values into target/debs/bookworm/linux-image-<KVERSION_SHORT>-amd64-unsigned_<KERNEL_VERSION>-<KERNEL_SUBVERSION>_amd64.deb. For example target/debs/bookworm/linux-image-6.1.0-22-2-amd64-unsigned_6.1.94-1_amd64.deb

Run the make command to build SONiC linux kernel kernel and keep the sonic-slave build container alive

make -f Makefile.work QUIET=n BLDENV=bookworm KEEP_SLAVE_ON=yes target/debs/bookworm/linux-image-6.1.0-22-2-amd64-unsigned_6.1.94-1_amd64.deb

Once in slave you can navigate to the sonic-linux-kernel submodule and invoke the make command directly.

cd src/sonic-linux-kernel
CONFIGURED_ARCH=amd64 CONFIGURED_PLATFORM=broadcom make 

Find Kernel version

You can find the lastest Debian Linux kernel version using the debian tracker. You can also view the Debian Linux kernel versions available by searching for linux_ at: http://deb.debian.org/debian/pool/main/l/linux/. For minor kernel version update choose a version with the same first two numbers and the latest last number (e.g. go from 6.1.106 to 6.1.128). The new KERNEL_VERSION would be 6.1.128

To find the KVERSION_SHORT open the .dsc for the version you want. You will see linux-support-<KVERSION_SHORT>

For example http://deb.debian.org/debian/pool/main/l/linux/linux_6.1.128-1.dsc KVERSION_SHORT is 6.1.0-31.

Now we know for the upgrade the new values are

KERNEL_VERSION=6.1.128
KVERSION_SHORT=6.1.0-31

Build The New SONiC Linux Kernel

In src/sonic-linux-kernel/Makefile update KERNEL_VERSION and KVERSION_SHORT.
https://github.com/sonic-net/sonic-linux-kernel/blob/202311/Makefile#L8

SONiC pulls the Debian kernel from https://sonicstorage.blob.core.windows.net by default. Newer kernel versions are not be available there until someone from Microsoft uploads it. So temporarily change this to point to the Debian servers. In src/sonic-linux-kernel/Makefile update LINUX_SOURCE_BASE_URL to http://deb.debian.org/debian/pool/main/l/linux/. https://github.com/sonic-net/sonic-linux-kernel/blob/202311/Makefile#L57

Fixing Preconfig Patches

The two patches in src/sonic-linux-kernel/patch/preconfig/ require the md5sum of the file. So they need updated to specify the new md5sum. The patches are in the order of top to bottom of the series file. So repeat these steps for however many patches there are. It probably only matters that the final one is correct but may as well just do it for them all since there is only 2.

Follow the steps in "Locally Built SONiC Kernel" to build the kernel. It will fail but it will generate the directory src/sonic-linux-kernel/linux-6.1.128.

For every patch, in the series file order:

Open the file src/sonic-linux-kernel/linux-6.1.99/debian/config/defines.

Manually modify the file by applying the changes in the first patch (packaging-update-abiname.patch). Get the md5sum of the modified file (use md5sum src/sonic-linux-kernel/linux-6.1.99/debian/config/defines) and put that in packaging-update-abiname.patch in the section that changes debian/control.md5sum.

Fixing SONiC Patches

SONiC applies patches to the Debian Linux Kernel. Now you have changed the Debian Linux Kernel some of these SONiC patches will no longer cleanly apply (e.g. patch already included in upstream).

Comment out patches or fix them.

After this you should be able to locally build the SONiC Linux kernel stand alone.

Update linux-kernel.mk

The makefile in the sonic-linux-kernel submodule uses the ?= notation. This means the value is only set if it is not already set. This can be helpful as it allows the submodule to be build as a standalone makefile, however, when building it as part of sonic-buildimage we must also update linux-kernel.mk as it overrides the values in the submodule.

Edit rules/linux-kernel.mk to update KVERSION_SHORT & KERNEL_VERSION.
https://github.com/sonic-net/sonic-buildimage/blob/master/rules/linux-kernel.mk#L3

Update References to Linux Kernel

Many parts of sonic-buildimage reference the version in the source code. Run rg 6.1.0-22-2 to see a full list. Currently these files are all in the sonic-buildimage repository and the saibcm-modules-dnx submodule.

Do a bulk change to change all references

find . -type f -exec sed -i 's/6.1.0-22-2/6.1.0-31-2/g' {} +

Testing

Build the image and install it on a switch.

Ensure the containers are stable using docker ps -a .

Check for kernel log messages using journalctl -k -r

[saiarcot895]

I think that covers everything! This is also basically the process that I follow for each kernel update.

[robertlperry]

Hi Team,

This may not be the right place to ask, but I am not sure of a better place to ask this related question.

Can we consider a set cadence for upgrading the minor kernel version? This allows for more frequent and predicable rollout of CVE fixes/updates and helps operators who deploy SONiC in environments compliance requirements.

Thoughts @saiarcot895 @qiluo-msft @bingwang-ms @xumia @lguohan

A bit more context can be found below.
#20940

Thank you.

[saiarcot895]

This document specifies that the kernel be updated every 6 months. However, due to a number of reasons, that cadence cannot be met. There'll be discussions related to a new cadence happening soon in the Platform/OS Working Group.