kernel/run.py script provides a simple way to build, deploy and run the
system in various settings and configuration. For a complete set of parameters
and config options refer to the
run.py --help instructions.
As an example, the following invocation
python3 run.py --kfeatures test-userspace --cmd='log=info init=redis.bin' --mods rkapps init --ufeatures rkapps:redis --machine qemu --qemu-settings='-m 1024M' --qemu-cores 2
- compile the kernel with Cargo feature
- pass the kernel the command-line arguments
log=info init=redis.binon start-up (sets logging to info and starts redis.bin for testing)
- Compile two user-space modules
rkapps(with cargo feature redis) and
init(with no features)
- Deploy and run the compiled system on
qemuwith 1024 MiB of memory and 2 cores allocated to the VM
If Docker is used as build environment, it is necessary to first compile the system with the required features inside the Docker container:
python3 run.py --kfeatures test-userspace --mods rkapps init --ufeatures rkapps:redis -n
Afterwards, the aforementioned command can be used to run NRK outside the
Docker container with the given configuration. The
run.py script will
recognize that the system has already been build and will directly start
Sometimes it's helpful to know what commands are actually executed by
For example to figure out what the exact qemu command line invocation was. In
--verbose can be supplied.
Depending on the underlying system configuration NRK may abort because a connection to the local network can not be established. In this case, the following steps can help to resolve this issue:
- Disable AppArmor. Detailed instructions can be found here.
- Manually start the DHCP server immediately after NRK has started:
sudo dhcpd -f -d tap0 --no-pid -cf ./kernel/tests/dhcpd.conf
kernel/run.py script supports execution on baremetal machines with
python3 run.py --machine b1542 --verbose --cmd "log=info"
This invocation will try to run nrk on the machine described by a
b1542.toml config file.
A TOML file for a machine has the following format:
[server] # A name for the server we're trying to boot name = "b1542" # The hostname, where to reach the server hostname = "b1542.test.com" # The type of the machine type = "skylake2x" # An arbitrary command to set-up the PXE boot enviroment for the machine # This often involves creating a hardlink of a file with a MAC address # of the machine and pointing it to some pxe boot directory pre-boot-cmd = "./pxeboot-configure.sh -m E4-43-4B-1B-C5-DC -d /home/gz/pxe" # run.py support only booting machines that have an idrac management console: [idrac] # How to reach the ilo/iDRAC interface of the machine hostname = "b1542-ilo.test.com" # Login information for iDRAC username = "user" password = "pass" # Serial console which we'll read from console = "com2" # Which iDRAC version we're dealing with (currently unused) idrac-version = "3" # Typical time until machine is booted boot-timeout = 320 [deploy] # Server where binaries are deployed for booting with iPXE hostname = "ipxe-server.test.com" username = "user" ssh-pubkey = "~/.ssh/id_rsa" # Where to deploy kernel and user binaries ipxe-deploy = "/home/gz/public_html/"
An iPXE environment that the machine will boot from needs to be set-up. The iPXE bootloader should be compiled with UEFI and ELF support for running with nrk.
Note that the current support for bare-metal execution is currently limited to DELL machines with an iDRAC management console (needed to reboot the server). Ideally, redfish or SNMP support will be added in the future.
This document aims to list out steps to discover CXL type 3 devices inside the Linux kernel. Since there is no hardware available, the only way to achieve that is through Qemu emulation. Unfortunately, even the Qemu mainstream branch does not support these devices, so the tutorial uses a custom version of Qemu that supports CXL type 3 devices.
First, download and build the custom Qemu version on your machine.
sudo apt install build-essential libpmem-dev libdaxctl-dev ninja-build cd ~/cxl git clone https://gitlab.com/bwidawsk/qemu.git cd qemu git checkout cxl-2.0v4 ./configure --enable-libpmem make -j 16
Check the version:
QEMU emulator version 6.0.50 (v6.0.0-930-g18395653c3) Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
Next, download the latest kernel version and build an image from the source.
cd ~/cxl git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git cd linux make defconfig
defconfig generates default configuration values and stores them in the
.config file. The kernel requires some special configuration changes to handle these devices. Only a few of these configuration flags are present in the
.config file, so do not worry if you cannot find all these flags.
CONFIG_ACPI_HMAT=y CONFIG_ACPI_APEI_PCIEAER=y CONFIG_ACPI_HOTPLUG_MEMORY=y CONFIG_MEMORY_HOTPLUG=y CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y CONFIG_MEMORY_HOTREMOVE=y CONFIG_CXL_BUS=y CONFIG_CXL_MEM=y
Once the configuration related changes are done, compile the kernel code to the generate the image file.
make -j 16 cd ~/cxl
The image file should be in
-kernel parameter to use the kernel image directly.
Let's try to use that -
qemu/build/qemu-system-x86_64 -kernel linux/arch/x86_64/boot/bzImage \ -nographic -append "console=ttyS0" -m 1024 --enable-kvm
The kernel runs until it tries to find the root fs; then it crashes. One easy way to resolve this is to create a ramdisk.
mkinitramfs -o ramdisk.img
cto exit kernel and then press
qto exit Qemu.
Now, run qemu with the newly created ramdisk:
qemu/build/qemu-system-x86_64 -kernel linux/arch/x86_64/boot/bzImage -nographic \ -append "console=ttyS0" -m 1024 -initrd ramdisk.img --enable-kvm
The kernel runs properly this time. Now, it is time to add CXL related parameters before running Qemu:
qemu/build/qemu-system-x86_64 -kernel linux/arch/x86_64/boot/bzImage -nographic \ -append "console=ttyS0" -initrd ramdisk.img -enable-kvm \ -m 1024,slots=12,maxmem=16G -M q35,accel=kvm,cxl=on \ -object memory-backend-file,id=cxl-mem1,share=on,mem-path=cxl-window1,size=512M \ -object memory-backend-file,id=cxl-label1,share=on,mem-path=cxl-label1,size=1K \ -object memory-backend-file,id=cxl-label2,share=on,mem-path=cxl-label2,size=1K \ -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0,len-window-base=1,window-base=0x4c00000000,memdev=cxl-mem1 \ -device cxl-rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0,port=0 \ -device cxl-rp,id=rp1,bus=cxl.0,addr=1.0,chassis=0,slot=1,port=1 \ -device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M,lsa=cxl-label1 \ -device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M,lsa=cxl-label2
Qemu exposes the CXL devices to the kernel and the kernel discovers these devices. Verify the devices by running:
dmesg | grep '3:00'