If you've found and fixed a bug, we better write a test for it. nrk uses several test-frameworks and methodologies to ensure everything works as expected:
- Regular unit tests: Those can be executed running
cargo testin the project folder. Sometimes adding
RUST_TEST_THREADS=1is necessary due to the structure of the runner/frameworks used. This should be indicated in the individual READMEs.
- A slightly more exhaustive variant of unit tests is property based testing. We use proptest to make sure that the implementation of kernel sub-systems corresponds to a reference model implementation.
- Integration tests are found in the kernel, they typically launch a qemu instance and use rexpect to interact with the guest.
- Fuzz testing: TBD.
To run the unit tests of the kernel:
RUST_BACKTRACE=1 RUST_TEST_THREADS=1 cargo test --bin nrk
To run the integration tests of the kernel:
RUST_TEST_THREADS=1 cargo test --test '*'
If you would like to run a specific integration test you can pass it with
RUST_TEST_THREADS=1 cargo test --test '*' -- userspace_smoke
If you would like to run a specific set of integration tests, you can specify the file name with
RUST_TEST_THREADS=1 cargo test --test s00_core_test
In case an integration test fails, adding
--nocapture at the end (needs to
come after the
--) will make sure that the underlying
run.py invocations are
printed to the stdout. This can be helpful to figure out the exact
invocation that a test is doing so you can invoke it yourself manually for
Parallel testing for he kernel is not possible at the moment due to reliance on build flags for testing.
The commitable.sh script automatically runs the unit and integration tests:
cd kernel bash commitable.sh
Typically these can just be declared in the code using
#[test]. Note that
tests by default will run under the
unix platform. A small hack is necessary
to allow tests in the
x86_64 to compile and run under unix too: When run on a
x86-64 unix platform, the platform specific code of the kernel in
will be included as a module named
x86_64_arch whereas normally it would be
arch. This is a double-edged sword: we can now write tests that test the
actual bare-metal code (great), but we can also easily crash the test process by
calling an API that writes an MSR for example (e.g, things that would require
ring 0 priviledge level).
Integration tests typically spawns a QEMU instance and beforehand compiles the kernel/user-space with a custom set of Cargo feature flags. Then it parses the qemu output to see if it gave the expected output. Part of those custom compile flags will also choose a different main() function than the one you're seeing (which will go off to load and schedule user-space programs for example).
There is two parts to the integration test.
- The host side (that will go off and spawn a qemu instance) for running the
integration tests. It is found in
- The corresponding main functions in the kernel that gets executed for a
particular example are located at
To add a new integration test the following tests may be necessary:
kernel/Cargo.tomlto add a feature (under
[features]) for the test name.
- Optional: Add a new
xmainfunction and test implementation in it to
kernel/src/integration_main.rswith the used feature name as an annotation. It may also be possible to re-use an existing xmain function, in that case make not of the feature name used to include it.
- Add a runner function to one of the files in
kernel/teststhat builds the kernel with the cargo feature runs it and checks the output.
Integration tests are divided into categories and named accordingly (partially to ensure the tests run in a sensible order):
s00_*: Core kernel functionality like boot-up and fault handling
s01_*: Low level kernel services: SSE, memory allocation etc.
s02_*: High level kernel services: ACPI, core booting mechanism, NR, VSpace etc.
s03_*: High level kernel functionality: Spawn cores, run user-space programs
s04_*: User-space runtimes
s05_*: User-space applications
s06_*: Rackscale (distributed) tests
Benchmarks are named as such:
s10_*: User-space applications benchmarks
s11_*: Rackscale (distributed) benchmarks
s11_* benchmarks may be configured with two features:
baseline: Runs NrOS configured similarly to rackscale, for comparison
affinity-shmem: Runs the
ivshmem-serverusing shmem with NUMA affinity. This option requires preconfiguring hugetlbfs with
sudo hugeadm --create-global-mounts, having a kernel with 2MB huge pages enabled, and then also adding 1024 2MB pages per node, with a command like:
echo <page-num> | sudo numactl -m <node-num> tee -a /proc/sys/vm/nr_hugepages_mempolicyThe number of huge pages per node may be verified with
nrk has support for three network interfaces at the moment: virtio, e1000 and
vmxnet3. virtio and e1000 are available by using the respective rumpkernel
drivers (and it's network stack). vmxnet3 is a standalone implementation that
smoltcp for the network stack and is also capable of running in ring 0.
The integration tests that run multiple instances of nrk require
bridged tap interfaces. For those integration tests, the test framework calls
run.py with the
--network-only flag which will destroy existing conflicting
tap interfaces and create new tap interface(s) for the test based on the
number of hosts in the test. Then, to run the nrk instances, run.py is invoked
To setup the network for a single client and server (
--workers clients+server), run the following command:
python3 run.py --kfeatures integration-test --cmd "test=network_only" net --workers 2 --network-only
A simple check is to use ping (on the host) to test the network stack
functionality and latency. Adaptive
ping -A, flooding
ping -f are good modes
to see that the low-level parts of the stack work and can handle an "infinite"
amount of packets.
Some expected output if it's working:
$ ping 172.31.0.10 64 bytes from 172.31.0.10: icmp_seq=1 ttl=64 time=0.259 ms 64 bytes from 172.31.0.10: icmp_seq=2 ttl=64 time=0.245 ms 64 bytes from 172.31.0.10: icmp_seq=3 ttl=64 time=0.267 ms 64 bytes from 172.31.0.10: icmp_seq=4 ttl=64 time=0.200 ms
For network tests, it's easiest to start a DHCP server for the tap interface so the VM receives an IP by communicating with the server:
# Stop apparmor from blocking a custom dhcp instance service apparmor stop # Terminate any (old) existing dhcp instance sudo killall dhcpd # Spawn a dhcp server, in the kernel/ directory do: sudo dhcpd -f -d tap0 --no-pid -cf ./tests/dhcpd.conf
A fully automated CI test that checks the network using ping is available as well, it can be invoked with the following command:
RUST_TEST_THREADS=1 cargo test --test '*' -- s04_userspace_rumprt_net
socat is a helpful utility on the host to interface with the network, for
example to open a UDP port and print on incoming packets on the command line,
the following command can be used:
socat UDP-LISTEN:8889,fork stdout
Similarly we can use
netcat to connect to a port and send a payload:
nc 172.31.0.10 6337
The integration tests
s04_userspace_rumprt_net make use
of those tool to verify that networking is working as expected.
tcpdump is another handy tool to see all packets that are exchanged on a given
interface etc. For debugging nrk network issues, this command is useful as it displays
all packets on
tcpdump -i tap0 -vvv -XX