This project was done as part of Google Summer of Code 2019 and the following are evaluation reports prepared for the same. These reports were originally published during the GSoC period on the NetBSD blog and can be found here, here and here.

First Evaluation Report

The first coding period of The Google Summer of Code has come to an end. It has been a great experience so far and I got the opportunity to learn a lot of new stuff. This is a report on the work I have during this coding period.

About TriforceAFL

TriforceAFL is a modified version of AFL that supports fuzzing using QEMU’s full system emulation. This offers several advantages such as the fact that pieces of the kernel need not be recompiled with AFL or that the kernel does not need to be built with coverage support. More details on other advantages, design and implementation of TriforceAFL can be found here.

The TriforceLinuxSyscallFuzzer and the TriforceOpenBSDSyscallFuzzer are syscall fuzzers built on top of TriforceAFL. The end goal of this project is to adapt TriforceAFL for NetBSD syscall fuzzing.

Adapted TriforceAFL for pkgsrc-wip

One of the end goals of the project is to make the fuzzer available as a pkgsrc package. To do so, TriforceAFL had to be first ported to pkgsrc. TriforceAFL uses qemu, so the appropriate patches to qemu for NetBSD were applied and few other minor issues resolved. The working package is now available in pkgsrc-wip.

The NetBSD Syscall Fuzzer

TriforceNetBSDSyscallFuzzer can be now used to perform system call fuzzing of NetBSD kernels using AFL and QEMU. The setup scripts and the driver program are functioning. The syscalls list has been updated for NetBSD and basic input generation works. Documentation detailing the process of setup(of the NetBSD installation/kernel image), building and fuzzing along with the code is available on github.

The fuzzer functions properly and detects crashes which can be reproduced using the driver. Although it can severely benefit from better input generation and optimisation. This will be the focus in the next coding period.

Summary

In the coming weeks, the work of optimizing the fuzzer is to be done, so that it is more efficient at catching bugs, I am looking forward to doing so and making TriforceNetBSDSyscallFuzzer available on NetBSD through pkgsrc.

Lastly I would like to thank my mentor, Kamil Rytarowski for always helping me through the process and guiding me whenever I needed any help.

Second Evaluation Report

I have been working on adapting TriforceAFL for NetBSD kernel syscall fuzzing. This blog post summarizes the work done until the second evaluation.

Input Generation

For a feedback driven mutation based fuzzer that is TriforceAFL, fuzzing can be greatly improved by providing it with proper input test cases. The fuzzer can then alter parts of the valid input leading to more coverage and hopefully more bugs. The TriforceNetBSDSyscallFuzzer itself was a working fuzzer at the end of the first evaluation, but it was missing some proper input generation for most of the syscalls. A greater part of the time during this coding period was spent adding and testing basic templates for a majority of NetBSD syscalls, scripts have also been added for cases where more complex input generation was required. This should now allow the fuzzer to find bugs it previously could not have.

Templates for 160 of the 483 syscalls in NetBSD have been added, below is the complete list:

    1 exit, 2 fork, 3 read, 4 write, 5 open, 6 close, 7 compat_50_wait4, 8 compat_43_ocreat, 9 link, 10 unlink, 12 chdir, 13 fchdir, 14 compat_50_mknod, 15 chmod, 16 chown, 17 break, 19 compat_43_olseek, 20 getpid, 22 unmount, 23 setuid, 24 getuid, 25 geteuid, 26 ptrace, 33 access, 34 chflags, 35 fchflags, 36 sync, 37 kill, 39 getppid, 41 dup, 42 pipe, 43 getegid, 44 profil, 45 ktrace, 47 getgid, 49 __getlogin, 50 __setlogin, 51 acct, 55 compat_12_oreboot, 56 revoke, 57 symlink, 58 readlink, 59 execve, 60 umask, 61 chroot, 62 compat_43_fstat43, 63 compat_43_ogetkerninfo, 64 compat_43_ogetpagesize, 66 vfork, 73 munmap, 78 mincore, 79 getgroups, 80 setgroups, 81 getpgrp, 82 setpgid, 83 compat_50_setitimer, 86 compat_50_getitimer, 89 compat_43_ogetdtablesize, 90 dup2, 95 fsync, 96 setpriority, 97 compat_30_socket, 100 getpriority, 106 listen, 116 compat_50_gettimeofday, 117 compat_50_getrusage, 120 readv, 121 writev, 122 compat_50_settimeofday, 123 fchown, 124 fchmod, 126 setreuid, 127 setregid, 128 rename, 131 flock, 132 mkfifo, 134 shutdown, 135 socketpair, 136 mkdir, 137 rmdir, 140 compat_50_adjtime, 147 setsid, 161 compat_30_getfh, 165 sysarch, 181 setgid, 182 setegid, 183 seteuid, 191 pathconf, 192 fpathconf, 194 getrlimit, 195 setrlimit, 199 lseek, 200 truncate, 201 ftruncate, 206 compat_50_futimes, 207 getpgid, 209 poll, 231 shmget, 232 compat_50_clock_gettime, 233 compat_50_clock_settime, 234 compat_50_clock_getres, 240 compat_50_nanosleep, 241 fdatasync, 242 mlockall, 243 munlockall, 247 _ksem_init, 250 _ksem_close, 270 __posix_rename, 272 compat_30_getdents, 274 lchmod, 275 lchown, 276 compat_50_lutimes, 289 preadv, 290 pwritev, 286 getsid, 296 __getcwd, 306 utrace, 344 kqueue, 157 compat_20_statfs, 158 compat_20_fstatfs, 416 __posix_fadvise50, 173 pread, 174 pwrite, 197 mmap, 462 faccessat, 463 fchmodat, 464 fchownat, 461 mkdirat, 459 mkfifoat, 460 mknodat, 468 openat, 469 readlinkat, 458 renameat, 470 symlinkat, 471 unlinkat, 453 pipe2, 467 utimensat

A separate script (targ/gen2.py) generating tricker input cases was added for the following:

    104 bind, 105 setsockopt, 118 getsockopt, 98 connect, 30 accept, 31 getpeername, 32 getsockname, 133 sendto, 29 recvfrom, 21 compat_40_mount, 298 compat_30_fhopen 299 compat_30_fhstat, 300 compat_20_fhstatfs, 93 compat_50_select, 373 compat_50_pselect, 345 compat_50_kevent, 92 fcntl, 74 mprotect, 203 mlock, 273 minherit, 221 semget, 222 semop, 202 __sysctl

Reproducibility

The fuzzer uses the simplest way to reproduce a crash which is by storing the exact input for the test case that resulted in a crash. This input can then be passed to the driver program, which will be able to parse the input and execute the syscalls that were executed in the order that they were executed. A better prototype reproducer generator has now been added to the fuzzer that provides us with human readable and executable C code. This C code can be compiled and executed to reproduce the crash. To generate the reproducers simply run the genRepro script from the targ directory. If everything goes right, the C reproducers will now be available in targ/reproducers, in separate files as follows:

// id:000009,sig:00,src:005858+002155,op:splice,rep:8
#include <sys/syscall.h>
#include <unistd.h>

int main() {
    __syscall( SYS_mincore, 0x/* removed from the report due to security concerns */, 0x/* ... */, 0x/* ... */);
    return 0;
}

// id:000010,sig:00,src:005859+004032,op:splice,rep:2
#include <sys/syscall.h>
#include <unistd.h>

int main() {
    __syscall( SYS_mprotect, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    __syscall( SYS_mincore, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    return 0;
}

// id:000011,sig:00,src:005859+004032,op:splice,rep:4
#include <sys/syscall.h>
#include <unistd.h>

int main() {
    __syscall( SYS_mprotect, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    __syscall( SYS_mincore, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    return 0;
}

The reproducers currently do not include the allocated memory and such, so not all reproducers will work. More improvements are to come, but this will hopefully make analysis of the crashes easier.

Fuzzing

The fuzzer was run for ~4 days. We were getting ~50 execs/sec on a single core. Please note that this is using qemu with softemu (TCG) as hardware acceleration cannot be used here. During this period the fuzzer detected 23 crashes that it marked as unique. Not all of these were truly unique, there is the scope of adding a secondary filter here to detect truly unique crashes. Most of them were duplicates of the crashes highlighted below:

1. compat_43_osendmsg - tcp_output: no template

call 114 - compat_43_osendmsg
arg 0: argStdFile 4 - type 12
arg 1: argVec64 77d0549cc000 - size 4
arg 2: argNum 8003

read 83 bytes, parse result 0 nrecs 1
syscall 114 (4, 77d0549cc000, 8003)

[ 191.8169124] panic: tcp_output: no template
[ 191.8169124] cpu0: Begin traceback...
[ 191.8269174] vpanic() at netbsd:vpanic+0x160
[ 191.8269174] snprintf() at netbsd:snprintf
[ 191.8269174] tcp_output() at netbsd:tcp_output+0x2869
[ 191.8385864] tcp_sendoob_wrapper() at netbsd:tcp_sendoob_wrapper+0xfe
[ 191.8469824] sosend() at netbsd:sosend+0x6e3
[ 191.8469824] do_sys_sendmsg_so() at netbsd:do_sys_sendmsg_so+0x231
[ 191.8469824] do_sys_sendmsg() at netbsd:do_sys_sendmsg+0xac
[ 191.8569944] compat_43_sys_sendmsg() at netbsd:compat_43_sys_sendmsg+0xea
[ 191.8569944] sys___syscall() at netbsd:sys___syscall+0x74
[ 191.8655484] syscall() at netbsd:syscall+0x181
[ 191.8655484] --- syscall (number 198) ---
[ 191.8655484] 40261a:
[ 191.8655484] cpu0: End traceback...
[ 191.8655484] fatal breakpoint trap in supervisor mode
[ 191.8655484] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x202 cr2
 0x7f7795402000 ilevel 0x4 rsp 0xffffc58032d589f0
[ 191.8655484] curlwp 0xfffff7e8b1c63220 pid 700.1 lowest kstack 0xffffc58032d55
2c0
Stopped in pid 700.1 (driver) at        netbsd:breakpoint+0x5:  leave

2. mincore - uvm_fault_unwire_locked: address not in map

call 78 - mincore
arg 0: argNum d000000
arg 1: argNum 7600000000000000
arg 2: argNum 1b0000000000
read 65 bytes, parse result 0 nrecs 1
syscall 78 (d000000, 7600000000000000, 1b0000000000)

[ 141.0578675] panic: uvm_fault_unwire_locked: address not in map
[ 141.0578675] cpu0: Begin traceback...
[ 141.0691345] vpanic() at netbsd:vpanic+0x160
[ 141.0691345] snprintf() at netbsd:snprintf
[ 141.0774205] uvm_fault_unwire() at netbsd:uvm_fault_unwire
[ 141.0774205] uvm_fault_unwire() at netbsd:uvm_fault_unwire+0x29
[ 141.0774205] sys_mincore() at netbsd:sys_mincore+0x23c
[ 141.0884435] sys___syscall() at netbsd:sys___syscall+0x74
[ 141.0884435] syscall() at netbsd:syscall+0x181
[ 141.0884435] --- syscall (number 198) ---
[ 141.0996065] 40261a:
[ 141.0996065] cpu0: End traceback...
[ 141.0996065] fatal breakpoint trap in supervisor mode
[ 141.0996065] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x202 cr2
 0x761548094000 ilevel 0 rsp 0xffff870032e48d90
[ 141.0996065] curlwp 0xffff829094e51b00 pid 646.1 lowest kstack 0xffff870032e45
2c0
Stopped in pid 646.1 (driver) at        netbsd:breakpoint+0x5:  leave

3. extattrctl - KASSERT fail

call 360 - extattrctl
arg 0: argBuf 7b762b514044 from 2 bytes
arg 1: argNum ff8001
arg 2: argFilename 7b762b515020 - 2 bytes from /tmp/file0
arg 3: argNum 0
arg 4: argNum 0
arg 5: argNum 2100000000
read 59 bytes, parse result 0 nrecs 1
syscall 360 (7b762b514044, ff8001, 7b762b515020, 0, 0, 2100000000)

[ 386.4528838] panic: kernel diagnostic assertion "fli->fli_trans_cnt == 0" fail
ed: file "src/sys/kern/vfs_trans.c", line 201
[ 386.4528838] cpu0: Begin traceback...
[ 386.4528838] vpanic() at netbsd:vpanic+0x160
[ 386.4648968] stge_eeprom_wait.isra.4() at netbsd:stge_eeprom_wait.isra.4
[ 386.4724138] fstrans_lwp_dtor() at netbsd:fstrans_lwp_dtor+0xbd
[ 386.4724138] exit1() at netbsd:exit1+0x1fa
[ 386.4724138] sys_exit() at netbsd:sys_exit+0x3d
[ 386.4832968] syscall() at netbsd:syscall+0x181
[ 386.4832968] --- syscall (number 1) ---
[ 386.4832968] 421b6a:
[ 386.4832968] cpu0: End traceback...
[ 386.4832968] fatal breakpoint trap in supervisor mode
[ 386.4944688] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x202 cr2
 0xffffc100324bd000 ilevel 0 rsp 0xffffc10032ce9dc0
[ 386.4944688] curlwp 0xfffff6278e2fc240 pid 105.1 lowest kstack 0xffffc10032ce6
2c0
Stopped in pid 105.1 (driver) at        netbsd:breakpoint+0x5:  leave

pkgsrc Package

Lastly, the TriforceNetBSDSyscallFuzzer has now been made available in the form of a pkgsrc package in pkgsrc/wip as triforcenetbsdsyscallfuzzer. The package will require wip/triforceafl which was ported earlier. All other changes mentioned can be found in the github repo.

script(1) recording A typescript recording of a functional TriforceAFL fuzzer setup and execution is available here. Download it and replay it with script -p.

Future Work

Work that remains to be done include:

1. Restructuring of files

   The file structure needs to be modified to suit the specific case of the Host and Target being the same OS. Right now, files are separated into Host and Target directories, this is not required.

2. Testing with Sanitizers enabled

   Until now the fuzzing done was without using KASAN or kUBSAN. Testing with them enabled and fuzzing with them will be of major focus in the third coding period.

3. Improving the ’reproducer generator’

   There is some scope of improvement for the prototype that was added. Incremental updates to it are to be expected.

4. Analysis of crash reports and fixing bugs

5. Documentation

Summary

So far, the TriforceNetBSDSyscallFuzzer has been made available in the form of a pkgsrc package with the ability to fuzz most of NetBSD syscalls. In the final coding period of GSoC. I plan to analyse the crashes that were found until now. Integrate sanitizers, try and find more bugs and finally wrap up neatly with detailed documentation.

Last but not least, I would like to thank my mentor, Kamil Rytarowski for helping me through the process and guiding me. It has been a wonderful learning experience so far!

Final Evaluation Report

This is the third report summarising the work done in the third coding period for the GSoC project of Adapting TriforceAFL for NetBSD kernel syscall fuzzing.

This post also outlines the work done throughout the duration of GSoC, describes the implications of the same and future improvements to come.

Current State

As of now TriforceAFL has been made available in the form of a pkgsrc package (wip/triforceafl). This package allows you to essentially fuzz anything in QEMU’s full system emulation mode using AFL. TriforceNetBSDSyscallFuzzer is built on top of TriforceAFL, specifically to fuzz the NetBSD kernel syscalls. It has also now been made available as wip/triforcenetbsdsyscallfuzzer. Several minor issues found in the above two packages have now been resolved, and the project restructured.

Issues found include:

• The input generators would also test the the generated inputs, causing several syscalls to be executed which messed up permissions of several other directories and lead to other unwanted consequences. Testing of syscalls has now been disabled as the working ones are specified.
• Directory specified for the BIOS, VGA BIOS and keymaps for QEMU in the runFuzz script was incorrect. This was because wip/triforceafl did not install the specified directory. The package has now been patched to install the pc-bios directory.
• The project was structured in a manner where the host was Linux and the target was NetBSD. Since that is no longer the case and the fuzzer is meant to be run on a NetBSD machine, the project was restructured so that everything is now in one directory and there is no longer a need to move files around.

The packages should now work as intended by following the instructions outlined in the README.

The fuzzer was able to detect a few bugs in the last coding period, details can be found in the last report. During this coding period, the Fuzzer was able to detect 79 unique crashes in a period of 2 weeks running on a single machine. The kernel was built with DEBUG + LOCKDEBUG + DIAGNOSTIC. Work is underway to analyse, report and fix the new bugs and to make the process faster.

With an initial analysis on the outputs on the basis of the syscall that lead to the crash, 6 of the above crashes were unique bugs, the rest were duplicates or slight variants, of which 3 have been previously reported. Here are the backtraces of the new bugs found (The reproducers were run with kUBSan Enabled):

BUG1:

[ 110.4035826] panic: cv_enter,172: uninitialized lock (lock=0xffffe3c1b9
fc0c50, from=ffffffff81a436e9)
[ 110.4035826] cpu0: Begin traceback...
[ 110.4035826] vpanic() at netbsd:vpanic+0x1fd
[ 110.4035826] snprintf() at netbsd:snprintf
[ 110.4035826] lockdebug_locked() at netbsd:lockdebug_locked+0x45e
[ 110.4035826] cv_timedwait_sig() at netbsd:cv_timedwait_sig+0xe7
[ 110.4035826] lfs_segwait() at netbsd:lfs_segwait+0x6e
[ 110.4035826] sys___lfs_segwait50() at netbsd:sys___lfs_segwait50+0xe2
[ 110.4035826] sys___syscall() at netbsd:sys___syscall+0x121
[ 110.4035826] syscall() at netbsd:syscall+0x1a5
[ 110.4035826] --- syscall (number 198) ---
[ 110.4035826] 40261a:
[ 110.4035826] cpu0: End traceback...
[ 110.4035826] fatal breakpoint trap in supervisor mode
[ 110.4035826] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x282 cr2
 0x73f454b70000 ilevel 0x8 rsp 0xffff8a0068390d70
[ 110.4035826] curlwp 0xffffe3c1efc556a0 pid 709.1 lowest kstack 0xffff8a006838d
2c0
Stopped in pid 709.1 (driver) at        netbsd:breakpoint+0x5:  leave
db{0}> bt
breakpoint() at netbsd:breakpoint+0x5
vpanic() at netbsd:vpanic+0x1fd
snprintf() at netbsd:snprintf
lockdebug_locked() at netbsd:lockdebug_locked+0x45e
cv_timedwait_sig() at netbsd:cv_timedwait_sig+0xe7
lfs_segwait() at netbsd:lfs_segwait+0x6e
sys___lfs_segwait50() at netbsd:sys___lfs_segwait50+0xe2
sys___syscall() at netbsd:sys___syscall+0x121
syscall() at netbsd:syscall+0x1a5
--- syscall (number 198) ---
40261a:

BUG2:

[ 161.4877660] panic: LOCKDEBUG: Mutex error: rw_vector_enter,296: spin lock hel
d
[ 161.4877660] cpu0: Begin traceback...
[ 161.4877660] vpanic() at netbsd:vpanic+0x1fd
[ 161.4877660] snprintf() at netbsd:snprintf
[ 161.4877660] lockdebug_abort1() at netbsd:lockdebug_abort1+0x115
[ 161.4877660] rw_enter() at netbsd:rw_enter+0x645
[ 161.4877660] uvm_fault_internal() at netbsd:uvm_fault_internal+0x1c5
[ 161.4877660] trap() at netbsd:trap+0xa71
[ 161.4877660] --- trap (number 6) ---
[ 161.4877660] config_devalloc() at netbsd:config_devalloc+0x644
[ 161.4877660] config_attach_pseudo() at netbsd:config_attach_pseudo+0x1c
[ 161.4877660] vndopen() at netbsd:vndopen+0x1f3
[ 161.4877660] cdev_open() at netbsd:cdev_open+0x12d
[ 161.4877660] spec_open() at netbsd:spec_open+0x2d0
[ 161.4877660] VOP_OPEN() at netbsd:VOP_OPEN+0xba
[ 161.4877660] vn_open() at netbsd:vn_open+0x434
[ 161.4877660] sys_ktrace() at netbsd:sys_ktrace+0x1ec
[ 161.4877660] sys___syscall() at netbsd:sys___syscall+0x121
[ 161.4877660] syscall() at netbsd:syscall+0x1a5
[ 161.4877660] --- syscall (number 198) ---
[ 161.4877660] 40261a:
[ 161.4877660] cpu0: End traceback...
[ 161.4877660] fatal breakpoint trap in supervisor mode
[ 161.4877660] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x286 cr2
 0xfffffffffffff800 ilevel 0x8 rsp 0xffff9c80683cd4f0
[ 161.4877660] curlwp 0xfffffcbcda7d36a0 pid 41.1 lowest kstack 0xffff9c80683ca2
c0
db{0}> bt
breakpoint() at netbsd:breakpoint+0x5
vpanic() at netbsd:vpanic+0x1fd
snprintf() at netbsd:snprintf
lockdebug_abort1() at netbsd:lockdebug_abort1+0x115
rw_enter() at netbsd:rw_enter+0x645
uvm_fault_internal() at netbsd:uvm_fault_internal+0x1c5
trap() at netbsd:trap+0xa71
--- trap (number 6) ---
config_devalloc() at netbsd:config_devalloc+0x644
config_attach_pseudo() at netbsd:config_attach_pseudo+0x1c
vndopen() at netbsd:vndopen+0x1f3
cdev_open() at netbsd:cdev_open+0x12d
spec_open() at netbsd:spec_open+0x2d0
VOP_OPEN() at netbsd:VOP_OPEN+0xba
vn_open() at netbsd:vn_open+0x434
sys_ktrace() at netbsd:sys_ktrace+0x1ec
sys___syscall() at netbsd:sys___syscall+0x121
syscall() at netbsd:syscall+0x1a5
--- syscall (number 198) ---
40261a:

BUG3:

[ 350.9942146] UBSan: Undefined Behavior in /home/ubuntu/triforce/kernel/
src/sys/kern/kern_ktrace.c:1398:2, member access within misaligned address 0x2b0
000002a for type 'struct ktr_desc' which requires 8 byte alignment
[ 351.0025346] uvm_fault(0xffffffff85b73100, 0x2b00000000, 1) -> e
[ 351.0025346] fatal page fault in supervisor mode
[ 351.0025346] trap type 6 code 0 rip 0xffffffff81b9dbf9 cs 0x8 rflags 0x286 cr2
 0x2b00000032 ilevel 0 rsp 0xffff8780684d7fb0
[ 351.0025346] curlwp 0xffffa992128116e0 pid 0.54 lowest kstack 0xffff8780684d42
c0
kernel: page fault trap, code=0
Stopped in pid 0.54 (system) at netbsd:ktrace_thread+0x1fd:     cmpq    %rbx,8(%
r12)
db{0}> bt
ktrace_thread() at netbsd:ktrace_thread+0x1fd

Reproducing Crashes

Right now the best way to reproduce a bug detected is to use the Fuzzer’s driver program itself:

./driver -tv < crash_file

The crash_file can be found in the outputs directory and is a custom file format made for the driver. Memory allocation and Socket Creation remain to be added to the reproducer generator(genRepro) highlighted in the previous post and will be prioritised in the future.

Implications

Considering that we have a working fuzzer now, it is a good time to analyse how effective TriforceAFL is compared to other fuzzers.

Recently Syzkaller has been really effective in finding bugs in NetBSD. Both TriforceAFL and Syzkaller create multiple instances of the system to be fuzzed, gather coverage data, mutate input accordingly and continue fuzzing, but there are several differences in the way they work.

Key differences between the two include:

1. KCOV

   Syzkaller relies on the KCOV module for coverage data in NetBSD whereas TriforceAFL gets coverage information from its modified version of QEMU.

2. VMs

   Syzkaller creates multiple VM’s and manages them with syz-manager. Whereas TriforceAFL simply forks the VM for each testcase.

3. Communication

   Syzkaller uses RPC and ssh to communicate with the VMs whereas TriforceAFL uses a custom hypercall.

4. Hardware Acceleration

   Syzkaller can use hardware acceleration to run the VMs at native speeds, whereas TriforceAFL can only utilize QEMU’s full system emulation mode, as it relies on it for coverage data.

Comparison

These differences lead to very different results. To get a perspective, here are some stats from syzkaller, which can be found on the syzbot dashboard.

Comparatively in 1st Weekend of Fuzzing, Triforce could only find 3 bugs whereas syzkaller found 18.

• Compared to syzkaller, the number of bugs found by TriforceAFL in the first few days were significantly less, but nevertheless TriforceAFL was able to find variants of bugs found by syzkaller and 1 which was different with simpler reproducers.
• Going by the stats provided by Syzbot and AFL, Syzkaller does ~80 execs / min whereas TriforceAFL can do ~1500 execs / min on average, Although this statistic is without any of the sanitizers enabled for TriforceAFL and kASan enabled for Syzkaller.
• TriforceAFL has an advantage when it comes to the fact that it does not rely on KCOV for coverage data. This means it can get coverage data for fuzzing other interfaces easily too. This will be beneficial when we move onto Network and USB Fuzzing. Issues have been found with KCOV where in certain cases the fuzzer lost track of the kernel trace, especially in networking where after enqueuing a networking packet the fuzzer lost track as the packet was then handled by some other thread. Coverage data gathered using TriforceAFL will not be sensitive to this, considering that noise from the kernel is handled in some way to an extent.
• Efforts were made in the original design of TriforceAFL to make the traces as deterministic as possible at a basic block level. To quote from this article: Sometimes QEMU starts executing a basic block and then gets interrupted. It may then re-execute the block from the start or translate a portion of the block that wasn’t yet executed and execute that. This introduced non-determinism. To reduce the non-determinism, cpu-exec.c was altered to disable QEMU’s “chaining” feature, and moved AFL’s tracing feature to cputbexec to trace a basic block only after it has been executed to completion. Although nothing has been specifically done to reduce noise from the kernel, such as disabling coverage during interrupts.
• TriforceAFL runs 1 fuzzing process per instance, whereas Syzkaller can run upto 32 fuzzing processes. But multiple fuzzing instance can be created with TriforceAFL as detailed in this document to take advantage of parallelization.
• Maxime Villard was also able to rework TriforceNetBSDSyscallFuzzer to now also support compatnetbsd32 kernel. This work will be integrated as soon as possible.
• On the other hand, Syzkaller has syzbot which can be thought of as a 24/7 fuzzing service. TriforceAFL does not have a such a service or a dedicated server for fuzzing. A service like this will surely be advantageous in the future, but it is still a bit too early to set up one. To truly efficiently utilize such a service, it would be better to improve TriforceAFL in all ways possible first.

Targets to be met before a 24/7 fuzzing service is setup, include but are not limited to:

1. Automatic initial Analysis & Report Generation

   Right now, There is nothing that can automatically perform an initial analysis to detect truly unique crashes and prepare reports with backtraces, this will be required and very helpful.

2. Better Reproducers

   As mentioned before, the generated reproducers do not take care of Memory Allocation and Socket Creation, etc. These need to be included, if a good C reproducer is expected.

3. Parallel Fuzzing and Management

   There is no central interface that summarises the statistics from all fuzzing instances, and manages such instances in case of anomalies/errors. Something like this would be needed for a service that will be run for longer periods of time without supervision.

4. Updated AFL and QEMU versions

   Updated AFL and QEMU might significantly increase executions per second and lead to better mutation of inputs and also decrease the memory requirements.

5. Fuzzing more Interfaces

   Right now we are only fuzzing syscalls, Network and USB Layers are great future prospects, at least prototype versions can be added in the near future.

Future Work

Although GSoC will be officially ending, I am looking forward to continuing the development of TriforceAFL, adding features and making it more effective. Some improvements that can be expected include:

• Fuzzing different interfaces - Network Fuzzing, USB Fuzzing
• Updating AFL and QEMU versions
• Better Reproducers
• Syzbot like service

A new repo has been created at https://github.com/NetBSD/triforce4netbsd. Collaborative work in the future will be updated here.

Conclusion

TriforceAFL has been successfully adapted for NetBSD and all of the original goals of the GSoC proposal have been met, but the work is far from complete. Work done so far shows great potential, and incremental updates will surely make TriforceAFL a great fuzzer. I am looking forward to continuing the work and making sure that this is the case. GSoC has been an amazing learning experience! I would like to thank Maxime Villard for taking an active interest in TriforceAFL and for assisting in testing and development. I would like to thank my mentor Kamil Rytarowski for being the guiding force throughout the project and for assisting me whenever I needed help. And finally I would like to thank Google for giving me this wonderful opportunity.