I'm cross-compiling a large codebase (LLVM and sub-projects such as Clang) and want to access the full tree - the source and the build artifacts from under qemu. This post documents the results of my experiments in various ways to do this. Note that I'm explicitly choosing to run timings using something that approximates the work I want to do rather than any microbenchmark targeting just file access time.
Requirements:
The test simply involves taking a cross-compiled LLVM tree (with RISC-V as the
only enabled target) and running the equivalent of ninja check-llvm
on it
after exposing / transferring it to the VM using the listed method. Results in
chart form for overall time taken in the "empty cache" case:
9pfs | |
---|---|
virtiofsd | |
squashfs | |
ext4 |
By way of comparison, it takes ~10m40s to naively transfer the data by piping it over ssh (tarring on one end, untarring on the other).
For this use case, building and mounting a filesystem seems the most
compelling option with the ext4 being overall simplest (no need to use
overlayfs) assuming you have an e2fsprogs new enough to support tar input to
mkfs.ext4
. I'm not sure why I'm not seeing the purported performance
improvements when using virtiofsd.
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
)
before building the filesystem (if necessary) and launching QEMU.tar
command described below.time ../bin/llvm-lit -sv .
running from
test/
within the build directory, and combining this with the timing for
whatever command(s) are needed to transfer and extract the source+build
tree.lit
with --order=lexical
to
ensure runtimes aren't affected by the order of tests (e.g. long-running
tests being scheduled last).-virtfs local,path=$HOME/llvm-project/,mount_tag=llvm,security_model=none,id=llvm
to the QEMU command line.sudo mount -t 9p -o trans=virtio,version=9p2000.L,msize=512000 llvm /mnt
.
I haven't swept different parameters for
msize
and mount reports 512000 is the maximum for the virtio transport.chown
ing everything (which would also require
security_model=mapped-xattr
or security_model=mapped-file
to be passed
to qemu. In the future you should be able to do sudo mount --bind --map-users 1000:1001:1 --map-groups 984:1001:1 /mnt mapped
mkdir -p upper work llvm-project
sudo mount -t overlay overlay -o lowerdir=/mnt,upperdir=$HOME/upper,workdir=$HOME/work $HOME/llvm-project
/usr/lib/virtiofsd --socket-path=/tmp/vhostqemu --shared-dir=$HOME/llvm-project --cache=always
-object
seems to need to match the -m
argument you used): -chardev socket,id=char0,path=/tmp/vhostqemu \
-device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=llvm \
-object memory-backend-memfd,id=mem,size=64G,share=on -numa node,memdev=mem
sudo mount -t virtiofs llvm /mnt
mkdir -p upper work llvm-project
sudo mount -t overlay overlay -o lowerdir=/mnt,upperdir=$HOME/upper,workdir=$HOME/work $HOME/llvm-project
find
to generate the
individual directory exclusions.mksquashfs ~/llvm-project llvm-project.squashfs -no-compression -force-uid 1001 -force-gid 1001 -e \ .git $(find $HOME/llvm-project/build -maxdepth 1 -mindepth 1 -type d -not -name 'stage2cross' -printf '-e build/%P ')
-drive file=$HOME/llvm-project.squashfs,if=none,id=llvm,format=raw
sudo mount -t squashfs /dev/vdb /mnt
-d
parameter of mkfs.ext4
which allows you to create a
filesystem, using the given directory or tarball to initialise it. As
mkfs.ext4
lacks features for filtering the input or forcing the uid/gid of
files, we rely on tarball input (only just added in e2fsprogs 1.47.1).fallocate -l 15GiB llvm-project.img
tar --create \
--file=- \
--owner=1001 \
--group=1001 \
--exclude=.git \
$(find $HOME/llvm-project/build -maxdepth 1 -mindepth 1 -type d -not -name 'stage2cross' -printf '--exclude=build/%P ') \
-C $HOME/llvm-project . \
| mkfs.ext4 -d - llvm-project.img
-device virtio-blk-device,drive=hdb -drive file=$HOME/llvm-project.img,format=raw,if=none,id=hdb
sudo mount -t ext4 /dev/vdb $HOME/llvm-project
netcat
), a better networking backend to QEMU, and so
on. But the intent is to show the cost of the naive "obvious" solution.tar --create \
--file=- \
--owner=1001 \
--group=1001 \
--exclude=.git \
$(find $HOME/llvm-project/build -maxdepth 1 -mindepth 1 -type d -not -name 'stage2cross' -printf '--exclude=build/%P ') \
-C $HOME/llvm-project . \
| ssh -p10222 asb@localhost "mkdir -p llvm-project && tar xf - -C \
llvm-project"