Thanks to the Debian 64-bit RISC-V port it's really easy to build a sysroot appropriate for cross-compiling Clang/LLVM and its separate test suite. Either use my rootless-deboostrap-wrapper script or the command I documented in LLVM's cross-compilation instructions, being sure to see the note on working around a Ninja dependency issue. For a bootable QEMU image, Debian-based recipes are similarly straightforward. But we don't have the luxury of a precompiled distribution for 32-bit RISC-V and so we'll lean on Yocto to produce the needed sysroot by building from source. I cover three cases:
In this article I use release 6.0 ('Wrynose') and the bitbake-setup helper
tool.
For documentation, I found the Yocto quick build
guide, and
bitbake-setup
docs, and
image customisation
guide
helpful.
I'm not a Yocto developer, so if you are reading this and think there are other approaches to consider or alternative ways of solving the problem that are better, please do drop me a note!
I'm running on Arch Linux which isn't one of the tested Yocto host distributions, but seemed to work just fine.
I found I needed to enable the en_US locale:
sudo sed /etc/locale.gen -i -e "s/^\#en_US.UTF-8 UTF-8.*/en_US.UTF-8 UTF-8/"
sudo locale-gen
And install the following additional packages:
sudo pacman -S inetutils chrpath cpio diffstat rpcsvc-proto flex bison zstd
Now we will check out bitbake into a work directory and set a directory to be used to hold downloaded files:
mkdir yocto-work && cd yocto-work
git clone https://git.openembedded.org/bitbake
./bitbake/bin/bitbake-setup settings set default dl-dir $HOME/.cache/yocto/dl
As is often the case, the workload I'm interested in here is LLVM. If you're looking to build a sysroot to cross-compile something else, you may need a slightly different package list.
In this first stanza, we use bitbake-setup to initialise our development
environment. Because there isn't a predefined machine target for riscv32 in
bitbake/default-registry/configurations/poky-wrynose.conf.json, we
avoid selecting machine and will address it later. Importantly, we set a
SSTATE_DIR which will be used for the shared state cache, avoiding
rebuilding packages when not necessary (I'm not totally sure when this isn't
exposed in bitbake-setup settings like dl-dir is). Another relevant
variable is BB_HASHSERVE_BB_DIR which controls where the hash equivalence
database is stored. But with current bitbake-setup this defaults to
SSTATE_DIR, so there's no need to set it explicitly.
./bitbake/bin/bitbake-setup init --non-interactive \
--skip-selection machine \
./bitbake/default-registry/configurations/poky-wrynose.conf.json \
poky \
distro/poky
printf 'SSTATE_DIR = "%s"\n' "$HOME/.cache/yocto/sstate" >> bitbake-builds/site.conf
With that done, we can source the generated definitions to enter the build
environment (note we're using the default setup directory, you can override it
to something other than poky-wrynose by using --setup-dir-name) and
use enable-fragment to set the qemuriscv32 machine:
. bitbake-builds/poky-wrynose/build/init-build-env
bitbake-config-build enable-fragment machine/qemuriscv32
Now configure the build, indicating the additional libraries that need to be
present and run bitbake to actually produce it:
cat >> conf/local.conf <<'EOF'
IMAGE_INSTALL:append = " \
glibc-dev \
libgcc \
libgcc-dev \
libatomic \
libatomic-dev \
libstdc++ \
libstdc++-dev \
"
EOF
bitbake core-image-minimal
This results in 4624 build tasks and takes quite some time to complete if you
haven't run it before (i.e. aren't hitting in the sstate cache). The next
section of this article explores how to produce the needed output while
building much less, but let's finish the job and extract a rootfs from what
was built. I would like to now follow advice in the
documentation
and run runqemu-extract-sdk on the rootfs archive (I submitted a little
patch
upstream)
to fix this command for .zst which was applied:
runqemu-extract-sdk tmp/deploy/images/qemuriscv32/core-image-minimal-qemuriscv32.rootfs.tar.zst ~/rv32sysroot
At this point, you have a sysroot that's almost directly usable for
cross-compiling Clang/LLVM (with --target=riscv32-poky-linux) but there are
three finalisation steps we will perform:
$PATH after sourcing
build/init-build-env.mkdir -p "$HOME/rv32sysroot/usr/lib/gcc"
ln -s ../riscv32-poky-linux "$HOME/rv32sysroot/usr/lib/gcc/riscv32-poky-linux"
sysroot-relativelinks.py "$HOME/rv32sysroot"
ln -s usr/include "$HOME/rv32sysroot/include"
The core-image-minimal recipe above is straightforward, but does a lot more
work than strictly necessary. We can reduce this by instead adding a
dependency-only recipe that explicitly lists the needed build-time
dependencies and contains logic to produce the sysroot.
First, create a layer:
. bitbake-builds/poky-wrynose/build/init-build-env
bitbake-layers create-layer --add-layer ../layers/meta-rv32-llvm-sysroot
Then add the recipe:
recipe_dir="../layers/meta-rv32-llvm-sysroot/recipes-devtools/rv32-llvm-deps-sysroot"
mkdir -p "$recipe_dir"
cat > "$recipe_dir/rv32-llvm-deps-sysroot.bb" <<'EOF'
SUMMARY = "Dependency-only recipe to export an RV32 sysroot"
LICENSE = "MIT-0"
INHIBIT_DEFAULT_DEPS = "1"
EXCLUDE_FROM_WORLD = "1"
PACKAGE_ARCH = "${MACHINE_ARCH}"
DEPENDS = "virtual/libc libgcc virtual/${MLPREFIX}compilerlibs zlib zstd-native"
inherit deploy nopackages
do_configure[noexec] = "1"
do_compile[noexec] = "1"
do_install[noexec] = "1"
do_populate_sysroot[noexec] = "1"
do_deploy() {
export_dir="${WORKDIR}/${PN}-export"
rm -rf "$export_dir"
mkdir -p "$export_dir"
cp -a "${RECIPE_SYSROOT}/." "$export_dir/"
sysroot-relativelinks.py "$export_dir"
mkdir -p "$export_dir/usr/lib/gcc"
ln -s ../riscv32-poky-linux "$export_dir/usr/lib/gcc/riscv32-poky-linux"
ln -s usr/include "$export_dir/include"
tar -C "$export_dir" -cf - . | \
zstd -T0 -f -o "${DEPLOYDIR}/${PN}-${MACHINE}.tar.zst"
}
addtask deploy after do_prepare_recipe_sysroot before do_build
EOF
The do_deploy function implements the sysroot preparation logic that largely
mirrors the previous section. Otherwise, DEPENDS specifies the needed
dependencies (of these, virtual/${MLPREFIX}compilerlibs is a bit magic:
this resolves to the compiler runtime provider which pulls in things like
libstdc++).
Build the sysroot with:
bitbake rv32-llvm-deps-sysroot
This performs ~948 build tasks and will produce the sysroot tarball at
tmp/deploy/images/qemuriscv32/rv32-llvm-deps-sysroot-qemuriscv32.tar.zst.
You can use it by doing something like:
SYSROOT="$HOME/rv32depssysroot"
rm -rf "$SYSROOT"
mkdir -p "$SYSROOT"
tar --zstd -C "$SYSROOT" -xf \
tmp/deploy/images/qemuriscv32/rv32-llvm-deps-sysroot-qemuriscv32.tar.zst
The sysroot is slightly larger than the one in the section above because it
contains large unstripped static archives like usr/lib/libstdc++.a.
We could probably quibble on the definition of "featureful" as listed in the subheading above. For me, this means an image that boots using systemd and you can ssh into, roughly approximating what you get from my debootstrap recipes. But by adding other packages to the image recipe you can certainly make it more featureful.
First, let's start to set up the build environment and directories we'll use
for additional recipes. We use distro/poky-altcfg which is just Poky with
systemd as the init
manager.
cd yocto-work
./bitbake/bin/bitbake-setup init --non-interactive \
--setup-dir-name poky-wrynose-systemd \
--skip-selection machine \
./bitbake/default-registry/configurations/poky-wrynose.conf.json \
poky \
distro/poky-altcfg
. bitbake-builds/poky-wrynose-systemd/build/init-build-env
bitbake-config-build enable-fragment machine/qemuriscv32
bitbake-layers create-layer --add-layer ../layers/meta-rv32-qemu-image
mkdir -p \
../layers/meta-rv32-qemu-image/recipes-core/images \
../layers/meta-rv32-qemu-image/recipes-core/rv32-qemu-config/files
Some may prefer to split different aspects of image configuration into
independent recipes, but I opt to combine it into one for simplicity (in this
case, just configuring systemd-networkd dhcp and adding a config file that
will enable sudo for our user account):
cat > ../layers/meta-rv32-qemu-image/recipes-core/rv32-qemu-config/rv32-qemu-config.bb <<'EOF'
SUMMARY = "Configuration for RV32 systemd images"
LICENSE = "MIT-0"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT-0;md5=f41b3a5f969eb450434cf0e4f33449b9"
SRC_URI = " \
file://20-wired.network \
file://90-rv32-qemu \
"
RDEPENDS:${PN} = "systemd-networkd sudo"
FILES:${PN} = " \
${sysconfdir}/systemd/network/20-wired.network \
${sysconfdir}/sudoers.d/90-rv32-qemu \
"
S = "${UNPACKDIR}"
do_install() {
install -d ${D}${sysconfdir}/systemd/network
install -m 0644 ${S}/20-wired.network ${D}${sysconfdir}/systemd/network/20-wired.network
install -d ${D}${sysconfdir}/sudoers.d
install -m 0440 ${S}/90-rv32-qemu ${D}${sysconfdir}/sudoers.d/90-rv32-qemu
}
EOF
cat > ../layers/meta-rv32-qemu-image/recipes-core/rv32-qemu-config/files/20-wired.network <<'EOF'
[Match]
Type=ether
[Network]
DHCP=yes
EOF
cat > ../layers/meta-rv32-qemu-image/recipes-core/rv32-qemu-config/files/90-rv32-qemu <<'EOF'
%sudo ALL=(ALL) ALL
EOF
Now, we create the image recipe that will:
user account and set passwords of root and user to root and
user respectively. This follows the approach in the Yocto
docs.runqemu will configure things so we can connect ssh in via a
Unix domain socket (as done in the debootstrap-based article). Alternatively
you can choose to set QB_SLIRP_OPT = "-netdev user,id=net0,hostfwd=tcp:127.0.0.1:2222-:22" if you'd rather just connect
to localhost:2222.ROOT_PASSWORD_HASH="$(printf "%q" "$(openssl passwd -6 root)")"
USER_PASSWORD_HASH="$(printf "%q" "$(openssl passwd -6 user)")"
cat > ../layers/meta-rv32-qemu-image/recipes-core/images/rv32-qemu-systemd-ssh-image.bb <<EOF
SUMMARY = "Bootable RV32 QEMU image with systemd, networkd, and SSH access"
LICENSE = "MIT-0"
inherit image extrausers
IMAGE_FSTYPES = "ext4"
IMAGE_FEATURES = "allow-root-login"
QB_DEFAULT_FSTYPE = "ext4"
QB_CMDLINE_IP_SLIRP = "ip=none"
QB_SLIRP_OPT = "-netdev user,id=net0,hostfwd=unix:/tmp/yoctorv32.sock-:22"
SERIAL_CONSOLES = "115200;ttyS0"
ROOT_PASSWORD_HASH = "$ROOT_PASSWORD_HASH"
USER_PASSWORD_HASH = "$USER_PASSWORD_HASH"
IMAGE_INSTALL = "packagegroup-core-boot"
IMAGE_INSTALL += "os-release"
IMAGE_INSTALL += "systemd-networkd"
IMAGE_INSTALL += "systemd-serialgetty"
IMAGE_INSTALL += "rv32-qemu-config"
IMAGE_INSTALL += "openssh"
IMAGE_INSTALL += "sudo"
IMAGE_INSTALL += "bash"
IMAGE_INSTALL += "iproute2"
IMAGE_INSTALL += "iputils"
IMAGE_INSTALL += "procps"
EXTRA_USERS_PARAMS = " \\
groupadd sudo; \\
usermod -p '\${ROOT_PASSWORD_HASH}' root; \\
useradd -m -d /home/user -s /bin/bash -G sudo -p '\${USER_PASSWORD_HASH}' user; \\
"
EOF
Now write necessary configuration and build (disabling a number of distro features that would lead to larger build time). The following results in 4305 build tasks on my machine:
printf 'DL_DIR = "%s"\n' "$HOME/.cache/yocto/dl" >> conf/local.conf
printf 'SSTATE_DIR = "%s"\n' "$HOME/.cache/yocto/sstate" >> conf/local.conf
cat >> conf/local.conf <<'EOF'
PACKAGE_CLASSES = "package_ipk"
EXTRA_IMAGE_FEATURES = ""
IMAGE_FEATURES = ""
DISTRO_FEATURES:remove = "x11 wayland opengl alsa bluetooth wifi 3g nfc pcmcia usbgadget usbhost nfs zeroconf pulseaudio gobject-introspection-data"
SERIAL_CONSOLES = "115200;ttyS0"
EOF
bitbake rv32-qemu-systemd-ssh-image
Finally we can boot the image (snapshot means changes to the filesystem
image won't persist, just drop this if that isn't what you desire):
DEPLOY="$PWD/tmp/deploy/images/qemuriscv32"
rm /tmp/yoctorv32.sock
runqemu "$DEPLOY/rv32-qemu-systemd-ssh-image-qemuriscv32.rootfs.qemuboot.conf" nographic slirp snapshot
And connect via ssh with something like ssh root@unix/tmp/yoctorv32.sock.