Muxup

Building 32-bit RISC-V sysroots and images with Yocto

2026-05-11T12:00:00Z

Thanks to the Debian 64-bit RISC-V port it's really easy to build a sysroot appropriate for cross-compiling Clang/LLVM and its separate test suite. Either use my rootless-deboostrap-wrapper script or the command I documented in LLVM's cross-compilation instructions, being sure to see the note on working around a Ninja dependency issue. For a bootable QEMU image, Debian-based recipes are similarly straightforward. But we don't have the luxury of a precompiled distribution for 32-bit RISC-V and so we'll lean on Yocto to produce the needed sysroot by building from source. I cover three cases: 1) building a sysroot for cross-compiling projects like LLVM, 2) doing the same but in a way that requires fewer build steps, 3) building an image approximating my debootstrap image recipes.

In this article I use release 5.3 ('Whinlatter'), which introduced the bitbake-setup helper tool. For documentation, I found the Yocto quick build guide, and bitbake-setup docs, and image customisation guide helpful.

I'm not a Yocto developer, so if you reading this and think there are other approaches to consider or alternative ways of solving the problem that are better, please do drop me a note!

Common setup

I'm running on Arch Linux which isn't one of the tested Yocto host distributions, but seemed to work just fine.

I found I needed to enable the en_US locale:

sudo sed /etc/locale.gen -i -e "s/^\#en_US.UTF-8 UTF-8.*/en_US.UTF-8 UTF-8/"
sudo locale-gen

And install the following additional packages:

sudo pacman -S inetutils chrpath cpio diffstat rpcsvc-proto flex bison zstd

Now we will check out bitbake into a work directory and set a directory to be used to hold downloaded files:

mkdir yocto-work && cd yocto-work
git clone https://git.openembedded.org/bitbake
./bitbake/bin/bitbake-setup settings set default dl-dir $HOME/.cache/yocto/dl

Producing a sysroot based on core-image-minimal

As is often the case, the workload I'm interested in here is LLVM. If you're looking to build a sysroot to cross-compile something else, you may need a slightly different package list.

In this first stanza, we use bitbake-setup to initialise our development environment. Because there isn't a predefined machine target for riscv32 in bitbake/default-registry/configurations/poky-whinlatter.conf.json, we avoid selecting machine and will address it later. Importantly, we set a SSTATE_DIR which will be used for the shared state cache, avoiding rebuilding packages when not necessary (I'm not totaly sure when this isn't exposed in bitbake-setup settings like dl-dir is).

./bitbake/bin/bitbake-setup init --non-interactive \
  --skip-selection machine \
  ./bitbake/default-registry/configurations/poky-whinlatter.conf.json \
  poky \
  distro/poky

printf 'SSTATE_DIR = "%s"\n' "$HOME/.cache/yocto/sstate" >> bitbake-builds/site.conf

With that done, we can source the generated definitions to enter the build environment (note we're using the default setup directory, you can override it to something other than poky-whinlatter by using --setup-dir-name) and use enable-fragment to set the qemuriscv32 machine:

. bitbake-builds/poky-whinlatter/build/init-build-env
bitbake-config-build enable-fragment machine/qemuriscv32

Now configure the build, indicating the additional libraries that need to be present and run bitbake to actually produce it:

cat >> conf/local.conf <<'EOF'
IMAGE_INSTALL:append = " \
  glibc-dev \
  libgcc \
  libgcc-dev \
  libatomic \
  libatomic-dev \
  libstdc++ \
  libstdc++-dev \
"
EOF

bitbake core-image-minimal

This results in 4482 build tasks and takes quite some time to complete if you haven't run it before (i.e. aren't hitting in the sstate cache). The next section of this article explores how to produce the needed output while building much less, but let's finish the job and extract a rootfs from what was built. I would like to now follow advice in the documentation and do runqemu-extract-sdk tmp/deploy/images/qemuriscv32/core-image-minimal-qemuriscv32.rootfs.tar.zst ~/rv32sysroot, except that fails because the runqemu-extract-sdk script doesn't recognise .tar.zst (I've submitted a patch). So instead we manually extract the .tar from the .tar.zst and then run the runqemu-extract-sdk script:

zstd -d -k -f tmp/deploy/images/qemuriscv32/core-image-minimal-qemuriscv32.rootfs.tar.zst
runqemu-extract-sdk tmp/deploy/images/qemuriscv32/core-image-minimal-qemuriscv32.rootfs.tar ~/rv32sysroot

At this point, you have a sysroot that's almost directly usable for cross-compiling Clang/LLVM (with --target=riscv32-poky-linux) but there are three finalisation steps we will perform:

Add an additional symlink to the tree so that upstream Clang's search procedure for the GCC install finds the correct directory. The combination of these two downstream patches which Yocto applies to its own Clang builds would make this unnecessary. I'm not sure if upstreaming has ever been pursued.
Convert all absolute symlinks to relative ones. Yocto provides a Python script for this, which is in our $PATH after sourcing build/init-build-env.
(Optional) Apply workaround for a ninja issue that would otherwise mean incremental builds don't work.

mkdir -p "$HOME/rv32sysroot/usr/lib/gcc" && ln -s ../riscv32-poky-linux "$HOME/rv32sysroot/usr/lib/gcc/riscv32-poky-linux"
sysroot-relativelinks.py $HOME/rv32sysroot
ln -s usr/include $HOME/rv32sysroot/include

Producing a sysroot with fewer build steps

The core-image-minimal recipe above is straightforward, but does a lot more work than strictly necessary. We can reduce this by instead adding a dependency-only recipe that explicitly lists the needed build-time dependencies and contains logic to produce the sysroot.

First, create a layer:

. bitbake-builds/poky-whinlatter/build/init-build-env
bitbake-layers create-layer --add-layer ../layers/meta-rv32-llvm-sysroot

Then add the recipe:

recipe_dir="../layers/meta-rv32-llvm-sysroot/recipes-devtools/rv32-llvm-deps-sysroot"
mkdir -p "$recipe_dir"

cat > "$recipe_dir/rv32-llvm-deps-sysroot.bb" <<'EOF'
SUMMARY = "Dependency-only recipe to export an RV32 sysroot"
LICENSE = "MIT-0"

INHIBIT_DEFAULT_DEPS = "1"
EXCLUDE_FROM_WORLD = "1"
PACKAGE_ARCH = "${MACHINE_ARCH}"

DEPENDS = "virtual/libc libgcc virtual/${MLPREFIX}compilerlibs zlib"

inherit deploy nopackages

do_configure[noexec] = "1"
do_compile[noexec] = "1"
do_install[noexec] = "1"
do_populate_sysroot[noexec] = "1"

do_deploy() {
  export_dir="${DEPLOYDIR}/${PN}-${MACHINE}"
  rm -rf "$export_dir"
  mkdir -p "$export_dir"
  cp -a "${RECIPE_SYSROOT}/." "$export_dir/"

  sysroot-relativelinks.py "$export_dir"

  mkdir -p "$export_dir/usr/lib/gcc"
  ln -s ../riscv32-poky-linux "$export_dir/usr/lib/gcc/riscv32-poky-linux"
  ln -s usr/include "$export_dir/include"
}
addtask deploy after do_prepare_recipe_sysroot before do_build
EOF

The do_deploy function implements the sysroot preparation logic that largely mirrors the previous section. Otherwise, DEPENDS specifies the needed dependencies (of these, virtual/${MLPREFIX}compilerlibs is a bit magic - this resolves to the compiler runtime provider which pulls in things like libstdc++).

Build the sysroot with:

bitbake rv32-llvm-deps-sysroot

This performs ~850 build tasks and will produce the sysroot at tmp/deploy/images/qemuriscv32/rv32-llvm-deps-sysroot-qemuriscv32/.

The sysroot is slightly larger than the one in the section above because it contains large unstripped static archives like usr/lib/libstdc++.a.

Producing a featureful image bootable in QEMU

Watch this space!

Article changelog

2026-05-11: Initial publication date.

Bootable QEMU image menagerie with rootless debootstrap

2026-05-04T12:00:00Z

Quite some time ago I shared a script and methodology for performing a cross-architecture debootstrap in a rootless way. I had a short note on producing an image bootable in QEMU, but it was fairly minimal. This page provides a cookbook / quick reference on producing such images across various Debian target architectures supported by QEMU. The goal is that the starting point here "gets the basics right" for local experimentation, but of course you are encouraged to evolve the recipe for your needs.

The basic process is to:

Build a root filesystem with rootless-debootstrap-wrapper.
Configure just enough networking, DNS, serial login, and SSH.
Create a 30 GiB ext4 filesystem image directly with mkfs.ext4.
Boot it with qemu-system-*, passing the Debian kernel and initrd directly.

We use Debian trixie for amd64, arm64, armhf, ppc64el, riscv64, and s390x. We use sid for ppc64 big endian and loong64. I ran all of this on a current Arch Linux install.

Common setup

sudo pacman -S debootstrap fakeroot qemu-user-static qemu-user-static-binfmt \
  qemu-emulators-full e2fsprogs socat debian-archive-keyring debian-ports-archive-keyring

Put rootless-debootstrap-wrapper somewhere in your PATH, then create a working directory:

mkdir -p qemu-debian-images
cd qemu-debian-images
mkdir -p "$HOME/debcache"

Paste the following into your terminal, which will be called to do the common guest-side configuration. The main thing that's slightly non-standard in this setup are the systemd drop-in overrides which allow authorised SSH keys to be specified by teh systemd credential mechanism. If that's not something you're interested in doing, you can skip the parts touch /etc/systemd/system/ssh* altogether.

configure_qemu_rootfs() {
  rootfs=$1
  console=$2
  suite=$3
  hostname=$4

  "$rootfs/_enter" sh <<EOF
mkdir -p /etc/systemd/network /etc/ssh/sshd_config.d

cat > /etc/systemd/network/10-qemu.network <<'INNER'
[Match]
Type=ether

[Network]
DHCP=yes
INNER

cat > /etc/ssh/sshd_config.d/20-qemu-login.conf <<'INNER'
PermitRootLogin yes
PasswordAuthentication yes
INNER
rm -f /etc/ssh/ssh_host_*_key /etc/ssh/ssh_host_*_key.pub

cat > /etc/systemd/system/ssh.service.d/10-ephemeral-authorized-keys.conf <<'INNER'
[Service]
ImportCredential=ssh.ephemeral-authorized_keys-all
ExecStart=
ExecStart=/usr/sbin/sshd -D \$SSHD_OPTS -o "AuthorizedKeysFile .ssh/authorized_keys" -o "AuthorizedKeysCommand /usr/bin/cat \${CREDENTIALS_DIRECTORY}/ssh.ephemeral-authorized_keys-all" -o "AuthorizedKeysCommandUser root"
INNER

cat > /etc/systemd/system/sshd-vsock@.service.d/10-ephemeral-authorized-keys.conf <<'INNER'
[Service]
ImportCredential=ssh.ephemeral-authorized_keys-all
ExecStart=
ExecStart=-/usr/sbin/sshd -i \$SSHD_OPTS -o "AuthorizedKeysFile .ssh/authorized_keys" -o "AuthorizedKeysCommand /usr/bin/cat \${CREDENTIALS_DIRECTORY}/ssh.ephemeral-authorized_keys-all" -o "AuthorizedKeysCommandUser root"
INNER

/usr/bin/systemd-firstboot --locale=C.UTF-8 --hostname=${hostname} --force
ln -sf ../locale.conf /etc/default/locale
printf '127.0.1.1 %s\n' "$hostname" >> /etc/hosts
printf 'uninitialized\n' > /etc/machine-id
mkdir -p /var/lib/dbus
rm -f /var/lib/dbus/machine-id
ln -sf /etc/machine-id /var/lib/dbus/machine-id

systemctl enable systemd-networkd systemd-resolved systemd-timesyncd ssh
systemctl enable serial-getty@${console}.service

ln -sf ../run/systemd/resolve/resolv.conf /etc/resolv.conf
printf 'root:root\n' | chpasswd
adduser --gecos ",,," --disabled-password user
usermod -aG sudo user
printf 'user:user\n' | chpasswd
EOF

  if [ "$suite" = trixie ]; then
    cat >> "$rootfs/etc/apt/sources.list" <<'EOF'
deb https://security.debian.org/debian-security trixie-security main
deb https://deb.debian.org/debian trixie-updates main
EOF
  fi
}

This should not be exposed on any public network without further configuration. You can ssh in to either the root user or user via ssh, using password root or user respectively. The commands below expose ssh via a unix domain socket. One potential gotcha: this unix domain socket must not have any - in its name as that collides with the splitting done for the hostfwd argument. The examples given below avoid this issue. The boot commands pass net.ifnames=0, so the single QEMU network device is consistently named eth0 and matched by the networkd config above (I found this more reliable than ln -sf /dev/null /etc/udev/rules.d/80-net-setup-link.rules).

For simplicity we make use of mkfs.ext4's ability to populate the image from a directory. Pleasingly, mkfs.xfs gained a similar ability in the xfsprogs 6.17.0 release in Oct 2025. If you have a new enough version, and you prefer an XFS rootfs over ext4 you can tweak the recipes below to do the following for the final image population step:

fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.xfs -f -q -L rootfs \
    -d file,name="$WORK/rootfs.img",size=30g \
    -p "$ROOTFS",atime=0

amd64 / x86-64

Build:

WORK=$PWD/amd64-trixie-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=amd64 \
  --suite=trixie \
  --mirror=https://deb.debian.org/debian \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --include=linux-image-amd64,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" ttyS0 trixie qemu-amd64-trixie
cp "$ROOTFS"/boot/vmlinuz-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd amd64-trixie-qemu
qemu-system-x86_64 \
  -accel kvm \
  -machine q35 \
  -cpu host \
  -smp 2 \
  -m 8G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-pci,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_amd64.sock-:22 \
  -device virtio-net-pci,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-pci,rng=rng \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=ttyS0 net.ifnames=0"

The above assumes you are running on a x86-64 host, hence enables KVM. If not, then drop -accel kvm and use -cpu max instead of -cpu host.

arm64 / AArch64

Build:

WORK=$PWD/arm64-trixie-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=arm64 \
  --suite=trixie \
  --mirror=https://deb.debian.org/debian \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --include=linux-image-arm64,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" ttyAMA0 trixie qemu-arm64-trixie
cp "$ROOTFS"/boot/vmlinuz-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd arm64-trixie-qemu
qemu-system-aarch64 \
  -machine virt \
  -cpu cortex-a57 \
  -smp 2 \
  -m 8G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-device,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_arm64.sock-:22 \
  -device virtio-net-device,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-device,rng=rng \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=ttyAMA0 net.ifnames=0"

armhf / 32-bit ARM

For this one I had to add the relevant virtio modules to the initrd.

Build:

WORK=$PWD/armhf-trixie-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=armhf \
  --suite=trixie \
  --mirror=https://deb.debian.org/debian \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --include=linux-image-armmp,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" ttyAMA0 trixie qemu-armhf-trixie
printf '%s\n' virtio_mmio virtio_blk virtio_net >> "$ROOTFS/etc/initramfs-tools/modules"
"$ROOTFS/_enter" update-initramfs -u -k all
cp "$ROOTFS"/boot/vmlinuz-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd armhf-trixie-qemu
qemu-system-arm \
  -machine virt \
  -cpu cortex-a15 \
  -smp 2 \
  -m 4G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-device,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_armhf.sock-:22 \
  -device virtio-net-device,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-device,rng=rng \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=ttyAMA0 net.ifnames=0"

riscv64

Build:

WORK=$PWD/riscv64-trixie-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=riscv64 \
  --suite=trixie \
  --mirror=https://deb.debian.org/debian \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --include=linux-image-riscv64,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" ttyS0 trixie qemu-riscv64-trixie
cp "$ROOTFS"/boot/vmlinux-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd riscv64-trixie-qemu
qemu-system-riscv64 \
  -machine virt \
  -cpu rv64 \
  -smp 2 \
  -m 8G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-device,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_riscv64.sock-:22 \
  -device virtio-net-device,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-device,rng=rng \
  -bios /usr/share/qemu/opensbi-riscv64-generic-fw_dynamic.bin \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=ttyS0 net.ifnames=0"

The above assumes you have opensbi installed in /usr/share/qemu (it is put here by the qemu-system-riscv-firmware package on Arch).

ppc64el

Build:

WORK=$PWD/ppc64el-trixie-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=ppc64el \
  --suite=trixie \
  --mirror=https://deb.debian.org/debian \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --include=linux-image-powerpc64le,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" hvc0 trixie qemu-ppc64el-trixie
cp "$ROOTFS"/boot/vmlinux-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd ppc64el-trixie-qemu
qemu-system-ppc64 \
  -machine pseries \
  -cpu power9 \
  -smp 2 \
  -m 8G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-pci,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_ppc64el.sock-:22 \
  -device virtio-net-pci,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-pci,rng=rng \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=hvc0 net.ifnames=0"

s390x (SystemZ)

Build:

WORK=$PWD/s390x-trixie-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=s390x \
  --suite=trixie \
  --mirror=https://deb.debian.org/debian \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --include=linux-image-s390x,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" ttysclp0 trixie qemu-s390x-trixie
cp "$ROOTFS"/boot/vmlinuz-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd s390x-trixie-qemu
qemu-system-s390x \
  -machine s390-ccw-virtio \
  -smp 2 \
  -m 8G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-ccw,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_s390x.sock-:22 \
  -device virtio-net-ccw,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-ccw,rng=rng \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=ttysclp0 net.ifnames=0"

ppc64 big-endian

This is a Debian ports target, so we use sid and the ports mirror.

Build:

WORK=$PWD/ppc64-sid-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=ppc64 \
  --suite=sid \
  --mirror=https://deb.debian.org/debian-ports \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --keyring=/usr/share/keyrings/debian-ports-archive-keyring.gpg \
  --include=linux-image-powerpc64,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" hvc0 sid qemu-ppc64-sid
cp "$ROOTFS"/boot/vmlinux-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd ppc64-sid-qemu
qemu-system-ppc64 \
  -machine pseries \
  -cpu power9 \
  -smp 2 \
  -m 8G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-pci,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_ppc64.sock-:22 \
  -device virtio-net-pci,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-pci,rng=rng \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=hvc0 net.ifnames=0"

loong64 / LoongArch

For this one, we need EDK2 which you can obtain from Debian's qemu-efi-loongarch64 package (QEMU_EFI.fd).

Build:

WORK=$PWD/loong64-sid-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=loong64 \
  --suite=sid \
  --mirror=https://deb.debian.org/debian \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --include=linux-image-loong64,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" ttyS0 sid qemu-loong64-sid
cp "$ROOTFS"/boot/vmlinuz-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd loong64-sid-qemu
cp ../QEMU_EFI.fd .
qemu-system-loongarch64 \
  -machine virt,firmware=QEMU_EFI.fd \
  -smp 2 \
  -m 8G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-pci,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_loong64.sock-:22 \
  -device virtio-net-pci,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-pci,rng=rng \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=ttyS0 net.ifnames=0"

Logging in

As noted above, you can log in with root/root or user/user. The launch commands above run QEMU with -nographic causing your terminal to be connected to the guest serial console. Ctrl-c alone won't kill the virtual machine, so it's helpful to know:

Ctrl-a x exits QEMU immediately.
Ctrl-a c switches between the guest serial console and the QEMU monitor. From the monitor, quit exits QEMU and system_powerdown asks the guest to shut down cleanly.
Ctrl-a h prints QEMU's help for the other Ctrl-a shortcuts.

Once the guest is booted, you can connect via ssh to the Unix domain socket that forwards to guest port 22. Assuming you're on a recent system with systemd-ssh-proxy (and the ssh config file it adds) present, this can be done with e.g.:

ssh root@unix/tmp/qemu_amd64.sock

Without systemd-ssh-proxy, you can specify ProxyCommand instead:

# For socat:
ssh -o "ProxyCommand=socat - UNIX-CONNECT:/tmp/qemu_amd64.sock" root@vm
# Or for OpenBSD netcat:
ssh -o "ProxyCommand=nc -U /tmp/qemu_amd64.sock" root@vm

If you'd rather use a TCP port, replace the -netdev part of the qemu launch command with something like the following and connect to localhost:2222:

-netdev user,id=net,hostfwd=tcp:127.0.0.1:2222-:22

The systemd-provided config for use of systemd-ssh-proxy disables host identity checks, which is what you typically want with this setup. If using one of the ProxyCommand options above you may want to add -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null to your `ssh invocation.

Alternative: SSH over vsock

It's possible to avoid QEMU user-mode networking and use ssh via AF_VSOCK. This can even work without any additional image changes as systemd-ssh-generator in the guest will generate an appropriate socket-activated sshd service if vsock is present. On the host, you'll need to pick a numeric address for the vsock ('guest CID') that isn't already in use on the system, and change the qemu command line to add the appropriate vsock device with that CID assigned. The vsock device used depends on the machine being emulated - e.g. whether to attach on PCI or the virtio device bus.

For amd64, ppc64el, ppc64, and loong64, add:

-device vhost-vsock-pci,guest-cid=42

For arm64, armhf, and riscv64, add:

-device vhost-vsock-device,guest-cid=42

For s390x, use:

-device vhost-vsock-ccw,guest-cid=42

Assuming your host has systemd-ssh-proxy and its OpenSSH config installed, you can connect with:

ssh root@vsock/42

Using an injected SSH key

Images set up using the recipes above allow a public key to be specified at boot time using the systemd system credential mechanism. Just append the following to the qemu launch command and you can ssh in using that key:

-smbios "type=11,value=io.systemd.credential.binary:ssh.ephemeral-authorized_keys-all=$(base64 -w0 ~/.ssh/id_ed25519.pub)"

Article changelog

2026-05-11:
- Add notes on ssh over AF_VSOCK.
- Add note about ssh host key checking.
- Add support for injecting ssh keys using systemd's credential mechanism.
2026-05-10:
- Add note about serial console shortcuts.
- Use systemd-ssh-proxy.
2026-05-09:
- Tweak shell commands slightly (no cp -f) and use Type=ether for systemd-networkd match.
2026-05-07:
- Add note about how to use an XFS rootfs.
- Get rid of vestigial errexit usage in common setup script.
2026-05-05: Use net.ifnames=0 command line argument rather than ln -sf /dev/null /etc/udev/rules.d/80-net-setup-link.rules.
2026-05-04: Initial publication date.

Minipost: Routing a Linux user's traffic through a WireGuard interface

2026-03-31T12:00:00Z

Simple goal: take advantage of my home router's WireGuard support and have one of my external servers connect using this, and pass all traffic from a certain user through that interface.

Create WireGuard credentials

This part of the note won't be that useful to you unless you're using a Fritzbox router. But if you're me or someone suspiciously like me you may want to know to:

Navigate to https://192.168.178.1/#/access/wireguard
Click "Add WireGuard connection" and ensure "Connect a single device" is selected on the modal that appears. Then click "Next".
Enter a unique name for the connection (I typically use $remote_host_name-wg) and click Finish. Follow request to confirm by pressing a button on the router.
Click "Download settings" and a wg_config.conf will be downloaded.

Add user

VPN_USER=asbvpn
sudo useradd -m -g users -G wheel -s /bin/bash "$VPN_USER"
sudo passwd "$VPN_USER"

Configure systemd-networkd

First, extracting the relevant values from the wg_config.conf:

WG_CONF=wg_config.conf
PRIVATE_KEY="$(sed -n 's/^PrivateKey = //p' "$WG_CONF")"
PUBLIC_KEY="$(sed -n 's/^PublicKey = //p' "$WG_CONF")"
PRESHARED_KEY="$(sed -n 's/^PresharedKey = //p' "$WG_CONF")"
ENDPOINT="$(sed -n 's/^Endpoint = //p' "$WG_CONF")"

ADDRS="$(sed -n 's/^Address = //p' "$WG_CONF")"
IPV4_ADDR="$(printf '%s\n' "$ADDRS" | cut -d, -f1)"
IPV6_ADDR="$(printf '%s\n' "$ADDRS" | cut -d, -f2)"

What we want to do is:

Define the wireguard interface wg0 and specify the necessary keys, IP addresses etc for it to be brought up successfully.
Specify a routing policy so that all traffic from the given user account goes via that interface.
- As you can see below, we specify a RouteTable called "vpn", associate that with the interface, and specify rules for that table.
- Ideally this would "fail closed" and no traffic from the user would be routed if wg0 is down. That appears to use additional rules managed outside of systemd-networkd. I haven't tried to implement this.

The above can be achieved with:

sudo mkdir -p /etc/systemd/networkd.conf.d
sudo tee /etc/systemd/networkd.conf.d/90-vpn-table.conf >/dev/null <<'EOF'
[Network]
RouteTable=vpn:100
EOF

sudo tee /etc/systemd/network/50-wg0.netdev >/dev/null <<EOF
[NetDev]
Name=wg0
Kind=wireguard

[WireGuard]
PrivateKey=$PRIVATE_KEY
RouteTable=vpn

[WireGuardPeer]
PublicKey=$PUBLIC_KEY
PresharedKey=$PRESHARED_KEY
AllowedIPs=0.0.0.0/0
AllowedIPs=::/0
Endpoint=$ENDPOINT
EOF

sudo tee /etc/systemd/network/50-wg0.network >/dev/null <<EOF
[Match]
Name=wg0

[Network]
Address=$IPV4_ADDR
Address=$IPV6_ADDR

[RoutingPolicyRule]
User=$VPN_USER
Table=vpn
Priority=10000
Family=both
EOF

sudo chgrp systemd-network /etc/systemd/network/50-wg0.netdev
sudo chmod 0640 /etc/systemd/network/50-wg0.netdev
sudo systemctl restart systemd-networkd

Doing it this way, we've stored the secret keys in the 50-wg0.netdev file itself but restricted access to the file. It's possible to have the keys stored in a separate file, but for my setup it didn't seem worthwhile.

Then check the status with e.g.:

sudo networkctl status wg0
sudo ip rule show
sudo ip route show table 100
sudo wg show wg0
sudo -u $VPN_USER curl https://ifconfig.me/all

IPv6 does not work in this setup (curl -6 google.com will fail),

Copying authorized_keys to new user

This is more a note to myself than anything else:

sudo install -d -m 700 -o $VPN_USER -g users /home/$VPN_USER/.ssh
sudo install -m 600 -o $VPN_USER -g users "$HOME/.ssh/authorized_keys" /home/$VPN_USER/.ssh/authorized_keys

Article changelog

2026-03-31: Initial publication date.

Minipost: Additional figures for per-query energy consumption of LLMs

2026-02-17T12:00:00Z

Last month I wrote up a fairly long piece on per-query energy consumption of LLMs using the data from InferenceMAX (note: InferenceMAX has since been renamed to InferenceX). Much of the write-up was dedicated to exploring what you can actually conclude from these figures and how that interacts with some of the implementation decisions in the benchmark, but I feel the results still give a useful yardstick. Beyond concerns about overly-specialised serving engine configurations and whether the workload is representative of real-world model serving in a paid API host, the other obvious limitation is that InferenceMAX is only testing GPT-OSS 120b and DeepSeek R1 0528 when there is a world of other models out there. I dutifully added "run my own tests using other models" to the todo list and here we are. By "here we are" I of course mean I made no progress towards that goal but Zach Mueller at Lambda started publishing model cards with the needed data - thanks Zach!

The setup for Lambda is simple - each model card lists the observed token generation throughput and total throughput (along with other stats) for an input sequence length / output sequence length (ISL/OSL) of 8192/1024, as benchmarked using vllm bench serve. The command used to serve the LLM (using sglang or vllm depending on the model) is also given. As a starting point this is no worse than the InferenceMAX data, and potentially somewhat better due to figures being taken from a configuration that's not overly specialised to a particular query length.

The figures each Lambda model card gives us that are relevant for calculating the energy per query are: the hardware used, token generation throughput and total token throughput (input+output tokens). Other statistics such as the time to first token, inter-token latency, and parallel requests tested help confirm whether this is a configuration someone would realistically use. Using an equivalent methodology to before, we get the Watt hours per query by:

Determining the total Watts for the GPU cluster. We take the figures used by SemiAnalysis (2.17kW for a single B200) and multiply by the number of GPUs.
Calculate the joules per token by dividing this total Watts figure by the total token throughput. This gives a weighted average of the joules per token for the measured workload, reflecting the ratio of isl:osl.
Multiply this weighted average of joules per token by the tokens per query (isl+osl) to get the joules per query. Then divide by 3600 to get Wh.

Collecting the data from the individual model cards we can generate the following (as before, using minutes of PlayStation 5 gameplay as a point of comparison):

data = {
    "Qwen/Qwen3.5-397B-A17B": {
        "num_b200": 8,
        "total_throughput": 11092,
    },
    "MiniMaxAI/MiniMax-M2.5": {
        "num_b200": 2,
        "total_throughput": 8062,
    },
    "zai-org/GLM-5-FP8": {
        "num_b200": 8,
        "total_throughput": 6300,
    },
    "zai-org/GLM-4.7-Flash": {
        "num_b200": 1,
        "total_throughput": 8125,
    },
    "arcee-ai/Trinity-Large-Preview": {
        "num_b200": 8,
        "total_throughput": 15611,
    },
}

# 8192 + 1024
TOKENS_PER_QUERY = 9216

# Taken from <https://inferencex.semianalysis.com/>
B200_KW = 2.17

# Reference power draw for PS5 playing a game. Taken from
# <https://www.playstation.com/en-gb/legal/ecodesign/> ("Active Power
# Consumption"). Ranges from ~217W to ~197W depending on model.
PS5_KW = 0.2


def wh_per_query(num_b200, total_throughput, tokens_per_query):
    total_cluster_kw = num_b200 * B200_KW
    total_cluster_watts = total_cluster_kw * 1000
    # joules_per_token is a weighted average for the measured mix of input
    # and output tokens.
    joules_per_token = total_cluster_watts / total_throughput
    joules_per_query = joules_per_token * tokens_per_query
    # Convert joules to watt-hours
    return joules_per_query / 3600.0

def ps5_minutes(wh):
    ps5_watts = PS5_KW * 1000
    return (wh / ps5_watts) * 60.0

MODEL_WIDTH = 31
WH_WIDTH = 8
PS5_WIDTH = 8

header = f"{'Model':<{MODEL_WIDTH}} | {'Wh/q':<{WH_WIDTH}} | {'PS5 min':<{PS5_WIDTH}}"
separator = f"{'-' * MODEL_WIDTH} | {'-' * WH_WIDTH} | {'-' * PS5_WIDTH}"

print(header)
print(separator)

for model, vals in data.items():
    wh = wh_per_query(vals["num_b200"], vals["total_throughput"], TOKENS_PER_QUERY)
    ps5_min = ps5_minutes(wh)

    wh_str = f"{wh:.2f}" if wh < 10 else f"{wh:.1f}"
    print(f"{model.strip():<{MODEL_WIDTH}} | {wh_str:<{WH_WIDTH}} | {ps5_min:.2f}")

This gives the following figures (reordered to show Wh per query in ascending order, and added a column for interactivity (1/TPOT)):

Model	Intvty (tok/s)	Wh/q	PS5 min.
zai-org/GLM-4.7-Flash (bf16)	34.0	0.68	0.21
MiniMaxAI/MiniMax-M2.5 (fp8)	30.3	1.38	0.41
arcee-ai/Trinity-Large-Preview (bf16)	58.8	2.85	0.85
Qwen/Qwen3.5-397B-A17B (bf16)	41.7	4.01	1.20
zai-org/GLM-5-FP8 (fp8)	23.3	7.05	2.12

As a point of comparison, the most efficient 8 GPU deployment of fp8 DeepSeek R1 0528 from my figures in the previous article was 3.32 Wh per query.

And that's all I really have for today. Some interesting datapoints with hopefully more to come as Lambda puts up more model cards in this format. There's a range of interesting potential further experiments to do, but for now, I just wanted to share this initial look.

Article changelog

2026-02-17: Initial publication date.

shandbox

2026-02-11T12:00:00Z

shandbox is a simple Linux sandboxing script that serves my needs well. Perhaps it works for you too? No dependencies between a shell and util-linux (unshare and nsenter).

In short, it aims to provide fairly good isolation for personal files (i.e. your $HOME) while being very convenient for day to day use. It's designed to be run as an unprivileged user - as long as you can make new namespaces you should be good to go. By default /home/youruser/sandbox shows up as /home/sandbox within the sandbox, and other than standard paths like /usr, /etc, /tmp, and so on it's left for you to either copy things into the sandbox or expose them via a mount. There's a single shared sandbox (i.e. processes within the sandbox can see and interact with each other, and the exposed sandbox filesystem is shared as well), which trades off some ease of use for the security you might get with a larger number of more targeted sandboxes. On the other hand, you only gain security from a sandbox if you actually use it and this is a setup that offers very low friction for me. The network is not namespaced (although this is something you could change with a simple edit).

Usability is both subjective and highly dependent on your actual use case, so the tradeoffs may or may not align with what is interesting for you! Bubblewrap is an example of a mature alternative unprivileged sandboxing tool that offers a lot of configurability as well as options with greater degrees of sandboxing. Beyond that, look to Firecracker based solutions or gvisor. shandbox obviously aims to provide a reasonable sandbox as much as Linux namespaces alone are able to offer, but if you're looking for a security property stronger than "makes it harder for something to edit or access unwanted files" it's down to you to both carefully review its implementation and consider alternatives.

Usage example

$ shandbox run uvx pycowsay
Installed 1 package in 5ms

  ------------
< Hello, world >
  ------------
   \   ^__^
    \  (oo)\_______
       (__)\       )\/\
           ||----w |
           ||     ||
$ shandbox status
running (pid 1589364)

log:
  2026-02-11 13:02:51 stopped
  2026-02-11 13:05:06 started (pid 1589289)
$ shandbox add-mount ~/repos/llvm-project /home/sandbox/llvm-project
mounted /home/asb/repos/llvm-project -> /home/sandbox/llvm-project
$ shandbox run touch /home/sandbox/llvm-project/write-attempt
touch: cannot touch '/home/sandbox/llvm-project/write-attempt': Read-only file system
$ shandbox remove-mount /home/sandbox/llvm-project
unmounted /home/sandbox/llvm-project
$ shandbox add-mount --read-write ~/repos/llvm-project /home/sandbox/llvm-project
mounted /home/asb/repos/llvm-project -> /home/sandbox/llvm-project
$ shandbox run touch /home/sandbox/llvm-project/write-attempt

shandbox enter will open a shell within the sandbox for easy interactive usage. As a convenience, if the current working directory is in $HOME/sandbox (e.g. $HOME/sandbox/foo) then the working directory within the sandbox for shandbox run or shandbox enter will be set to the appropriate path within the sandbox (/home/sandbox/foo in this case). i.e., the case where this mapping is trivial. Environment variables are not passed through.

Functionality overview

shandbox start: Start the sandbox, creating the necessary namespaces and mount layout. Fails if the sandbox is already running.
shandbox stop: Stop the sandbox by killing the process holding the namespaces. Fails if the sandbox is not running.
shandbox restart: Stop the sandbox and start it again.
shandbox status: Print whether the sandbox is running and if it is, the pid. Also print the last 20 lines of the log.
shandbox enter: Open bash within the sandbox, starting the sandbox first if it's not already running.
shandbox run <command> [args...]: Run a command inside the sandbox. The current working directory is translated to an in-sandbox path if it falls within the sandbox home directory. Starts the sandbox first if it isn't already running.
shandbox add-mount [--read-write] <host-path> <sandbox-path>: Bind-mount a host path into the running sandbox. Mounts are read-only by default; pass --read-write to allow writes. The sandbox must already be running. Both directories and individual files are supported.
shandbox remove-mount <sandbox-path>: Remove a previously added bind mount from the running sandbox.

Implementation approach

The core sandboxing functionality is provided by the Linux namespaces functionality exposed by unshare and nsenter. The script's implementation should be quite readable but I'll try to summarise some key points here.

The goal is that:

Within the sandbox, you appear as an unprivileged user, with uid and gid equal to your usual Linux user.
It should be possible to expose additional files or directories to the sandbox once it's running.
Applications running within the sandbox have no way (modulo bugs or vulnerabilities in the kernel or accessible applications) of reaching files on the host filesystem that aren't explicitly exposed.
- To underline: This is a goal, it is not a guarantee.
It's possible to launch multiple processes within the sandbox which can all see each other, and have the same shared sandboxed filesystem.
This is all doable as an unprivileged user.

To implement that:

Two sets of namespaces are used to provide this isolation: the outer 'shandbox_root' has the user mapped to root within the namespace and retains access to standard / (allowing us to mount additional paths into after the sandbox has started). The inner 'shandbox_user' represents a new user namepsace mapping our uid/gid to an unprivileged user, but other namespaces are shared with 'shandbox_root'. Sandboxed processes are launched within the namespaces of 'shandbox_user'.
The process IDs of the initial process within 'sandbox_root' and 'sandbox_user' are saved and recalled so the script can use nsenter to enter the namespace.
To help make it easier to tell when you're in the sandbox, a dummy /etc/passwd is bind-mounted naming the current user as sandbox.
When shandbox start is executed, the necessary directories are bind mounted in a directory that will be used as root (/) for the user sandbox in .local/share/shandbox/root. This happens within the sandbox_root namespace, which then uses unshare again to create a new user namespace with an unprivileged user, executing within a chroot.
'sandbox_root' retains access to the host filesystem, which is necessary to allow mounting additional paths after the fact. Without this requirement, we could likely rewrite shandbox start to use pivot_root.

Making it your own

The script should be straight-forward enough to customise to your needs if they're not too dissimilar to what is offered out of the box. Some variables at the top provide things you may be more likely to want to change, such as the home directory location, and a list of files or directories in $HOME to always bind-mount into the sandbox home:

SANDBOX_HOME_DIR="$HOME/sandbox"
HOME_FILES_TO_MAP=".bashrc .vimrc"
HOME_DIRS_TO_MAP=".vim bin"
SB_HOME="/home/sandbox"
SB_PATH="$SB_HOME/bin:/usr/local/bin:/usr/bin"

Article changelog

2026-02-11: Initial publication date.

Per-query energy consumption of LLMs

2026-01-07T12:00:00Z

How much energy is consumed when querying an LLM? We're largely in the dark when it comes to proprietary models, but for open weight models that anyone can host on readily available, albeit eye-wateringly expensive, hardware this is something that can be measured and reported, right? In fact, given other people are doing the hard work of setting up and running benchmarks across all kinds of different hardware and software configurations for common open weight models, can we just re-use that to get a reasonable figure in terms of Watt-hours (Wh) per query?

For the kind of model you can run locally on a consumer GPU then of course there's some value in seeing how low the per-query energy usage might be on a large scale commercial setup. But my main interest is in larger and more capable models, the kind that you wouldn't realistically run locally and end up using in a pay-per-token manner either directly with your host of choice or through an intermediary like OpenRouter. In these cases where models are efficiently served with a minimum of 4-8 GPUs or even multi-node clusters it's not easy to get a feel for the resources you're using. I'm pretty happy that simple back of the envelope maths shows that whether providers are properly amortising the cost of their GPUs or not, it's implausible that they're selling per-token API access for open models at below the cost of electricity. That gives a kind of upper bound on energy usage, and looking at the pennies I spend on such services it's clearly a drop in the ocean compared to my overall energy footprint. But it's not a very tight bound, which means it's hard to assess the impact of increasing my usage.

We can look at things like Google's published figures on energy usage for Gemini but this doesn't help much. They don't disclose the length of the median prompt and its response, or details of the model used to serve that median query meaning it's not helpful for either estimating how it might apply to other models or how it might apply to your own usage (which may be far away from this mysterious median query). Mistral released data on the per query environmental impact (assuming for a 400 token query), but the size of the Mistral Large 2 model is not disclosed and they don't calculate a Wh per query figure. CO2 and water per query are very helpful to evaluate a particular deployment, but the actual energy used is a better starting point that can be applied to other providers assuming different levels of carbon intensity. If one of the API providers were to share statistics based on a real world deployment of one of the open models with a much higher degree of transparency (i.e. sharing stats on the number of queries served during the period, statistics on their length, and measured system power draw) that would be a useful source of data. But today we're looking at what we can conclude from the InferenceMAX benchmark suite published results.

I'd started looking at options for getting good figures thinking I might have to invest in the hassle and expense of renting a multi-GPU cloud instance to run my own benchmarks, then felt InferenceMAX may make that unnecessary. After writing this up along with all my provisos I'm perhaps tempted again to try to generate figures myself. Anyway, read on for a more detailed look at that benchmark suite. You can scroll past all the provisos and jump ahead to the figures giving the Wh/query figures implied by the benchmark results across different GPUs, different average input/output sequence lengths, and for gpt-oss 120B and DeepSeek-R1-0528. But I hope you'll feel a bit guilty about it.

If you see any errors, please let me know.

High-level notes on InferenceMAX

InferenceMAX benchmark suite has the stated goal to "provide benchmarks that both emulate real world applications as much as possible and reflect the continuous pace of software innovation." They differentiate themselves from other benchmarking efforts noting "Existing performance benchmarks quickly become obsolete because they are static, and participants often game the benchmarks with unrealistic, highly specific configurations."

The question I'm trying to answer is "what is the most 'useful AI' I can expect for a modern GPU cluster in a realistic deployment and how much energy does it consume". Any benchmark is going to show peak throughput higher than you'd expect to achieve in real workload and there's naturally a desire to keep it pinned on a specific model for as long as it isn't totally irrelevant in order to enable comparisons as hardware and software evolves with a common point of reference. But although I might make slightly different choices about what gets benchmarked and how, the InferenceMAX setup at first look seems broadly aligned with what I want to achieve.

They benchmark DeepSeek-R1-0528 (both at the native fp8 quantisation and at fp4) which is a 671B parameter model with 37B active weights released ~7 months ago and seems a fair representative of a large MoE open weight model. gpt-oss-120b is also benchmarked, providing a point of comparison for a much smaller and efficient to run model. Different input sequence length and output sequence length (ISL and OSL - the number of input and output tokens) are tested: 1k/1k, 1k/8k, 8k/1k, which provides coverage of different query types. Plus tests against a wide range of GPUs (including the 72-GPU GB200 NVL72 cluster) and sweeps different settings.

At the time of writing you might reasonably consider to be 'InferenceMAX' is split into around three pieces:

The frontend website you can see at inferencemax.semianalysis.com (not currently open source but planned to be)
The script for executing queries against the LLM serving infrastructure and collecting stats (currently in a seperate repo but planned to be incorporated into the main InferenceMAX repository),
The wrapper/runner scripts and GitHub actions workflows that live in the main InferenceMAX repository.
- This is actively contributed to by at least Nvidia and AMD engineers.

GitHub Actions is used to orchestrate the runs, ultimately producing a zip file containing JSON with the statistics of each configuration (e.g. here). The benchmark_serving.py script is invoked via the run_benchmark_serving wrapper in benchmark_lib.sh which hardcodes some options and passes through some others from the workflow YAML. The results logged by benchmark_serving.py are processed in InferenceMAX's process_result.py helper which will produce JSON in the desired output format. Together, these scripts provide statistics like throughput (input and output token), end to end latency, interactivity (output tokens per second) etc.

Further studying the benchmark setup

So, let's look at the benchmarking logic in more detail to look for any surprises or things that might affect the accuracy of the Wh-per-query figure I want to generate. I'll note that InferenceMAX is an ongoing project that is actively being developed. These observations are based on a recent repo checkout, but of course things may have changed since then if you're reading this post some time after it was first published.

Looking through I made the following observations. Some represent potential issues (see the next subheading for a list of the upstream issues I filed), while others are just notes based on aspects of the benchmark I wanted to better understand.

One of the required arguments to the benchmark serving script is --random-range-ratio. This is set by default to 0.8 in benchmark-tmpl.yml and in benchmark-multinode-tmpl.yml and is not overridden elsewhere.
- This argument is ultimately used in sample_random_requests. It uses np.random.randint to sample input/output lengths between the range_ratio * {input,output}_len and {input,output}_len.
- Taken together, this logic means for for a workload advertised as having 8k input or output tokens (8192), the benchmark will actually run with an average ~7373 (0.9*num_tokens, due to the length being a random number between 0.8*num_tokens and num_tokens) tokens.
- Because the throughput figures are calculated using the actual input and output token lengths, the figure does represent what was observed, it's just the workload doesn't quite match the description. The reported end to end latency for instance will be misleadingly lower than you would get for a workload that actually did have the expected input / output sequence lengths.
The various request functions in backend_request.func.py will set output.success = False if they don't get a HTTP 200 status code back for a request. There is no logic to retry a refused request and metrics will be calculated skipping any failed requests. This means an overloaded server will perform better on this benchmark for metrics like E2E latency and TTFT if it refuses requests rather than accept them and serve them slowly. As the number of failed requests isn't included in the results json it's not easy to tell if this is a factor for any benchmarks.
Many of the various scripts in the benchmarks/ subdirectory set a max-model-len parameter or the similar --max_seq_len parameter for trt-llm (e.g. the b200 config which if I'm not mistaken will ultimately be set from the max_model_len defined in generate_sweep_configs.py. This parameter is documented in vllm and in TensortRT-LLM and controls the maximum supported length of a request, including both the prompt and any generated output. Setting it 20 or 200 tokens above the sum of the benchmarked ISL+OSL to minimise memory use does not seem like a realistic real-world deployment, which seems the wrong choice given the InferenceMAX complaint that in other suites "participants often game the benchmarks with unrealistic, highly specific configurations". Benchmarks naturally show a 'best case', but if you're generating figures like $ per M tokens it's a figure that makes little sense if it reflects a configuration you wouldn't feasibly use/sell.
Throughput is calculated in benchmark_serving.py based on the total number of tokens divided by the duration of the benchmark. This is then normalised on a per-GPU basis in process_result.py. No problems here, I just wanted to clarify the source of the figure.
In terms of the source of the input tokens themselves, we can see that --dataset-name random is always passed to benchmark_serving.py. This leads to sample_random_requests being called, which will pick random token ids and create a list of tokens of the desired length (mapping these randomly picked IDs to tokens).
- The --ignore-eos flag is passed to the benchmark_serving.py script which will in turn set this option in the JSON when making the LLM request. backend_request_func.py sets this and also sets max_tokens to the desired output_len which should ensure that the response has that exact desired number of output tokens. ignore_eos means that the LLM server will keep generating tokens even after seeing the end of sequence token.
- It's interesting that some of the benchmark configurations enable multi-token prediction, and presumably find it beneficial even given the totally random token inputs. Is it possible that such configurations benefit from undesirable looped outputs (due to a combination of random inputs and continuing to sample tokens past the EOS marker) that potentially are very predictable and give an extra boost?
The --num-prompts parameter controls the total number of requests that are issued. The benchmark script is written so it will wait for all of these to complete (either successfully or unsuccessfully). This is typically set to the concurrency times 10, but some benchmark setups set it higher (presumably as the default figure finishes too quickly for good results).
In terms of how requests are submitted with a certain level of concurrency:
- See above for a discussion of the total number of requests
- --request-rate inf is always passed, so there's no limit on submitting requests up to the concurency limit.
- It precomputes a list of requests to submit and then uses a semaphore to limit concurrency but otherwise continuously submits requests up to the concurrency limit, and then waits until they call complete.
There are no tests that the configuration is serving the model with the expected quality currently, but there's an issue tracking at least adding a simple quality benchmark. Although none of the explored settings should impact the quality of output, it's always possible they trigger a bug and in this case it's not interesting to benchmark.
It would be helpful for reproducibility if more complete system information for the benchmark runners was released. This is being worked on.
You should of course consider whether the tested input and output sequence lengths correspond to a workload you are interested in (thank you to Aaron Zhao for reminding me to mention this. This benchmarking approach also doesn't consider caching. Both factors could be highly relevant if trying to estimate energy cost for a long context chat or 'agentic' flow. But I'm happy enough with the tested workloads as a starting point, and my main focus here is trying to get a degree of comfort with the reported numbers for the ISL/OSL combinations they've chosen to test.

Filed issues

I ended up filing the following issues upstream:

FIXED Token throughput per MW is described as reflecting the generated tokens but is actually processed+generated tokens
- The companion article introducing InferenceMAX has previously defined throughput as the rate at which the GPU generates tokens yet the figure displayed in the UI was the total number of output and input tokens per second. The definition in the article has now been fixed, and changes to the UI make it more obvious based on context that throughput refers to input+output tokens (as y-axis metric options now exist to show "input token throughput per GPU" and "output token throughput per GPU").
- This talking head video from Nvidia seems to make the same error, talking about the number of tokens 'generated' per second per GPU when looking at the relevant results these sem to be the total throughput (i.e. output plus the much faster to process input tokens).
Presented input/output token throughput per GPU for disaggregated setups not usefully comparable to standard multi-gpu setups
- In disaggregated setups you have some number of GPUs dedicated to prefill (processing input tokens) and some number dedicated to decode (generating output tokens). In this case, the reported input/output throughput figures refer to the input or output throughput per prefill GPU or per decode GPU. It doesn't make sense (IMHO) to plot this figure against the input/output throughput figures for a non-disaggregated setup. To make it comparable, the input/output throughput per GPU should be calculated by averaging across the whole cluster rather than just the GPUs dedicated to prefill or decode respectively.
Standard deviatation of interactivity (std_intvty) in result json is incorrectly calculated
- Not a big issue as the figure isn't used anywhere. Interactivity (tokens/second) metrics are calculated from the recorded time per output token. 1000/$tpot_metric is correct for the mean, median, and p99 figures but mathematically incorrect for the standard deviation. e.g. a small standard deviation for time per output token will result in a huge standard deviation being computed for interactivity.
FIXED Reference kW figures no longer shown in frontend for each GPU
- At some point updates to the frontend logic meant that the per-GPU kW figures used in calculating the token throughput per utility MW were no longer displayed. This has now been fixed.
How will full workflow run output be retained beyond 90 days
- The benchmark frontend helpfully links to the GitHub Actions run that generated the displayed results and has a datepicker to view previous results. Clicking through to GitHub means you can download the original .zip of the JSON format benchmark results which is something I take advantage of in the analysis later in this article. According to GitHub docs, the maximum retention period for Actions artifacts and logs is 90 days for a public repo. It would be good to have a mechanism so that this information is backed up rather than lost.
Contents of CONFIG_DIR path as used in launch_gb200-nv.sh is undisclosed
- Most benchmark configuration lives in the main repository, but unfortunately one of the Nvidia DeepSeek R1 configurations relies on a config dir that's not publicly available meaning it can't be audited or reproduced. This is a case where tightening up benchmark rules and review process can hopefully avoid it happening in the future.
Reconsider allowing setting max_model_len / max_seq_len to isl+osl+tiny_margin
- As explained above, a number of benchmarks set max_model_len (or for Nvidia's TensorRT, --max_seq_len) to some figure that is just above ISL+OSL. Although some degree of tuning is expected, to me this goes against the idea that "We want server configs to reflect real world deployments as much as possible" and the stated goal "to provide benchmarks that both emulate real world applications as much as possible and reflect the continuous pace of software innovation". It's hard to imagine a realistic deployment that would configure their serving engine in a way such that it errors if input+output tokens passes ~2k tokens for instance. Looking at the DeepSeek R1 0528 providers on OpenRouter, the vast majority offer greater than 128k context.
- By my understanding, with PagedAttention the KV cache is dynamically allocated anyway so this setting would largely impact other data structures. Plus vllm at least contains a startup check that there is sufficient VRAM to serve at least one request at the maximum configured context. I would really like to see what impact this setting has on benchmarks.
- The repository maintainers renamed my issue to a title that doesn't reflect my report. I'm hopeful they will review my recent comment and title it back.
Some reported metrics will be inflated if a serving engine sheds load
- This covers the observation made above that failed requests are simply skipped. As the number of failed requests isn't tracked, it's not easy to see if a particular configuration may appear better (better E2E latency, lower time to first token) as a reset of shedding load rather than queueing.
- The repository maintainers renamed this issue to "[feature suggestion for vllm/vllm benchmark_serving]" and closed it. I'm hopeful they will read my response and reconsider on the grounds that:
  - The benchmark_serving script isn't doing anything "wrong" necessarily. It is simply making an implementation choice with potential impact on results that the InferenceMAX harness isn't tracking.
  - The script is planned to be added to the repo soon anyway.
Benchmarked ISL and OSL averages 0.9*target_length meaning results are over-optimistic.
- This is the problem mentioned above where the introduced variance in input/output sequence length has an average lower than the headline rate. As noted, this means specifically the end to end latency figure is misleading, but also impacts tokens/second and throughput to the extent that the cost of serving a query doesn't scale with O(n).
- This will be fixed by PR 339 which upstreams the benchmark_serving.py script and in that modified branch changes sample_random_requests to sample a range with multiplier between 1 - RANGE_RATIO and 1 + RANGE_RATIO.

In the best case, you'd hope to look at the benchmark results, accept they're probably represent a higher degree of efficiency than you'd likely get on a real workload, that an API provider might achieve 50% of that and double the effective cost per query to give a very rough upper estimate on per-query cost But that only really works if the reported benchmark results roughly match the achievable throughput in a setup configured for commercial serving. Given the tuning to specific isl/osl values, I'm not at all confident thats the case and I don't know how wide the gap is.

Generating results

Firstly I wrote a quick script to check some assumptions about the data and look for anything that seems anomalous. Specifically:

Check that total throughput per GPU matches what you'd expect based on the input token and output token throughput per GPU, even in the disaggregated case. i.e. the total thoughput per GPU averaged over the whole cluster should equal the sum of the input and output throughput per GPU provided those figures are averaged over the whole cluster.
The ratio of input token throughput to output token throughput should be almost equal to the to the ratio of input to output tokens in the benchmark's workload. If not, there is something surprising that needs investigating.

Based on the information available in the generated result JSON and the reported all-in power per GPU (based on SemiAnalysis' model), we can calculate the Watt hours per query. First calculate the joules per token (watts per GPU divided by the total throughput per GPU). This gives a weighted average of the joules per token for the measured workload (i.e. reflecting the ratio of isl:osl). Multiplying joules per token by the tokens per query (isl+osl) gives the joules per query, and we can just divide by 3600 to get Wh.

There is some imprecision because we're constructing the figure for e.g. 8192/1024 ISL based on measurements with an average 0.9*8192 input and 0.9*1024 output length. The whole calculation would be much simpler if the benchmark harness recorded the number of queries executed and in what time, meaning we can directly calculate the Wh/query from the Wh for the system over the benchmark duration divided by the number of queries served (and remembering that in the current setup each query is on average 90% of the advertised sequence length).

This logic is wrapped up in a simple script.

There's been a recent change to remove the 'full sweep' workflows in favour of only triggering a subset of runs when there is a relevant change. But I grabbed my results from before this happened, from a December 15th 2025 run. However when finalising this article I spotted Nvidia managed to land some new NVL72 DeepSeek R1 0528 configurations just before Christmas, so I've merged in those results as well, using a run from December 19th. All data and scripts are collected together in this Gist.

Results

As well as giving the calculated Wh per query, the script also gives a comparison point of minutes of PS5 gameplay (according to Sony, "Active Power Consumption" ranges from ~217W to ~197W depending on model - we'll just use 200W). The idea here is to provide some kind of reference point for what a given Wh figure means in real-world times, rather than focusing solely on the relative differences between different deployments. Comparisons to "minutes of internet streaming" seem popular at the moment, presumably as it's because an activity basically everyone does. I'm steering away from that because I'd be comparing one value that's hard to estimate accurately and has many provisos to another figure that's hard to estimate accurately and has many provisos, which just injects more error and uncertainty into this effort to better measure/understand/contextualise energy used for LLM inference.

I'm now going to cherry-pick some results for discussion. Firstly for DeepSeek R1 0528 with 8k/1k ISL/OSL, we see that the reported configurations that give a usable level of interactivity at fp8 report between 0.96-3.74 Wh/query (equivalent to 0.29-1.12 minutes of PS5 gaming). The top row which is substantially more efficient is the newer GB200 NVL72 configuration added at the end of last year. It's not totally easy to trace the configuration changes given they're accompanied by a reworking of the associated scripts, but as far as I can see the configuration ultimately used is this file from the dynamo repository. Looking at the JSON the big gain comes from significantly higher prefill throughput (with output throughput per GPU remaining roughly the same). This indicates the older results (the second row) were bottlenecked waiting for waiting for prefill to complete.

Workload	Intvty (tok/s)	E2EL (s)	Details	Wh/Q	PS5 min
fp8 DS R1 0528 8k/1k	39.5	36.5	gb200 dynamo-sglang (72 GPUs disagg, conc: 2048, pfill_dp_attn, dec_dp_attn)	0.96	0.29
fp8 DS R1 0528 8k/1k	31.3	55.2	gb200 dynamo-sglang (72 GPUs disagg, conc: 1024, pfill_dp_attn, dec_dp_attn)	3.13	0.94
fp8 DS R1 0528 8k/1k	20.9	48.8	h200 trt (8 GPUs, conc: 64, dp_attn)	3.32	1.00
fp8 DS R1 0528 8k/1k	19.5	49.6	h200 sglang (8 GPUs, conc: 64)	3.39	1.02
fp8 DS R1 0528 8k/1k	23.9	39.9	b200-trt trt (8 GPUs, conc: 64)	3.39	1.02
fp8 DS R1 0528 8k/1k	22.3	44.5	b200 sglang (8 GPUs, conc: 64)	3.74	1.12

Now taking a look at the results for an fp4 quantisation of the same workload, the result is significantly cheaper to serve with similer or better interactivity and the NVL72 setup Nvidia submitted does have a significant advantage over the 4/8 GPU clusters. This time we see 0.63-1.67 Wh/query (equivalent to 0.19-0.50 minutes of PS5 power draw while gaming). Serving at a lower quantisation impacts the quality of results of course, but the improved efficiency, including on smaler 4 GPU setups helps demonstrate why models like Kimi K2 thinking are distributed as "native int4", with benchmark results reported at this quantisation and quantisation aware training used to maintain quality of result.

Workload	Intvty (tok/s)	E2EL (s)	Details	Wh/Q	PS5 min
fp4 DS R1 0528 8k/1k	41.6	24.6	gb200 dynamo-trt (40 GPUs disagg, conc: 1075, pfill_dp_attn, dec_dp_attn)	0.63	0.19
fp4 DS R1 0528 8k/1k	22.8	43.2	b200-trt trt (4 GPUs, conc: 128, dp_attn)	0.93	0.28
fp4 DS R1 0528 8k/1k	18.7	59.3	b200 sglang (4 GPUs, conc: 128)	1.25	0.38
fp4 DS R1 0528 8k/1k	30.3	39.4	b200 sglang (4 GPUs, conc: 64)	1.67	0.50

Looking now at the 1k/8k workload (i.e. generating significant output) and the cost is 15.0-16.3 Wh/query (equivalent to 4.49-4.89 minutes of PS5 power draw while gaming). As expected this is significantly higher than the 8k/1k workload as prefill (processing input tokens) is much cheaper per token than decode (generating output tokens)

Workload	Intvty (tok/s)	E2EL (s)	Details	Wh/Q	PS5 min
fp8 DS R1 0528 1k/8k	42.5	176.3	b200 sglang (8 GPUs, conc: 64)	15.0	4.49
fp8 DS R1 0528 1k/8k	31.9	232.2	h200 sglang (8 GPUs, conc: 64)	15.9	4.76
fp8 DS R1 0528 1k/8k	31.2	237.9	h200 trt (8 GPUs, conc: 64)	16.3	4.88
fp8 DS R1 0528 1k/8k	39.1	189.5	b200-trt trt (8 GPUs, conc: 64)	16.3	4.89

Again, fp4 has a significant improvement in efficiency:

Workload	Intvty (tok/s)	E2EL (s)	Details	Wh/Q	PS5 min
fp4 DS R1 0528 1k/8k	29.7	251.5	b200-trt trt (4 GPUs, conc: 256, dp_attn)	2.73	0.82
fp4 DS R1 0528 1k/8k	37.7	197.5	b200-trt trt (8 GPUs, conc: 256, dp_attn)	4.31	1.29
fp4 DS R1 0528 1k/8k	34.2	221.2	b200 sglang (4 GPUs, conc: 128)	4.75	1.43
fp4 DS R1 0528 1k/8k	33.1	223.1	b200-trt trt (4 GPUs, conc: 128)	4.79	1.44

As you'd expect for a much smaller model at native fp4 quantisation, GPT-OSS-120B is much cheaper to serve. e.g. for 8k/1k:

Workload	Intvty (tok/s)	E2EL (s)	Details	Wh/Q	PS5 min
fp4 GPT-OSS 120B 8k/1k	45.8	20.8	b200-trt trt (1 GPUs, conc: 128)	0.11	0.03
fp4 GPT-OSS 120B 8k/1k	93.1	10.5	b200-trt trt (2 GPUs, conc: 128, dp_attn)	0.11	0.03
fp4 GPT-OSS 120B 8k/1k	44.3	21.4	b200 vllm (1 GPUs, conc: 128)	0.11	0.03
fp4 GPT-OSS 120B 8k/1k	145.7	6.7	b200-trt trt (2 GPUs, conc: 64, dp_attn)	0.14	0.04
fp4 GPT-OSS 120B 8k/1k	103.8	9.2	b200 vllm (2 GPUs, conc: 64)	0.20	0.06

Or for 1k/8k:

Workload	Intvty (tok/s)	E2EL (s)	Details	Wh/Q	PS5 min
fp4 GPT-OSS 120B 1k/8k	80.5	91.6	b200-trt trt (1 GPUs, conc: 128)	0.49	0.15
fp4 GPT-OSS 120B 1k/8k	72.3	102.0	b200 vllm (1 GPUs, conc: 128)	0.55	0.16
fp4 GPT-OSS 120B 1k/8k	144.9	51.1	b200-trt trt (2 GPUs, conc: 128, dp_attn)	0.55	0.17
fp4 GPT-OSS 120B 1k/8k	129.4	57.0	b200-trt trt (2 GPUs, conc: 128)	0.61	0.18

Conclusion

Well, this took rather a lot more work than I thought it would and I'm not yet fully satisfied with the result. Partly we have to accept a degree of fuzziness about marginal energy usage of an individual query - it's going to depend on the overall workload of the system so there's going to be some approximation when you try to cost a single query.

I'm glad that InferenceMAX exists and am especially glad that it's open and publicly developed, which is what has allowed me to dive into its implementation to the extent I have and flag concerns/issues. I feel it's not yet fully living up to its aim of providing results that reflect real world application, but I hope that will improve with further maturation and better rules for benchmark participants. Of course, it may still make most sense to collect benchmark figures myself and even if doing so, being able to refer to the benchmarked configurations and get an indication of what hardware can achieve what performance is helpful in doing so. Renting a 72-GPU cluster is expensive and as far as I can see not typically available for a short time, so any benchmarking run by myself would be limited to 4-8 GPU configurations. If the gap in efficiency is huge for such setups vs the NVL72 then these smaller setups are maybe less interesting.

If I found the time to run benchmarks myself, what would I be testing? I'd move to DeepSeek V3.2. One of the big features of this release was the movement to a new attention mechanism which scales much closer to linearly with sequence length. With e.g. Kimi Linear and Qwen3-Next, other labs are moving in a similar direction experimentally at least. I'd try to set up 8 GPU configuration with sglang/vllm configured in a way that it would be capable of serving a commercial workload with varied input/output sequence lengths and test this is the case (Chutes provide their deployed configs which may be another reference point). I'd want to see how much the effective Wh per million input/output tokens varies depending on the different isl/osl workloads. These should be relatively similar given the linear attention mechanism, and if so it's a lot easier to estimate the rough energy cost of a series of your own queries of varied length. I would stick with the random input tokens for the time being.

So where does that leave us? All of this and we've got figures for two particular models, with one benchmark harness, a limited set of input/output sequence lengths, and a range of potential issues that might impact the conclusion. I think this is a useful yardstick / datapoint, though I'd like to get towards something that's even more useful and that I have more faith in.

Article changelog

2026-02-17:
- Changed GitHub links to point to SemiAnalysisAI/InferenceX rather than InferenceMAX/InferenceMAX, as they were broken by the upstream rename.
2026-01-09:
- Fix broken link.
- Add note that more complete system info would be helpful for reproducibility.
- Add note about variety of input/output sequence lengths tested.
2026-01-07: Initial publication date.

QEMU-based instruction execution counting

2025-12-02T12:00:00Z

Although analysing performance by way of instruction counting has obvious limitations, it can be helpful (especially when combined with appropriate analysis scripts) to get rapid feedback on the impact of code generation changes or to explore hypotheses about why code from one compiler might be performing differently from another - for instance, by looking at instruction mix in the most executed translation blocks. In this post we'll look at how to capture the necessary data to perform such an analysis using a QEMU plugin. Future posts will give details of the analysis scripts I've used, and walk through an example or two of putting them to use.

Modifying QEMU

Over the past few years, QEMU's plugin API has developed a fair bit. QEMU includes several plugins, and hotblocks provides almost what we want but doesn't allow configurability of the number of blocks it will print information on. I submitted a small patch series (and submitted it a second time addressing this and other minor issues found along the way. The series has now been accepted by the maintainer.

To build QEMU with this patch:

git clone https://github.com/qemu/qemu && cd qemu
git checkout v10.1.2
cat - <<'EOF' > hotblocks.patch
index 98404b6885..8ecf033997 100644
--- a/contrib/plugins/hotblocks.c
+++ b/contrib/plugins/hotblocks.c
@@ -73,28 +73,29 @@ static void exec_count_free(gpointer key, gpointer value, gpointer user_data)
 static void plugin_exit(qemu_plugin_id_t id, void *p)
 {
     g_autoptr(GString) report = g_string_new("collected ");
-    GList *counts, *it;
+    GList *counts, *sorted_counts, *it;
     int i;

     g_string_append_printf(report, "%d entries in the hash table\n",
                            g_hash_table_size(hotblocks));
     counts = g_hash_table_get_values(hotblocks);
-    it = g_list_sort_with_data(counts, cmp_exec_count, NULL);
+    sorted_counts = g_list_sort_with_data(counts, cmp_exec_count, NULL);

-    if (it) {
+    if (sorted_counts) {
         g_string_append_printf(report, "pc, tcount, icount, ecount\n");

-        for (i = 0; i < limit && it->next; i++, it = it->next) {
+        for (i = 0, it = sorted_counts; (limit == 0 || i < limit) && it;
+             i++, it = it->next) {
             ExecCount *rec = (ExecCount *) it->data;
             g_string_append_printf(
-                report, "0x%016"PRIx64", %d, %ld, %"PRId64"\n",
+                report, "0x%016"PRIx64", %d, %ld, %"PRIu64"\n",
                 rec->start_addr, rec->trans_count,
                 rec->insns,
                 qemu_plugin_u64_sum(
                     qemu_plugin_scoreboard_u64(rec->exec_count)));
         }

-        g_list_free(it);
+        g_list_free(sorted_counts);
     }

     qemu_plugin_outs(report->str);
@@ -170,6 +171,13 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
                 fprintf(stderr, "boolean argument parsing failed: %s\n", opt);
                 return -1;
             }
+        } else if (g_strcmp0(tokens[0], "limit") == 0) {
+            char *endptr = NULL;
+            limit = g_ascii_strtoull(tokens[1], &endptr, 10);
+            if (endptr == tokens[1] || *endptr != '\0') {
+                fprintf(stderr, "unsigned integer parsing failed: %s\n", opt);
+                return -1;
+            }
         } else {
             fprintf(stderr, "option parsing failed: %s\n", opt);
             return -1;
diff --git a/docs/about/emulation.rst b/docs/about/emulation.rst
index 4a7d1f4178..e8793b0f9c 100644
--- a/docs/about/emulation.rst
+++ b/docs/about/emulation.rst
@@ -463,6 +463,18 @@ Example::
   0x000000004002b0, 1, 4, 66087
   ...

+Behaviour can be tweaked with the following arguments:
+
+.. list-table:: Hot Blocks plugin arguments
+  :widths: 20 80
+  :header-rows: 1
+
+  * - Option
+    - Description
+  * - inline=true|false
+    - Use faster inline addition of a single counter.
+  * - limit=N
+    - The number of blocks to be printed. (Default: N = 20, use 0 for no limit).

 Hot Pages
 .........
EOF
patch -p1 < hotblocks.patch
./configure --prefix=$(pwd)/inst --target-list="riscv32-linux-user riscv64-linux-user"
make -j$(nproc)
cd ..

Using this plugin to capture statistics from running a binary under qemu-user

Assuming you have an appropriate sysroot, you can run a binary and have the execution information emitted to stderr by doing something like:

QEMUDIR=$HOME/qemu/build
SYSROOT=$HOME/rvsysroot
$QEMUDIR/qemu-riscv64 \
  -L $SYSROOT \
  -plugin $QEMUDIR/contrib/plugins/libhotblocks.so,limit=0,inline=on \
  -d plugin,nochain \
  my_rv64_binary

This produces output like:

collected 2229 entries in the hash table
pc, tcount, icount, ecount
0x00007fffee7012ba, 1, 1, 3737
0x00007fffee7012be, 1, 3, 3737
0x00007ffff741e738, 1, 23, 1074
0x00007fffee71bb38, 1, 5, 884
0x00007ffff741bb2e, 1, 11, 662
...

This listing indicates the address of the translation block, the number of times it's been translated, the number of instructions it contains, and the number of times it was executed. Note that a translation block is not the same as a basic block in the compiler. A translation block can span multiple basic blocks in the case of fallthrough, and this can also mean an instruction may show up in multiple translation blocks.

At least for my use cases, I need something a bit more involved than this. In order to add collection of these statistics to an existing benchmark harness I need a wrapper script that transparently collects these statistics to a file. It's also helpful to capture the runtime address of executable mappings for loaded libraries, allowing translation blocks to be attributed easily to either the binary itself or libc, libm etc. We have gdb connect to QEMU's gdbserver in order to dump those mappings. Do ensure you're using a recent version of QEMU (the version suggested in the patch application instructions is definitely good) for this as I wasted quite some time running into a bug with file descriptor numbers that caused odd breakage.

This qemu-forwarder.sh script will capture the plugin's output in a .qemu_out file and the mappings in a .map file, both of which can be later consumed by a detailed analysis script.

#!/bin/sh
QEMUDIR=$HOME/qemu/build
SYSROOT=$HOME/rvsysroot
QEMU="$QEMUDIR/qemu-riscv64 \
  -L $SYSROOT \
  -plugin $QEMUDIR/contrib/plugins/libhotblocks.so,limit=0,inline=on \
  -d plugin,nochain"

SUFFIX=""
if [ -e "$1.qemu_out" ]; then
  NUM=1
  while [ -e "$1.qemu_out.$NUM" ]; do
    NUM=$((NUM + 1))
  done
  SUFFIX=".$NUM"
fi

GDB_SOCK=$(mktemp -u)
setarch $(uname -m) -R $QEMU -g $GDB_SOCK -D $1.qemu_out$SUFFIX "$@" &
QEMU_PID=$!

RETRY_COUNT=0
while ! [ -e "$GDB_SOCK" ]; do
  RETRY_COUNT=$((RETRY_COUNT + 1))
  if [ $RETRY_COUNT -eq 10 ]; then
    echo "Timed out waiting for gdb socket to be created"
    exit 1
  fi
  sleep 0.1
  if ! kill -0 $QEMU_PID 2>/dev/null; then
    echo "QEMU process died before gdb socket was created"
    wait $QEMU_PID
    exit $?
  fi
done

gdb -batch \
  -ex "set pagination off" \
  -ex "target remote $GDB_SOCK" \
  -ex "break main" \
  -ex "continue" \
  -ex "set logging file $1.map$SUFFIX" \
  -ex "set logging enabled on" \
  -ex "info proc mappings" \
  -ex "detach" > /dev/null 2>&1
wait $QEMU_PID

The above will work under LLVM's lit, though you will need to use a recent enough version that doesn't strip HOME from the environment (or else edit the script accordingly). It also produces output in sequentially numbered files, again motivated by the desire to run under this script from lit as used by llvm-test-suite's SPEC configuration which can involve multiple invocations of the same binary for a given benchmark (e.g. 500.perlbench_r).

Analysing the output

A follow-up post will introduce the scripting I've built around this.

Recording and analysing results from running SPEC

Assuming you have qemu-forwarder.sh, in your llvm-test-suite directory:

CONF=clang-head-test
CLANG_BIN_DIR=$HOME/llvm-project/build/release/bin
CFLAGS="-march=rv64gc_zba_zbb_zbs"
cat - <<EOF > $CONF.cmake
set(CMAKE_SYSTEM_NAME Linux)

set(CMAKE_SYSROOT $HOME/rvsysroot)

set(CMAKE_C_COMPILER $CLANG_BIN_DIR/clang)
set(CMAKE_CXX_COMPILER $CLANG_BIN_DIR/clang++)

set(CMAKE_C_COMPILER_TARGET riscv64-linux-gnu)
set(CMAKE_CXX_COMPILER_TARGET riscv64-linux-gnu)
set(CMAKE_C_FLAGS_INIT "$CFLAGS")
set(CMAKE_CXX_FLAGS_INIT "$CFLAGS")

set(CMAKE_LINKER_TYPE LLD)

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
EOF
cmake -G Ninja \
  -B build.$CONF \
  --toolchain=$CONF.cmake \
  -DTEST_SUITE_SPEC2017_ROOT=~/cpu2017 \
  -DTEST_SUITE_SUBDIRS=External/SPEC \
  -DTEST_SUITE_COLLECT_CODE_SIZE=OFF \
  -DTEST_SUITE_COLLECT_COMPILE_TIME=OFF \
  -DTEST_SUITE_USER_MODE_EMULATION=ON \
  -DTEST_SUITE_RUN_UNDER=$(pwd)/qemu-forwarder.sh
cmake --build build.$CONF
$CLANG_BIN_DIR/llvm-lit -v --filter-out='.+_s|specrand' build.$CONF

The 526.blender_r test takes twice as long as the others, so you may wish to skip it by instead executing something like:

$CLANG_BIN_DIR/llvm-lit -v --filter-out='.+_s|specrand|blender' build.$CONF

If you want to re-run tests you must delete the previous .qemu_out and .map files, which can be done with:

[ -n "build.$CONF" ] && find "build.$CONF" -type f -name "*.qemu_out*" -exec sh -c '
    for q_file do
        base_path="${q_file%.qemu_out*}"
        rm -f "$q_file" "${base_path}.map"*
    done
' sh {} +

In order to compare two SPEC builds, you can use something like the following hacky script. Using the captured translation block execution data to generate a plain executed instruction count is overkill as the example tests/tcg/plugin/insn.c can easily dump for this for you directly. But by collecting the data upfront, you can easily dive right into a more detailed analysis when you see a surprising difference in executed instruction counts without rerunning the binary.

#!/usr/bin/env python3

from pathlib import Path
from collections import defaultdict
import sys

def collect_totals(root_dir):
    totals = defaultdict(int)
    root_path = Path(root_dir)/"External"

    for file_path in root_path.rglob("*.qemu_out*"):
        benchmark_name = file_path.parts[4]

        try:
            with file_path.open("r") as f:
                file_total = 0
                for line in f:
                    parts = line.strip().split(',')
                    # Only sum lines that match the expected format.
                    if len(parts) == 4 and parts[2].strip().isdigit():
                        # icount * ecount.
                        file_total += int(parts[2]) * int(parts[3])
                totals[benchmark_name] += file_total
        except Exception as e:
            print(f"Error reading {file_path}: {e}")

    return totals

if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("Usage: spec-compare-helper <dir_a> <dir_b>")
        sys.exit(1)

    dir_a, dir_b = sys.argv[1], sys.argv[2]
    totals_a = collect_totals(dir_a)
    totals_b = collect_totals(dir_b)

    benchmarks = sorted(set(totals_a.keys()) | set(totals_b.keys()))

    print(f"{'Benchmark':<20} {'DirA':>15} {'DirB':>15} {'Diff (%)':>10}")
    print("=" * 60)

    for benchmark in benchmarks:
        val_a = totals_a.get(benchmark, 0)
        val_b = totals_b.get(benchmark, 0)
        diff_pct = ((val_b - val_a) / val_a * 100) if val_a else float("inf")

        print(f"{benchmark:<20} {val_a:>15} {val_b:>15} {diff_pct:>9.2f}%")

Which produces output looking something like this:

Benchmark                       DirA            DirB   Diff (%)
============================================================
500.perlbench_r         180245097594    182078714777      1.02%
502.gcc_r               220874510659    219647717585     -0.56%
505.mcf_r               131589945456    134271153130      2.04%
508.namd_r              220648061019    216682202888     -1.80%
510.parest_r            291341820355    291844973715      0.17%
511.povray_r             31911866906     31103201809     -2.53%
519.lbm_r                94166321698     86910581403     -7.71%
520.omnetpp_r           138002605692    137676301622     -0.24%
523.xalancbmk_r         283566182007    284735075518      0.41%
525.x264_r              380165035845    379862173371     -0.08%
526.blender_r           660528270138    659361380750     -0.18%
531.deepsjeng_r         355058534962    349621355155     -1.53%
538.imagick_r           238573643488    238560676372     -0.01%
541.leela_r             421886351310    405423320484     -3.90%
544.nab_r               415595728542    391443973852     -5.81%
557.xz_r                132548718317    130229753780     -1.75%

It's worth highlighting that as we're running this under user-mode emulation, the dynamic instruction count naturally never counts any instructions on the kernel side that you would see if profiling a real system.

Article changelog

2025-12-15: Note that the qemu patches have now been accepted in the maintainer's tree.
2025-12-02: Initial publication date.

Minipost: Olmo 3 training cost

2025-12-01T12:00:00Z

Recently I jotted down some notes on LLM inference vs training costs for DeepSeek and I wanted to add on an additional datapoint for training cost based on the recently released Olmo3 models from the Allen Institute for AI ("Ai2"). The model family has 7B and 32B parameter models, with 'Think' variants available for 7B and 32B but so far only a 7B 'Instruct' non-reasoning version (but watch this space). What's particularly interesting about the Olmo models to me is that beyond providing open weights, the training scripts and datasets are openly available as well.

Going by the reported benchmarks at least it's competitive with less open models at a similar size, and importantly they've increased the supported context length from the rather limiting 4k tokens supported by the Olmo 2 series to a much more usable 64k tokens. Given the relatively small size these models are less capable than relatively chunky models like DeepSeek R1/V3.x or Kimi K2, but I've been impressed by the capability of 32B dense models for basic queries, and from my non-scientific testing both the 32B and 7B Olmo3 variants seem to do a reasonable job of summarising things like discussion threads. You can experiment yourself at playground.allenai.org.

Energy required for training Olmo 3

One of the neat things about this level of openness is that it should act as a floor in terms of performance for future models of this size class assuming they're appropriately funded and don't take too many risks chasing novelty. Rerunning the training process with an updated dataset and some minor tweaks is something you could imagine doing on some regular cadence, ideally as a shared endeavour. Imagining this effort in the future, how much energy is required? The initial version of the detailed Olmo 3 technical report unfortunately has little to say on this. We can get a back of the envelope figure in terms of GPU hours for pre-training based on the reported 7700 tokens per second per GPU for the 7B base model and 1900 tokens per second for the 32B base model and the ~6T token dataset. But even better than that, we can just ask the Ai2 folks (sometimes the internet really does work wonderfully!). After asking on their public Discord I was rapidly furnished with this helpful answer:

For some detailed numbers, we measured power consumption throughout training, along with total GPU hours. We used ~234k H100 hours to pretrain the 7B, and ~1.05m H100 hours to pretrain the 32B. 1900 TPS is generally what our trainer is capable of, but with restarts, evaluations, checkpointing, and occasional network issues, the 32B took 1.05m hours. We measured an average power consumption of ~621W while pretraining the 7B and ~649W while pretraining the 32B, and this means that our GPUs consumed ~146MWh for the 7B and ~681MWh for the 32B. We'll include more detailed GPU hour information in a future version of the paper, including for post-training!
Ai2 Olmo 3 team on their Discord.

So that's 0.681 GWh in GPU power draw for pretraining the 32B model and 0.146 GWh in GPU power draw for pretraining the 7B model. As noted in the quote, this is inclusive of restarts, checkpointing etc. But perhaps won't include previous early stage experimentation. I look forward to an updated technical report with full details, but pretraining should cover the bulk of the compute requirements (as a reference point, today's DeepSeek V3.2 paper found it notable that the post-training compute budget exceeded 10% of the pretraining cost).

The 0.681 GWh figure doesn't account for full system power and cooling cost. I'd love to be corrected, but I believe a 1.5x-2x multiplier would be an assumption towards the upper end. But for the sake of this yardstick comparison let's look at a few comparisons based on the reported number:

0.681 GWh of electricity would cost about £180k at UK residential rates (capped at 26.35p per kWh currently). Substantially less in the USA.
A larger leisure centre with a pool consumes ~2.5 GWh of energy per year. I don't know if the idea of a "leisure centre" translates outside of the UK, but basically it's a swimming pool plus gym, squash/tennis courts etc.
- The linked page claims ~2 GWh of energy in gas and 0.5 GWh in electricity. For the gas, to compare like with like you'd need to consider the source of energy for the electricity used for Olmo training.
0.681 GWh is ~0.11% of LHC's annual 600 GWh energy consumption or ~0.05% of CERN's annual consumption.
We can estimate a Boeing 787-9 flying from London Heathrow to SFO consumes jet fuel containing ~0.58 GWh of energy.
- Calculated with 8638km distance, 5.62kg fuel/km (taking the most economic 787-9 long haul figure from this table on Wikipedia and 11.95kWh/kg specific energy of jet fuel).
- This is a yardstick rather than a direct comparison. A direct comparison to the GWh of electricity used for the GPU compute of the LLM would depend on the source of the electricity. If it was e.g. gas rather than solar/hydro/wind then you'd want to compare the number of GWh consumed to create that electricity which would of course be higher.
- As a further point of reference, FlightAware indicates 5 separate direct LHR to SFO flights scheduled per day.

More efficient LLM training

We can hope for new breakthroughs, more efficient hardware, better datasets and so on. But here is some work I noticed in the area. Fair warning: this isn't my field, and we have to recognise applying a research result to a production training run is sure to have challenges even if the research suggests the trade-offs are worthwhile. So consider this vague gesticulating about seemingly interesting work that is going on and find someone who knows what they're talking about to confirm the degree to which it is interesting/viable.

Mixture of Experts (MoE) models are substantially cheaper to train which is one reason the industry has moved in that direction. The next Ai2 Olmo model is expected to be MoE. The Qwen blog has a graph comparing the relative training cost in GPU hours of the dense Qwen3-32B vs Qwen3-30B-A3b vs Qwen3-Next-80B-A3B, where the latter makes further architectural changes, reporting a 10.7x reduction. ~2.5x of that is going to come from the reduced corpus size (15T tokens down from 36T), but that still leaves plenty of improvement from other factors.
Maybe it will be shown viable to train in lower precision such as MXFP8 or even NVFP4, which would allow much more throughput for a similar energy budget. Nvidia have worked to demonstrate this can be effective for both formats (see also this work from MIT).
Also from Nvidia, Nemotron Elastic showed a model architecture that allows deriving smaller models without doing a separate pre-training runs.

Finally, the cheapest way to train an LLM from scratch is...to find a way to avoid the need to. For models like Olmo 3 that release the base model and checkpoints, people can apply their own post-training or perform additional pre-training.

Bonus comparison point: Apertus

Apertus is a Swiss project to produce an open LLM, with 70B and 8B models released so far. Their full tech report notes the following "Once a production environment has been set up, we estimate that the model can be realistically trained in approximately 90 days on 4096 GPUs, accounting for overheads. If we assume 560 W power usage per Grace-Hopper module in this period, below the set power limit of 660 W, we can estimate 5 GWh power usage for the compute of the pretraining run."

Article changelog

2025-12-04: Add link to "Four Over Six" NVFP4 training paper.
2025-12-02: Added clarifying note about energy via gas in the leisure centre comparison.
2025-12-01: Initial publication date.

Minipost: Benchmarking the Hetzner AX102 vs CCX53

2025-11-30T12:00:00Z

I recently had reason to do a quick comparison of the performance of the Hetzner AX102 dedicated server and the high-end 'dedicated' CCX53 VPS on Hetzner Cloud and thought I may as well write up the results for posterity. I'm incapable of starting a post without some kind of disclaimer so here comes the one for this post: naturally the two products have major differences in terms of flexibility (spin-up/down at will, vs pay a small setup fee and endure a wait time depending on hardware availability). So depending on your use case, your requirements with respect to that flexibility may override any cost differential.

Specs

All costs are exclusive of VAT, assuming the lowest cost data center location, and inclusive of IPv4 address.

AX102:

16 core Ryzen 9 7950X3D (32 threads)
128GB DDR5 RAM
2 x 1.92TB NVMe
104 EUR/month, 39 EUR one-off setup fee.

CCX53

Unknown AMD CPU exposing 32vCPU (physical cores? threads?)
128GB RAM
600GB NVMe
192.49 EUR/month maximum charge. 0.3085 EUR per hour (if you keep the same VPS active over the month it won't exceed the monthly price cap, so you effectively get a small discount on the per-hour cost).

Benchmark

Building Clang+LLVM+LLD, everyone's favourite workload! Both systems are running an up to date Arch Linux (more details on setting this up on the CCX53 in the appendix below) with clang 21.1.6. The dedicated machine has the advantage of RAID 0 across the two SSDs, but also has encrypted rootfs configured. I didn't bother to set that up for the CCX53 VPS.

sudo pacman -Syu --needed clang lld cmake ninja wget
LLVM_VER=21.1.6
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-${LLVM_VER}/llvm-project-${LLVM_VER}.src.tar.xz
tar -xvf llvm-project-${LLVM_VER}.src.tar.xz
cd llvm-project-${LLVM_VER}.src

cmake -G Ninja \
  -DLLVM_ENABLE_PROJECTS='clang;lld' \
  -DLLVM_TARGETS_TO_BUILD="all" \
  -DLLVM_CCACHE_BUILD=OFF \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DLLVM_ENABLE_LLD=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_ENABLE_ASSERTIONS=ON \
  -S llvm \
  -B build
time cmake --build build

printf "### Version info ###\n"
clang --version | head -n 1

On both machines, ninja shows 5575 build steps.

Results:

AX102
- 10m27s (627s)
CCX53
- 14m11s (851s, about 1.36x the AX102)

Running the clang and LLVM tests with ./build/bin/llvm-lit -s --order=lexical llvm/test clang/test (which shows 9402 tests) gives:

AX102
- 3m39s (219s)
CCX53
- 4m28s (268s, about 1.24x the AX102)

I ran these multiple times, and in the case of the CCX53 across two different VMs in different regions and saw only a few percentage points variance.

Focusing on the results for build clang/llvm/lld, let's figure out the cost for 1000 from-scratch builds. Not so much as it's a representative workload, but because it gives an easy to compare metric that captures both the difference in price and in performance. So calculating time_per_build_in_hours * 1000 * cost_per_hour:

AX102
- (626.6 / 3600) * 1000 * (104/720) = 25.14 EUR
- Or if you include the setup fee and assume it's amortised over 12 months:
  - (626.6/3600) * 1000 * ((104 + (39/12))/720) = 25.93 EUR
CCX53
- (850.6 / 3600) * 1000 * (192.49/720) = 63.17 EUR
- Or using the 0.3085 EUR/hr price which you would pay if you didn't run for the whole month:
  - (850.6 / 3600) * 1000 * 0.3085 = 72.89 EUR

Appendix: CCX53 Arch Linux setup

This could be scripted, but I just created the VPS via their web UI. Then after it was provisioned, used that web UI to have it boot into a rescue system. Then do an Arch bootstrap that roughly mirrors the one I use on a dedicated build machine except that we don't bother with encrypting the rootfs. The CCX* server types at least use UEFI so we can keep using efistub for boot.

First get a bootstrap environment and enter it:

wget http://mirror.hetzner.de/archlinux/iso/latest/archlinux-bootstrap-x86_64.tar.zst
tar -xvf archlinux-bootstrap-x86_64.tar.zst --numeric-owner
sed -i '1s;^;Server=https://mirror.hetzner.de/archlinux/$repo/os/$arch\n\n;' root.x86_64/etc/pacman.d/mirrorlist
mount --bind root.x86_64/ root.x86_64/ # See <https://bugs.archlinux.org/task/46169>
printf "About to enter bootstrap chroot\n===============================\n"
./root.x86_64/bin/arch-chroot root.x86_64/

Now set info that will be used throughout the process:

export NEW_HOST_NAME=archvps
export PUBLIC_SSH_KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOfpPQ1j+XLsapAhONAQmvu6TZGT5y8jeziM4Vio1NrA asb@plurp"
export NEW_USER=asb

And now proceed to set up the disks, create filesystems, perform an initial bootstrap and chroot into the new rootfs:

pacman-key --init
pacman-key --populate archlinux
pacman -Sy --noconfirm xfsprogs dosfstools

sfdisk /dev/sda <<EOF
label: gpt

start=1MiB, size=255MiB, type=uefi
start=256MiB, type=linux
EOF

mkfs.fat -F32 /dev/sda1
mkfs.xfs /dev/sda2

mount /dev/sda2 /mnt
mkdir /mnt/boot
mount /dev/sda1 /mnt/boot
pacstrap /mnt base linux linux-firmware efibootmgr \
  xfsprogs dosfstools \
   python3 \
  openssh sudo net-tools git man-db man-pages vim
genfstab -U /mnt >> /mnt/etc/fstab

printf "About to enter newrootfs chroot\n===============================\n"
arch-chroot /mnt

Do final configuration from within the chroot:

sed /etc/locale.gen -i -e "s/^\#en_GB.UTF-8 UTF-8.*/en_GB.UTF-8 UTF-8/"
locale-gen
# Ignore "System has not been booted with systemd" and "Failed to connect to bus" error for next command.
systemd-firstboot --locale=en_GB.UTF-8 --timezone=UTC --hostname="$NEW_HOST_NAME"
ln -s /dev/null /etc/udev/rules.d/80-net-setup-link.rules # disable persistent network names

# No longer need to disable large fallback image as Arch stopped generating it
# by default

printf "efibootmgr before changes:\n==========================\n"
efibootmgr -u
# Set up efistub
efibootmgr \
  --disk /dev/sda \
  --part 1 \
  --create \
  --label 'Arch Linux' \
  --loader /vmlinuz-linux \
  --unicode "root=/dev/sda2 rw initrd=\initramfs-linux.img" \
  --verbose
printf "efibootmgr after changes:\n=========================\n"
efibootmgr -u

mkswap --size=8G --file /swapfile
cat - <<EOF > /etc/systemd/system/swapfile.swap
[Unit]
Description=Swap file

[Swap]
What=/swapfile

[Install]
WantedBy=multi-user.target
EOF
systemctl enable swapfile.swap

cat - <<EOF > /etc/systemd/network/10-eth0.network
[Match]
Name=eth0

[Network]
DHCP=yes
Address=$(ip -6 addr show dev eth0 scope global | grep "scope global" | cut -d' ' -f6)
Gateway=$(ip route show | head -n 1 | cut -d' ' -f 3)
Gateway=fe80::1
EOF
systemctl enable systemd-networkd.service systemd-resolved.service systemd-timesyncd.service
printf "PasswordAuthentication no\n" > /etc/ssh/sshd_config.d/20-no-password-auth.conf
systemctl enable sshd.service
useradd -m -g users -G wheel -s /bin/bash "$NEW_USER"
usermod --pass='!' root # disable root login
chmod +w /etc/sudoers
printf "%%wheel ALL=(ALL) ALL\n" >> /etc/sudoers
chmod -w /etc/sudoers
mkdir "/home/$NEW_USER/.ssh"
printf "%s\n" "$PUBLIC_SSH_KEY" > "/home/$NEW_USER/.ssh/authorized_keys"
chmod 700 "/home/$NEW_USER/.ssh"
chmod 600 "/home/$NEW_USER/.ssh/authorized_keys"
chown -R "$NEW_USER:users" "/home/$NEW_USER/.ssh"

Now set password:

passwd "$NEW_USER"

Then ctrl-d twice and set a symlink for resolv.conf:

ln -sf ../run/systemd/resolve/stub-resolv.conf root.x86_64/mnt/etc/resolv.conf

Finally, reboot.

Remember to ssh-keygen -R $THE_IP_ADDRESS so you don't get ssh host verification errors.

Article changelog

2025-11-30: Initial publication date.

Minipost: LLM inference vs training costs for DeepSeek

2025-11-29T12:00:00Z

Tl;dr: Based on published data from DeepSeek, we can estimate it takes something like ~70 days of inference traffic (served by DeepSeek themselves, ignoring any other providers) to match the GPU hours used for the final training run for V3 and R1.

Simon Willison recently reshared some figures on inference costs for LLMs. I couldn't agree more with the comment further down that thread "The big AI labs continue to be infuriatingly opaque about the actual figures for their total electricity and water consumption".

A number of responses wonder about the cost of training. If you accept the reported figures for serving a query, what impact does it have if you amortise the energy spent training the model over the served queries? Mistral did this for their lifecycle analysis but they grouped together "training and inference" and kept confidential the ratio of energy for training vs inference by reporting a figure that combined the training cost with 18 months of usage. The thread reminded me of another datapoint available for DeepSeek that seemed worth writing up. I think this gives some helpful intuition for the amortised cost of training for a widely used model of that size, but to state the obvious any attempt to apply that intuition to other models is totally reliant on how widely used it is.

DeepSeek have published figures both on training and on inference for DeepSeek's website and API users. I will attempt to consistently refer to the figure for training as "final run training cost" to reflect the fact the number of GPU hours used in experimentation and failed attempts isn't reported. For final run training for DeepSeek-R1:

2.788M H800 GPU hours for V3 which serves as the R1 base (see Table 1 in the V3 technical report).
0.147M H800 GPU hours for building R1 on top of V3 (see Supplementary Table 4 in the supplementary information for the R1 Nature article).
Total: 2.935M H800 GPU hours

Now for inference, back in February DeepSeek wrote up details of their inference system giving details of cost of serving, profit margin, and load over a 24h period. So yes, we're extrapolating from this datapoint and assuming it's representative. Given the worldwide inference of DeepSeek R1/V3 is surely much larger (being openly licensed there are many vendors who are serving it), I'm not overly worried about this aspect. Their reported average inference serving infrastructure occupancy is 226.75 nodes (each node containing 8 H800 GPUs), meaning 43536 H800 GPU hours per day. At that rate, it will take ~67.5 days of traffic for the same number of H800 GPU hours to be used for inference as for the final training run.

All this to say, for a widely used model of DeepSeek R1 scale when looking at the cost of inference, accounting for the amortised final run training cost is more likely to be a multiplier of 2x or less rather than something much larger. In terms of energy, this does assume that the power draw of the H800 GPUs while running inference is similar to the draw during training. And to underline again, the reported training cost surely doesn't include experimentation, aborted runs etc.

Article changelog

2025-11-29: Initial publication date.