arm64 builds broken since Alpine 3.23 update - APK triggers fail under QEMU

[ledermann]

I'm encountering an issue when building multi-arch Docker images based on ruby:3.4.7-alpine.

Problem

Since the recent update to Alpine 3.23 (#520), building for linux/arm64 under QEMU emulation (in GitHub Actions) fails. APK package triggers crash with:

#7 [linux/arm64 2/2] RUN apk add --no-cache ca-certificates
#7 2.199 (1/1) Installing ca-certificates (20251003-r0)
#7 2.240 Executing busybox-1.37.0-r29.trigger
#7 2.244 * execve: No such file or directory
#7 2.246 ERROR: lib/apk/exec/busybox-1.37.0-r29.trigger: exited with error 127
#7 2.246 Executing ca-certificates-20251003-r0.trigger
#7 2.248 * execve: No such file or directory
#7 2.249 ERROR: lib/apk/exec/ca-certificates-20251003-r0.trigger: exited with error 127
#7 2.261 2 errors; 9 MiB in 22 packages
#7 ERROR: process "/dev/.buildkit_qemu_emulator /bin/sh -c apk add --no-cache ca-certificates" did not complete successfully: exit code: 2

This happens when installing a package that has post-install triggers, such as ca-certificates.

Reproduction

I have created a minimal reproduction repository:
https://github.com/ledermann/ruby-alpine-3.23-apk-repro

This is the failing workflow run:
https://github.com/ledermann/ruby-alpine-3.23-apk-repro/actions/runs/19952870438/job/57216316282

The Dockerfile is just this:

FROM ruby:3.4.7-alpine
RUN apk add --no-cache ca-certificates

Pinning to ruby:3.4.7-alpine3.22 works as expected. The issue seems to be caused by changes in Alpine 3.23's busybox or apk-tools that are incompatible with QEMU.

[Earlopain]

It looks very specific about busybox. I found this upstream issue: https://gitlab.alpinelinux.org/alpine/aports/-/issues/17775, it mentions https://gitlab.alpinelinux.org/alpine/aports/-/commit/7aa508c1c5abe7a9a193086a2f4932a135b4f979. For the reporter, downgrading busybox resolves the issue.

I wonder if this is something that needs to be handled in https://github.com/docker/setup-qemu-action?

[yosifkit]

Works fine if you just install the qemu-user-static package, so yes this is an issue with that action and not anything in the images. You can just skip the action and install qemu-static yourself:

-            - uses: docker/setup-qemu-action@v3
+            - name: "setup qemu"
+              run: |
+                set -eux
+                sudo apt-get update
+                sudo apt-get install qemu-user-static

[ledermann]

Thanks! After reviewing my setup, I realized that I was running docker/setup-qemu-action after docker/setup-buildx-action. That ordering prevents Buildx from picking up the QEMU emulators during builder creation, and the README shows that QEMU should be set up before Buildx.

After switching the order, the build using docker/setup-qemu-action works again, no qemu-user-static required. Interestingly, this didn’t cause issues with Alpine 3.22 in my projects, but it does with Alpine 3.23.

IMHO, this issue can be closed, because it is not related to Ruby.

[chadlwilson]

FWIW, there is more investigation upstream at https://gitlab.alpinelinux.org/alpine/aports/-/issues/17775 now, and another similar issue at docker/build-push-action#1436

Definitely triggered by a change in Alpine 3.23/apk-tools v3 exposing some QEMU cross-arch issue in some particular configuration common with buildx/buildkit.

I have not tried any of the above yet, but I wonder if there is a logical reason why either installing qemu-user-static or adding on setup-qemu-action/binfmt would change the behaviour here compared to the "default" way buildx/buildkit use QEMU?

It is, after all, noted as optional on the README.