build-sys: Rework sealing to be one build step #1898

cgwalters · 2026-01-08T22:57:24Z

Now that we're doing a "from scratch" build we don't
have the mtime issue, and so we can change our build system
to do everything in a single step.

Assisted-by: OpenCode (Opus 4.5)

gemini-code-assist

Code Review

This pull request is an excellent refactoring of the build system, unifying the standard and sealed image build processes into a single, multi-stage Docker build. This simplifies the build logic, improves maintainability, and likely enhances build caching. The changes are well-structured, breaking down the complex sealing process into smaller, dedicated scripts and Dockerfile stages. The removal of outdated files and the consolidation of logic into a single, more powerful Dockerfile is a significant cleanup. My review includes a few suggestions to improve the robustness of some of the new shell scripts to better handle cases where file globs might match more than one file.

gemini-code-assist · 2026-01-08T22:59:30Z

contrib/packaging/finalize-uki

+kver=$(cd /usr/lib/modules && echo *)
+if [ -z "$kver" ] || [ "$kver" = "*" ]; then
+  echo "Error: No kernel found" >&2
+  exit 1
+fi


The command kver=$(cd /usr/lib/modules && echo *) is not robust if multiple kernel version directories exist under /usr/lib/modules. In such a case, kver would contain a space-separated list of versions, which would break subsequent commands. It's safer to ensure exactly one kernel directory is found and fail otherwise.

Suggested change

kver=$(cd /usr/lib/modules && echo *)

if [ -z "$kver" ] || [ "$kver" = "*" ]; then

echo "Error: No kernel found" >&2

exit 1

fi

kver_paths=(/usr/lib/modules/*)

if [ "${#kver_paths[@]}" -ne 1 ]; then

echo "Error: Expected 1 kernel version directory, but found ${#kver_paths[@]}" >&2

ls -l /usr/lib/modules/

exit 1

fi

kver=$(basename "${kver_paths[0]}")

gemini-code-assist · 2026-01-08T22:59:30Z

contrib/packaging/seal-uki

+kver=$(cd "${target}/usr/lib/modules" && echo *)
+if [ -z "$kver" ] || [ "$kver" = "*" ]; then
+  echo "Error: No kernel found" >&2
+  exit 1
+fi


The command kver=$(cd "${target}/usr/lib/modules" && echo *) is not robust if multiple kernel version directories exist. In such a case, kver would contain a space-separated list of versions, which would break subsequent commands. It's safer to ensure exactly one kernel directory is found and fail otherwise.

Suggested change

kver=$(cd "${target}/usr/lib/modules" && echo *)

if [ -z "$kver" ] || [ "$kver" = "*" ]; then

echo "Error: No kernel found" >&2

exit 1

fi

kver_paths=("${target}/usr/lib/modules"/*)

if [ "${#kver_paths[@]}" -ne 1 ]; then

echo "Error: Expected 1 kernel version directory, but found ${#kver_paths[@]}" >&2

ls -l "${target}/usr/lib/modules/"

exit 1

fi

kver=$(basename "${kver_paths[0]}")

gemini-code-assist · 2026-01-08T22:59:30Z

Dockerfile

+sdboot_unsigned=$(ls ./usr/lib/systemd/boot/efi/systemd-boot*.efi)
+sdboot_bn=$(basename ${sdboot_unsigned})


The use of ls with a glob to find the systemd-boot EFI binary can be fragile. If the glob matches more than one file, the sdboot_unsigned variable will contain a multi-line string, which could cause sbsign to behave unexpectedly. It's safer to ensure exactly one file is found.

sdboot_unsigned_files=(./usr/lib/systemd/boot/efi/systemd-boot*.efi) if [ ${#sdboot_unsigned_files[@]} -ne 1 ]; then echo "Error: Expected 1 systemd-boot EFI file, but found ${#sdboot_unsigned_files[@]}" >&2 ls -l ./usr/lib/systemd/boot/efi/ exit 1 fi sdboot_unsigned="${sdboot_unsigned_files[0]}" sdboot_bn=$(basename "${sdboot_unsigned}")

gemini-code-assist · 2026-01-08T22:59:30Z

contrib/packaging/switch-to-sdboot

+sdboot=$(ls /usr/lib/systemd/boot/efi/systemd-boot*.efi)
+sdboot_bn=$(basename "${sdboot}")


The use of ls with a glob to find the systemd-boot EFI binary can be fragile. If the glob matches more than one file, the sdboot variable will contain a multi-line string. This would cause the install command on line 24 to fail because it would be interpreted as multiple destination arguments. It's safer to ensure exactly one file is found.

Suggested change

sdboot=$(ls /usr/lib/systemd/boot/efi/systemd-boot*.efi)

sdboot_bn=$(basename "${sdboot}")

sdboot_files=(/usr/lib/systemd/boot/efi/systemd-boot*.efi)

if [ "${#sdboot_files[@]}" -ne 1 ]; then

echo "Error: Expected 1 systemd-boot EFI file, but found ${#sdboot_files[@]}" >&2

ls -l /usr/lib/systemd/boot/efi/

exit 1

fi

sdboot="${sdboot_files[0]}"

sdboot_bn=$(basename "${sdboot}")

cgwalters · 2026-01-09T18:25:44Z

error: Installing to disk: Setting up composefs boot: Setting up UKI boot: Writing 6.12.0-174.el10.x86_64.efi to ESP: The UKI has the wrong composefs= parameter (is 'sha512:1c88433c334194ecbf03356ba12fdcf7bb16c4ba075ff72d44666237136c162b35cc0bcfe923c8ace1fe27d724c18c6dc386d644d98ec5a890e63bab469b21b2', should be sha512:5c8b606415d239698de34b714935a684b38a763eddb573161213e98ed2ad91b69e10f236ded502af078744ce94ddbe353368be3dda5a182c631edd8e1ec0add0)

OK this was working for me at one point but then broke and it took me quite a while to figure out why: containers/storage#743 (comment)

cgwalters · 2026-01-09T19:42:19Z

OK this will depend on more composefs-rs work so we need to get back on git main, which is #1791

jeckersb · 2026-01-12T18:55:29Z

OK this will depend on more composefs-rs work so we need to get back on git main, which is #1791

Merged now, so I think this is unblocked?

cgwalters · 2026-01-12T19:04:12Z

I'm working on containers/composefs-rs#209 because I was hitting unexpected diffs - will flesh this out more

cgwalters · 2026-01-14T17:43:47Z

CI / test-integration (fedora-44, false, composefs-sealeduki-sdboot) (pull_request)Successful in 34m

🎉 finally!!!

cgwalters · 2026-01-14T20:33:28Z

This requires containers/composefs-rs#209

Though it'd also be cleaner if rebased on top of the buildsys changes from #1912

cgwalters · 2026-01-16T19:00:20Z

                     content: error: boot data installation failed: installing component EFI: No such file or directory (os error 2)

OK it took me+agent quite a while to figure out this is caused by a combination of coreos/bootupd#995 plus us not having #1816

Effectively we had skew between the base image and the yum repos which is really https://gitlab.com/redhat/centos-stream/containers/bootc/-/issues/1174

The base image may be built from a compose that has newer packages than what's available on the public mirrors. This causes version skew where packages like bootupd have different versions between the base image and our built image. For example, bootupd 0.2.32 changed the EFI file layout from /usr/lib/bootupd/updates/EFI/ to /usr/lib/efi/, and if we build with an older bootupd from mirrors while the target image has the newer layout, bootloader installation fails. Enable the CentOS Stream compose repos with higher priority to ensure we get matching versions. xref https://gitlab.com/redhat/centos-stream/containers/bootc/-/issues/1174 Signed-off-by: Colin Walters <walters@verbum.org>

Now that we're doing a "from scratch" build we don't have the mtime issue, and so we can change our build system to do everything in a single step. Assisted-by: OpenCode (Opus 4.5) Signed-off-by: Colin Walters <walters@verbum.org>

Add support for bind-mounting an extra source directory into container builds, primarily for developing against a local composefs-rs checkout. Usage: BOOTC_extra_src=$HOME/src/composefs-rs just build The directory is mounted at /run/extra-src inside the container. When using this, also patch Cargo.toml to use path dependencies pointing to /run/extra-src/crates/.... Signed-off-by: Colin Walters <walters@verbum.org> Assisted-by: OpenCode (Opus 4.5) Signed-off-by: Colin Walters <walters@verbum.org>

The composefs-rs PR 209 has been merged to main. This updates bootc to use the containers/composefs-rs repository at the merge commit. Key API changes: - Directory::default() -> Directory::new(Stat::uninitialized()) - read_filesystem() no longer takes stat_root parameter - New read_container_root() for OCI containers (propagates /usr metadata to root) - stat_root CLI flag renamed to no_propagate_usr_to_root with inverted logic See containers/composefs-rs#209 Signed-off-by: Colin Walters <walters@verbum.org>

We changed how composefs digests are computed to ensure that mounted filesystem via --mount=type=image and install-time view (OCI tar layer processing from containers-storage) match. There were various problems like differing metadata for `/` among other things. Signed-off-by: Colin Walters <walters@verbum.org>

Add comprehensive documentation for building sealed bootc images, focusing on the core concepts and the key command: `bootc container compute-composefs-digest`. Key additions: - Document how sealed images work (UKI + composefs digest + Secure Boot) - Explain the build workflow abstractly without distribution-specific details - Document the compute-composefs-digest command and its options - Add section on generating/signing UKIs with ukify - Document developer testing commands (just variant=composefs-sealeduki-sdboot) - Add validation tooling documentation This provides the foundation for distribution-specific documentation to build upon with concrete Containerfile examples. Assisted-by: OpenCode (Claude Sonnet 4) Signed-off-by: Colin Walters <walters@verbum.org>

Justfile changes: - Organize targets into groups (core, testing, docs, debugging, maintenance) - Add `list-variants` target to show available build variants - Simplify comments to be concise single-line descriptions - Move composefs targets (build-sealed, test-composefs) into core group CONTRIBUTING.md changes: - Reference `just --list` and `just list-variants` instead of duplicating - Remove tables that duplicate Justfile information - Fix broken link to cli.rs The Justfile is now self-documenting via `just --list` (grouped targets) and `just list-variants` (build configuration options). Assisted-by: OpenCode (Claude Sonnet 4) Signed-off-by: Colin Walters <walters@verbum.org>

This is useful when debugging issues with stale cached layers, such as package version skew between base images and repos. Signed-off-by: Colin Walters <walters@verbum.org>

github-actions bot added the area/documentation Updates to the documentation label Jan 8, 2026

bootc-bot bot requested a review from jeckersb January 8, 2026 22:57

gemini-code-assist bot reviewed Jan 8, 2026

View reviewed changes

cgwalters force-pushed the sealed-scratch-rebase branch from 0ba5f48 to 4c7882c Compare January 8, 2026 23:57

cgwalters force-pushed the sealed-scratch-rebase branch 3 times, most recently from 182d8fe to 35cbf82 Compare January 14, 2026 16:42

cgwalters force-pushed the sealed-scratch-rebase branch 2 times, most recently from e35a329 to 182f6c2 Compare January 16, 2026 14:42

cgwalters added 8 commits January 16, 2026 14:40

build-sys: Rework sealing to be one build step

99bcd29

Now that we're doing a "from scratch" build we don't have the mtime issue, and so we can change our build system to do everything in a single step. Assisted-by: OpenCode (Opus 4.5) Signed-off-by: Colin Walters <walters@verbum.org>

build-sys: Add BOOTC_nocache option to force rebuild without cache

75b5312

This is useful when debugging issues with stale cached layers, such as package version skew between base images and repos. Signed-off-by: Colin Walters <walters@verbum.org>

cgwalters force-pushed the sealed-scratch-rebase branch from 182f6c2 to 75b5312 Compare January 16, 2026 19:40

-kver=$(cd /usr/lib/modules && echo *)
-if [ -z "$kver" ] || [ "$kver" = "*" ]; then
-  echo "Error: No kernel found" >&2
-  exit 1
-fi
+kver_paths=(/usr/lib/modules/*)
+if [ "${#kver_paths[@]}" -ne 1 ]; then
+    echo "Error: Expected 1 kernel version directory, but found ${#kver_paths[@]}" >&2
+    ls -l /usr/lib/modules/
+    exit 1
+fi
+kver=$(basename "${kver_paths[0]}")

		sdboot_unsigned=$(ls ./usr/lib/systemd/boot/efi/systemd-boot*.efi)
		sdboot_bn=$(basename ${sdboot_unsigned})

		sdboot=$(ls /usr/lib/systemd/boot/efi/systemd-boot*.efi)
		sdboot_bn=$(basename "${sdboot}")

-sdboot=$(ls /usr/lib/systemd/boot/efi/systemd-boot*.efi)
-sdboot_bn=$(basename "${sdboot}")
+sdboot_files=(/usr/lib/systemd/boot/efi/systemd-boot*.efi)
+if [ "${#sdboot_files[@]}" -ne 1 ]; then
+    echo "Error: Expected 1 systemd-boot EFI file, but found ${#sdboot_files[@]}" >&2
+    ls -l /usr/lib/systemd/boot/efi/
+    exit 1
+fi
+sdboot="${sdboot_files[0]}"
+sdboot_bn=$(basename "${sdboot}")

build-sys: Rework sealing to be one build step #1898

Are you sure you want to change the base?

build-sys: Rework sealing to be one build step #1898

Uh oh!

Conversation

cgwalters commented Jan 8, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters commented Jan 9, 2026

Uh oh!

cgwalters commented Jan 9, 2026

Uh oh!

jeckersb commented Jan 12, 2026

Uh oh!

cgwalters commented Jan 12, 2026

Uh oh!

cgwalters commented Jan 14, 2026

Uh oh!

cgwalters commented Jan 14, 2026

Uh oh!

cgwalters commented Jan 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants