feat: optimize binary size, crun, zstd compression, online/offline build variants#117
Conversation
stevensbkang
left a comment
There was a problem hiding this comment.
Would you please consider the comment?
| RUNC_ARCH=${ARCH} | ||
| # Download crun - add error checking | ||
| # crun does not publish 32-bit ARM binaries; fail early for unsupported architectures | ||
| if [ "${ARCH}" = "arm" ]; then |
There was a problem hiding this comment.
We encountered the exact same limitation for containerd artefacts and ended up building our own https://github.com/portainer/kubesolo/blob/develop/build/containerd.Dockerfile.
I have done some testing and the following seems to do the job:
# crun build for arm/v7 using Debian Linux (glibc)
#
# Strategy: Build natively inside an arm/v7 Debian container via QEMU.
# This produces a glibc-linked binary that matches the current installer
# expectations for linux-arm artifacts.
#
# Build with: docker build --platform linux/arm/v7 -f ./build/crun.Dockerfile -t crun-arm32-builder .
# Extract with:
# docker create --name crun-extract crun-arm32-builder noop
# docker cp crun-extract:/crun ./internal/core/embedded/bin/crun
# docker rm crun-extract
FROM --platform=linux/arm/v7 debian:12-slim AS builder
ARG CRUN_VERSION=1.26
# Install build dependencies
RUN apt-get update && \
apt-get install -y --no-install-recommends \
autoconf \
automake \
build-essential \
ca-certificates \
git \
go-md2man \
gperf \
libcap-dev \
libseccomp-dev \
libsystemd-dev \
libtool \
linux-libc-dev \
pkg-config \
python3 \
&& rm -rf /var/lib/apt/lists/*
# Clone crun repository
RUN git clone --recursive https://github.com/containers/crun.git /src/crun
WORKDIR /src/crun
# Checkout specific version
RUN git checkout ${CRUN_VERSION} && \
git submodule update --init --recursive
# Generate configure script
RUN ./autogen.sh
# Configure for glibc build with systemd support to match upstream features
RUN ./configure \
--enable-systemd \
--enable-embedded-yajl \
--with-cap \
--with-seccomp
# Build
RUN make -j"$(nproc)"
# Package output
RUN mkdir -p /output/bin && \
cp crun /output/bin/ && \
strip /output/bin/crun
# Final stage: copy the built binary to a clean image
FROM scratch
COPY --from=0 /output/bin/crun /
Testing on a arm/v7 RPi:
root@pi2b:~# dpkg --print-architecture
armhf
root@pi2b:~# uname -m
armv7l
root@pi2b:/tmp# ./crun --version
crun version 1.26
commit: 3241e671f92c33b0c003cd7de319e4f32add6231
rundir: /run/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
Would you be able to add this to the PR, please? I will take care of the release to produce offline vs online in a separate PR 😄
There was a problem hiding this comment.
Ah, I didn't pay attention to the fact KubeSolo is supporting 32-bit ARM as well. I actually a compiled from source version, I could contribute, I am compiling from source for all flavors if you prefer. Regardless, I will update the PR
There was a problem hiding this comment.
I am finalizing this fix.
There was a problem hiding this comment.
Ok all pushed and rebased. Please review
5e55a06 to
aa2a92b
Compare
…ine builds
Phase 1: Replace runc (~10MB) with crun (~1MB) as OCI runtime.
Phase 2: zstd-compress embedded binaries (shim, crun, CNI plugins) before
go:embed, decompress at extraction time via klauspost/compress/zstd.
Phase 3: Split into online/offline build variants via build tags.
Online (default): CoreDNS + pause embedded, Portainer Agent and
local-path-provisioner pulled at runtime via client.Pull() fallback.
Offline (-tags offline): All OCI images embedded for air-gapped
deployments.
Estimated savings: ~160-180MB for the online variant.
Ref: nova8os#103
aa2a92b to
5c4589d
Compare
Description
Reduces KubeSolo binary size by ~160–180 MB for the default (online) build variant through three changes:
Phase 1: Replace runc with crun
Replace runc (~10 MB) with crun (~1 MB) as the OCI runtime. crun is a lightweight C-based runtime that is fully OCI-compliant and used as the default in Podman/Buildah. It supports cgroupv2 natively.
Note: crun does not publish pre-built 32-bit ARM (
armhf) binaries. Thedownload-deps.shscript will fail early with a clear error forGOARCH=arm.Phase 2: zstd-compress embedded binaries
Compress
containerd-shim-runc-v2,crun, and CNI plugin binaries withzstd -19beforego:embed. Binaries are decompressed at extraction time usinggithub.com/klauspost/compress/zstd(already an indirect dependency, promoted to direct).Phase 3: Online/offline build variants via build tags
Split into two build modes:
client.Pull()-tags offlineThe
importImage()function now falls back toclient.Pull()when an embedded image file is empty or missing, making the online variant seamless.Build Commands
Files Changed
Makefilebuild-offlineanddeps-offlinetargetsbuild/download-deps.sh--offlineflagcmd/kubesolo/main.goRuncBinaryFile→CrunBinaryFilego.modklauspost/compressto direct dependencyinternal/core/embedded/embedded.go.zstsuffixes, remove always-embedded optional imagesinternal/core/embedded/embedded_riscv64.go.zstchanges for riscv64internal/core/embedded/embedded_images_offline.go//go:build offlineinternal/core/embedded/embedded_images_offline_riscv64.gointernal/core/embedded/embedded_images_online.gointernal/core/embedded/load.goruncBinary→crunBinaryinternal/runtime/filesystem/binary.goExtractBinary()pkg/runtime/containerd/config.gorunc→crun, cgroupv2 detectionpkg/runtime/containerd/image.goclient.Pull()for missing imagespkg/runtime/containerd/service.goruncBinaryFile→crunBinaryFiletypes/types.goRuncBinaryFile→CrunBinaryFileTesting