feat: add init and upload in kernel-builder cli by drbh · Pull Request #378 · huggingface/kernels

drbh · 2026-03-18T17:44:45Z

This PR is a WIP to add the init command to the kernel-builder rust cli. This branch mainly ports the init command from the python cli, with some improvements to the interface and updates to the template

This PR adds an the init and upload command to the rust kernel-builder cli similar to implementation in the current python cli. It also adds a build command that is a facade for the nix run so devs can interact with a single tool for the full creation lifecycle of a kernel.

Mainly it aims to have better defaults to make the commands easier to use

the init command only requires the name of the kernels (folder) and the owner is pulled via a whoami request if the user is already logged into hf
the init command creates a new dir if a name is specified, if called within a dir, the cli will use the dir name as the kernel name
the upload command does not require a repo id if the id can be read from the build
the upload command does not require a path to the build folder and will default to looking for the build dir if no explicit path is specified

example usage

init from inside of a existing dir

mkdir some-kernel
cd some-kernel
kernel-builder init
# Downloading template from kernels-community/template...
# Initialized `drbh/some-kernelt` at /home/drbh/Projects/kernels/kernel-builder/some-kernel

init from outside of a dir

kernel-builder init some-kernel
# Downloading template from kernels-community/template...
# Initialized `drbh/some-kernelt` at /home/drbh/Projects/kernels/kernel-builder/some-kernel

build

kernel-builder build
# ...
# some-kernel-torch-ext> no Makefile or custom installCheckPhase, doing nothing
# some-kernel-torch-ext> Checking of ABI compatibility
# some-kernel-torch-ext> 🐍 Checking for compatibility with manylinux_2_28 and Python ABI version 3.9
# some-kernel-torch-ext> ✅ No compatibility issues found
# some-kernel-torch-ext> Checking loading kernel with get_kernel
# some-kernel-torch-ext> Check whether the kernel can be loaded with get-kernel: some_kernel
# some-kernel-torch-ext> Running phase: removeBytecodeHook
# some-kernel-torch-ext> Removing Python bytecode

upload

kernel-builder upload
# Found 7 build variant(s) in /home/drbh/Projects/kernels/kernel-builder/some-kernel/build
# Using branch `v1` (new)
# Uploading 36 operations...
# Kernel upload successful. Find the kernel at: https://hf.co/drbh/some-kernel

sayakpaul · 2026-03-20T05:09:35Z

Some high-level thoughts first:

the init command only requires the name of the kernels (folder) and the owner is pulled via a whoami request if the user is already logged into hf

Should we somehow inform the users about it and take confirmation before proceeding?

# Kernel upload successful. Find the kernel at: https://hf.co/drbh/some-kernel

Should this be Find the kernel at: https://hf.co/drbh/some-kernel/tree/{version}? This way, users can directly use that link to inspect files, etc.

sayakpaul

Very minor comments. No RUST expertise yet.

sayakpaul · 2026-03-20T05:30:45Z

template/CARD.md

@@ -0,0 +1,38 @@
+---


Do we want to maintain two copies? We have one in https://github.com/huggingface/kernels/blob/main/kernels/src/kernels/cli/card_template.md

danieldk · 2026-03-20T12:41:42Z

kernel-builder/src/build.rs

+
+    /// Nix flake target to run.
+    #[arg(long, default_value = "build-and-copy")]
+    pub target: BuildTarget,


I think top-level

kernel-builder build kernel-builder build-and-copy kernel-builder build-and-upload

are nicer, easier to discover. I think we can use #[command(flatten)] to share arguments between different build commands, something like:

// ... Build { #[command(flatten)] pub common_build_args: CommonBuildArgs, // ... }

makes sense, updated in latest and added those three commands at the top level

danieldk · 2026-03-20T12:42:58Z

kernel-builder/src/build.rs

+
+    /// Additional arguments passed through to `nix run`.
+    #[arg(last = true)]
+    pub nix_args: Vec<String>,


Not sure if we want this?

yea I guess if the user wants more advance flags they should call nix run directly. removed in latest

danieldk · 2026-03-20T12:47:07Z

kernel-builder/src/init.rs

+                let bare_name = path.file_name().and_then(OsStr::to_str).ok_or_else(|| {
+                    eyre::eyre!("Cannot determine directory name from `{path_str}`")
+                })?;
+                let owner = hf::whoami_username()?;


danieldk · 2026-03-20T12:51:08Z

kernel-builder/src/init.rs

+                for (from, to) in replacements {
+                    text = text.replace(from, to);
+                }
+                fs::write(&destination, text).wrap_err_with(|| {


I possible (not sure if it supports everything needed), I think it would be nice to use the fileset API that is also used for generating pyproject files. The nice thing is that it is atomic, if something fails while still preparing the output, nothing is written.

I think we would have to extend it with the notion of files that should not be overwritten (if a project already exists and you only want to add new files). But I think making it impossible to have an operation half-completed is a nice feature.

oh yea thats a nice improvement, updated to use the file set api in the latest changes. this is a better ux and pattern so we avoid getting into a strange state

danieldk · 2026-03-20T12:53:42Z

kernel-builder/src/init.rs

+        .wrap_err_with(|| format!("Cannot parse TOML in `{}`", build_toml_path.display()))?;
+
+    // Update [general].backends
+    if let Some(general) = document.get_mut("general").and_then(|v| v.as_table_mut()) {


I forget if we discussed this before, but it would be worthwhile putting jinja templates in the template, so that we can render everything editing TOML (but maybe parse + write to format).

This has the benefit that if we ever change the build.toml format, we don't have to change it both in the template and here in the code.

E.g., suppose that backends moves somewhere else, we have to change it in the template and here.

ah yes thats a great improvement. updated the kernel template to use jinja in the latest changes

danieldk · 2026-03-20T12:54:37Z

kernel-builder/src/init.rs

+    let mut impl_block = String::new();
+    for (i, (condition, device, ref_prefix)) in conditions.iter().enumerate() {
+        let directive = if i == 0 { "#if" } else { "#elif" };
+        impl_block.push_str(directive);
+        impl_block.push(' ');
+        impl_block.push_str(condition);
+        impl_block.push_str("\n  ops.impl(\"");
+        impl_block.push_str(func_name);
+        impl_block.push_str("\", ");
+        impl_block.push_str(device);
+        impl_block.push_str(", ");
+        impl_block.push_str(ref_prefix);
+        impl_block.push_str(func_name);
+        impl_block.push_str(");\n");
+    }


Could be a jinja template with if blocks.

updated to prefer jinja in latest

danieldk · 2026-03-20T12:55:14Z

kernel-builder/src/init.rs

+    let marker = "ops.def(\"";
+    let start = content.find(marker)? + marker.len();
+    let rest = &content[start..];
+    let end = rest.find('(')?;
+    Some(&rest[..end])
+}
+
+/// Replace `#if defined(CPU_KERNEL)...#endif` block with new content
+fn replace_ifdef_block(content: &str, replacement: &str) -> String {
+    const START_MARKER: &str = "#if defined(CPU_KERNEL)";
+    const END_MARKER: &str = "#endif";
+
+    let Some(start) = content.find(START_MARKER) else {
+        return content.to_owned();
+    };
+
+    let search_region = &content[start..];
+    let Some(end_offset) = search_region.find(END_MARKER) else {
+        return content.to_owned();
+    };
+
+    let end = start + end_offset + END_MARKER.len();
+
+    let mut result = String::with_capacity(content.len());
+    result.push_str(&content[..start]);
+    result.push_str(replacement);
+    result.push_str(&content[end..]);
+    result
+}


Would be easier to maintain with a jinja template and some if blocks I think.

updated to prefer jinja in latest

danieldk · 2026-03-20T12:56:05Z

kernel-builder/src/main.rs

+    /// Upload kernel build artifacts to the Hugging Face Hub.
+    Upload(UploadArgs),
+
+    #[command(hide = true)]


If I understand correctly, this removes the subcommand from help. Sounds bad? Leftover from testing?

removed in latest

danieldk

Reviewed the upload subcommand now as well. Really excited to get batched uploads!

danieldk · 2026-03-23T08:19:33Z

kernel-builder/src/upload.rs

+    #[arg(value_name = "KERNEL_DIR", default_value = ".")]
+    pub kernel_dir: PathBuf,


Only a small nit, I'd use

Suggested change

#[arg(value_name = "KERNEL_DIR", default_value = ".")]

pub kernel_dir: PathBuf,

#[arg(value_name = "KERNEL_DIR")]

pub kernel_dir: Option<PathBuf>,

It expresses the semantics a bit better, there is now a helper function to get the path from that. (No strong opinions though.)

agreed this is a more idiomatic, updated in latest

danieldk · 2026-03-23T08:24:36Z

kernel-builder/src/upload.rs

+    let mut operations: Vec<CommitOperation> = Vec::new();
+
+    collect_benchmark_ops(&kernel_dir, &existing_files, is_new_branch, &mut operations)?;
+    collect_readme_ops(&kernel_dir, &mut operations);


README should go to the main branch. I think probably the most elegant way is to make operations a HashMap<String, Vec<CommitOperation>> with the branch name as the key. That way we can have a nested loop to handle everything.

For later: we may want to do the card generation and upload in one go as part of upload, since we cannot do the card generation in the Nix sandbox iff we need to fetch something from the main repo to do it (depends a bit on whether we still need to enumerate all the build variants or whether that information is handled by the Hub itself through the new kernel type, if the remaining information in the card is relatively static, we can handle it during build as well).

Not for this PR though.

Some relevant internal discussions: https://huggingface.slack.com/archives/C090JN2P8NB/p1774236408982349

@drbh I think the most sensible thing for now would be to keep the card contents limited to what we can fetch from the build (after the build is successful). We will render capabilities, frameworks etc. on the Hub side upon each commit, anyway. So, the card could contain an example snippet, available function interfaces, etc. This is all implemented in Python CLI.

Totally okay if we want to do the segregation in a separate PR.

And then there is this other thing of always uploading the card to main revision.

danieldk · 2026-03-23T08:27:45Z

kernel-builder/src/upload.rs

+}
+
+// Collect benchmark file operations: add matching files, delete stale ones.
+fn collect_benchmark_ops(


Maybe we should call them commit_ops? I was initially a bit confused, because we already use the term ops for ops in the kernel itself. Adding the commit_ prefix may make it a bit clearer that we are referring to different ops here.

Suggested change

fn collect_benchmark_ops(

fn collect_benchmark_commit_ops(

danieldk · 2026-03-23T08:32:05Z

kernel-builder/src/upload.rs

+fn read_repo_id_from_build_toml(kernel_dir: &Path) -> Result<String> {
+    let build_toml_path = kernel_dir.join("build.toml");
+    let content = fs::read_to_string(&build_toml_path)
+        .wrap_err_with(|| format!("Cannot read `{}`", build_toml_path.display()))?;
+    let parsed: toml::Value = toml::from_str(&content)
+        .wrap_err_with(|| format!("Cannot parse `{}`", build_toml_path.display()))?;
+
+    parsed
+        .get("general")
+        .and_then(|g| g.get("hub"))
+        .and_then(|h| h.get("repo-id"))
+        .and_then(|v| v.as_str())
+        .filter(|s| !s.is_empty())
+        .map(|s| s.to_owned())
+        .ok_or_else(|| {
+            eyre::eyre!(
+                "No `general.hub.repo-id` in `{}`. Use --repo-id to specify it.",
+                build_toml_path.display()
+            )
+        })
+}


Use util::parse_build. We could also implement a fn repo_id(&self) -> Option<&str>) on Build if it is helpful and then we probably do not need this function at all.

yea thats much cleaner, updated in latest! thanks

danieldk · 2026-03-23T08:34:08Z

kernel-builder/src/upload.rs

+    for variant in variants {
+        let metadata_path = variant.join("metadata.json");
+        let data = fs::read_to_string(&metadata_path)
+            .wrap_err_with(|| format!("Cannot read `{}`", metadata_path.display()))?;
+        let parsed: serde_json::Value = serde_json::from_str(&data)
+            .wrap_err_with(|| format!("Cannot parse `{}`", metadata_path.display()))?;
+        let version = parsed.get("version").and_then(|v| v.as_u64());
+        versions.insert(version);
+    }
+
+    if versions.len() > 1 {
+        let version_strs: Vec<String> = versions
+            .iter()
+            .map(|v| match v {
+                Some(n) => n.to_string(),
+                None => "none".to_owned(),
+            })
+            .collect();
+        bail!(
+            "Found multiple versions in build variants: {}",
+            version_strs.join(", ")
+        );
+    }


Use pyproject::Metadata to deserialize metadata.json. If helpful, we could add a function similar to parse_build.

nice point, added a parse_metadata function in latest commits.

danieldk · 2026-03-23T08:43:40Z

kernel-builder/src/upload.rs

+fn walk_recursive(dir: &Path, files: &mut Vec<PathBuf>) -> Result<()> {
+    for entry in
+        fs::read_dir(dir).wrap_err_with(|| format!("Cannot read directory `{}`", dir.display()))?
+    {
+        let entry = entry?;
+        let path = entry.path();
+        if path.is_dir() {
+            walk_recursive(&path, files)?;
+        } else {
+            files.push(path);
+        }
+    }
+    Ok(())
+}


is_dir follows symbolic links, so if a build directory contains a link like ln -s . foo, this will get stuck in an infinite loop (until it runs out of stack or heap). So we either need to skip symbolic links or we need loop detection. Maybe it's worth considering BurntSushi's walkdir, since it covers a lot of edge cases and doesn't follow symlinks by default. It also returns an iterator, which is more efficient than pushing everything into a Vec. Not that we care too much about performance here, but it is a nice property.

https://docs.rs/walkdir/latest/walkdir/

very good catch, updated to use walkdir in latest. thanks for pointing out

danieldk

Thanks! Added some more smaller comments.

danieldk · 2026-03-25T16:11:41Z

kernel-builder/src/pyproject/card.rs

+    let content = fs::read_to_string(&init_path).ok()?;
+
+    let re = Regex::new(r#"__all__\s*=\s*\[\s*([^\]]*)\s*\]"#).ok()?;
+    let list_content = re.captures(&content)?.get(1)?.as_str();


Regex parsing is really fragile. Not for this PR, but we should use a Python AST parser. Maybe tree-sitter-python or whatever ruff/ty uses (if it's available from crates.io). Maybe it's worthwhile creating a ticket?

good point, I opt'ed to use https://crates.io/crates/rustpython-parser which is what ruff forked, only because ruff's parser has git dependencies and our use case is relatively simple

danieldk · 2026-03-25T16:14:41Z

kernel-builder/src/build.rs

+    /// Maximum number of Nix build jobs.
+    #[arg(long, default_value = "4")]
+    pub max_jobs: u32,
+
+    /// Number of CPU cores per build job.
+    #[arg(long, default_value = "4")]
+    pub cores: u32,


Should use Option, we want to use the default from nix.conf when the user has not specified any arguments. E.g. I have these turned on my builder for the number of cores/RAM available.

good point, agree that option is a better pattern (as noted earlier) thanks for catching. updated to prefer the NixArgs struct where these values are optional

danieldk · 2026-03-25T16:16:57Z

kernel-builder/src/main.rs

 use develop::{devshell, testshell};

+mod build;
+use build::{run_build, CommonBuildArgs};


Suggested change

use build::{run_build, CommonBuildArgs};

use build::run_build;

removed in latest changes

danieldk · 2026-03-25T16:17:03Z

kernel-builder/src/main.rs

+    /// Build the kernel locally (alias for build-and-copy).
+    Build {
+        #[command(flatten)]
+        args: CommonBuildArgs,


Use

kernels/kernel-builder/src/main.rs

Line 26 in 826d536

struct NixArgs {

nice! updated to use this struct

danieldk · 2026-03-25T16:19:28Z

kernel-builder/src/main.rs

+            repo_type,
+        } => {
+            let kernel_dir = build_args.kernel_dir.clone();
+            run_build(build_args, "build-and-copy")?;


Suggested change

run_build(build_args, "build-and-copy")?;

Remove, the idea behind build-and-upload is to upload directly from the Nix store without using duplicate storage for both the store version and the build version.

good point, updated to only call build then upload from the symlinked result dir

danieldk · 2026-03-25T16:20:35Z

kernel-builder/src/main.rs

+        Commands::Init(args) => run_init(args),
+        Commands::Upload(args) => run_upload(args),
+        Commands::Build { args } | Commands::BuildAndCopy { args } => {
+            run_build(args, "build-and-copy")


I think I would only do a nix build for this subcommand. This unnecessarily creates another copy, which can be very large (e.g. FA) and the kernels tools work with the result directory/symlink as well (at least kernels upload).

makes sense, updated in latest

danieldk · 2026-03-25T16:22:11Z

kernel-builder/src/upload.rs

+        .filter_map(|e| e.ok())
+        .filter(|e| e.file_type().is_file())
+        .map(|e| e.into_path())
+}


We should probably copy the upload test from the Python tests?

agreed, copied in latest changes

sayakpaul

I left some comments regarding upload and the card components. Additionally:

Can the users optionally specify revision and repo_id in upload? I guess we always upload to a revision during our uploads (performed after the build) to respect our versioning system.
I see we have init_e2e.rs tests. But I think we should also have tests for upload and the card generation bits, preferably mimicking how we do it in the current Python CLI?
Do we want to update the docs in a separate PR?
I think we should require users to have hf-xet installed to benefit from fast uploads and downloads if we're not doing in here already?

drbh temporarily deployed to testpypi March 19, 2026 15:24 — with GitHub Actions Inactive

drbh changed the title ~~feat: support init in builder cli~~ feat: add init and upload in kernel-builder cli Mar 19, 2026

drbh force-pushed the kernel-builder-init-command branch from 534af0b to 368adad Compare March 19, 2026 15:25

drbh temporarily deployed to testpypi March 19, 2026 15:25 — with GitHub Actions Inactive

drbh temporarily deployed to testpypi March 19, 2026 15:38 — with GitHub Actions Inactive

drbh temporarily deployed to testpypi March 19, 2026 15:53 — with GitHub Actions Inactive

drbh temporarily deployed to testpypi March 19, 2026 16:32 — with GitHub Actions Inactive

drbh marked this pull request as ready for review March 19, 2026 16:42

drbh temporarily deployed to testpypi March 19, 2026 16:54 — with GitHub Actions Inactive

sayakpaul reviewed Mar 20, 2026

View reviewed changes

danieldk reviewed Mar 20, 2026

View reviewed changes

danieldk reviewed Mar 23, 2026

View reviewed changes

drbh temporarily deployed to testpypi March 23, 2026 21:13 — with GitHub Actions Inactive

drbh force-pushed the kernel-builder-init-command branch from 57523a2 to f88776a Compare March 23, 2026 21:19

drbh temporarily deployed to testpypi March 23, 2026 21:20 — with GitHub Actions Inactive

drbh temporarily deployed to testpypi March 23, 2026 21:46 — with GitHub Actions Inactive

drbh temporarily deployed to testpypi March 23, 2026 21:53 — with GitHub Actions Inactive

drbh temporarily deployed to testpypi March 23, 2026 22:03 — with GitHub Actions Inactive

drbh marked this pull request as draft March 23, 2026 22:53

drbh temporarily deployed to testpypi March 24, 2026 19:20 — with GitHub Actions Inactive

drbh force-pushed the kernel-builder-init-command branch from a50eff0 to f6b9889 Compare March 24, 2026 19:25

drbh temporarily deployed to testpypi March 24, 2026 19:25 — with GitHub Actions Inactive

drbh temporarily deployed to testpypi March 24, 2026 20:29 — with GitHub Actions Inactive

drbh marked this pull request as ready for review March 24, 2026 21:05

danieldk reviewed Mar 25, 2026

View reviewed changes

drbh added 3 commits March 25, 2026 13:51

feat: support init in builder cli

4ae6fc4

feat: add upload support

5d35efa

fix: use cwd when init called within a dir

136d607

drbh added 17 commits March 25, 2026 13:51

feat: add build that wraps nix commands

9e4d2d5

fix: clippy clean up

60d2e22

fix: update minijinja apis

8b48e48

fix: lock rust dep hashes

0961e87

fix: improve getting started

b6baf09

feat: prefer jinja templates

e3e824f

feat: prefer fileset api

06e62eb

feat: simplify build commands

cd513dc

fix: improve upload kernel path resolution

2c90666

feat: refactor and cleanup upload logic

ff8466c

fix: bump hub dep commit and pin it

e262bd5

fix: update nix for template files

24b3396

feat: write card when building

fa82c7d

fix: fix metadata upload struct bug

00aa116

feat: prefer sync upload api

f1df972

feat: prefer nix module in build command

f7fb291

feat: support uploads to kernel repos

63dbeb1

drbh force-pushed the kernel-builder-init-command branch from 3ed8a0f to 63dbeb1 Compare March 25, 2026 18:52

drbh temporarily deployed to testpypi March 25, 2026 18:52 — with GitHub Actions Inactive

feat: improve build commands, parse ast for card and upload tests

b08573a

drbh temporarily deployed to testpypi March 25, 2026 18:53 — with GitHub Actions Inactive

fix: format tests

4b4b3fa

drbh temporarily deployed to testpypi March 25, 2026 18:56 — with GitHub Actions Inactive

sayakpaul reviewed Mar 26, 2026

View reviewed changes

sayakpaul mentioned this pull request Mar 26, 2026

README utility for main branches on the build repos on Hub #315

Closed

		#[arg(value_name = "KERNEL_DIR", default_value = ".")]
		pub kernel_dir: PathBuf,

	use build::{run_build, CommonBuildArgs};
	use build::run_build;

Conversation

drbh commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danieldk Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danieldk Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danieldk Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danieldk Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danieldk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

drbh commented Mar 18, 2026 •

edited

Loading

sayakpaul commented Mar 20, 2026 •

edited

Loading

danieldk Mar 20, 2026 •

edited

Loading

danieldk Mar 20, 2026 •

edited

Loading

danieldk Mar 20, 2026 •

edited

Loading

danieldk Mar 20, 2026 •

edited

Loading