feat: configuration of project sources by victor-linroth-sensmetry · Pull Request #130 · sensmetry/sysand

victor-linroth-sensmetry · 2025-11-28T11:08:34Z

Main new features added by this PR:

Config options for project sources in sysand.toml (or given config). Allows for portable specification of local and remote sources.
Command line options --editable, --path, --url-src and --url-kpar for sysand add that automatically updates config file. These also get removed with sysand remove.
Documentation explaining the above.

In order to simplify implementing the above the core library was build out with the following changes:

New macro crate with ProjectRead and ProjectMut derive macros for easy enum composition of project structs.
New CachedProject<Local, Remote> struct for when the Local project is easier to access (e.g. on disk or in memory) vs Remote, but you still want .sources() to give the remote sources.
Implementation of CombinedResolver simplified using derive(ProjectRead) and CachedProject.
New ProjectReference<Project> struct. When working with ReadEnvironment the projects need to be clonable, but this might not always be a good idea, e.g. some project hold ownership of temp dirs. ProjectReference is basically and Arc wrapper for a project to be used as a compatibility layer.
New AnyProject struct combining the different projects supported for source configuration. Should be extended to include Git projects in future.
LocalSrcProject and LocalKparProject now has optional nominal paths in addition to the regular path. This comes from the choice to have configured source always be relative to the project root but this not always being available everywhere. (Could perhaps be simplified if we added a Global Context).

In addition there where some small changes that came about when experimenting with different solutions:

MemoryEnv can now be used with a generic project.
MemoryResolver can be conveniently constructed from iterator/array/Vec.

These ended up not being used here, but they could be useful at some later point so where left in.

Lastly, the verbose/quiet options were removed from the config. They were mostly placeholders to begin with and now that there are more real options present in the config they don't serve much of a purpose. Removing them also means that the logger can be initialized almost immediately on start, opening up the potential for logging more things, like e.g. the loading of the config files.

tilowiklundSensmetry · 2025-11-28T11:34:16Z

Is there a particular reason for expanding CombinedResolver with an Override case, as opposed to having a generic OverrideResolver<T> and use StandardResolver = OverrideResolver<CombinedResolver>?

victor-linroth-sensmetry · 2025-11-28T11:38:56Z

Is there a particular reason for expanding CombinedResolver with an Override case, as opposed to having a generic OverrideResolver<T> and use StandardResolver = OverrideResolver<CombinedResolver>?

No particular reason, just the first thing that came to mind. Do you think that would be a better solution?

tilowiklundSensmetry · 2025-11-28T12:03:55Z

Is there a particular reason for expanding CombinedResolver with an Override case, as opposed to having a generic OverrideResolver<T> and use StandardResolver = OverrideResolver<CombinedResolver>?

No particular reason, just the first thing that came to mind. Do you think that would be a better solution?

It might be easier to test/mock, and be a bit easier to reuse elsewhere. You can simply have OverrideResolver<Memory/Null> instead of having to create a CombinedResolver<Memory/Null, Memory/Null, Memory/Null, Override, Memory/Null>.

No very strong preference though, so don't bother if it's a lot of work to refactor.

tilowiklundSensmetry · 2025-11-28T12:05:29Z

Everything looks good to me in principle, would be a good to have a proper code review from @andrius-puksta-sensmetry before merging though.

victor-linroth-sensmetry · 2025-11-28T12:08:16Z

No very strong preference though, so don't bother if it's a lot of work to refactor.

Well, this was precisely the type of feedback I was looking for, so I'll definitely consider it.

Everything looks good to me in principle, would be a good to have a proper code review from @andrius-puksta-sensmetry before merging though.

It's still a draft so far. I want to add some quality of life command line stuff also.

victor-linroth-sensmetry · 2026-01-20T12:21:07Z

Currently sysand_core::lock::Source and typed_path::Utf8UnixPath have to be in scope when using the ProjectRead macro. This is certainly less than ideal but it's not obvious to me what the best solution would be.

The problem stem from the fact that reference of types in macros should be absolute but the absolute paths vary depending on if you are inside or outside the crate. E.g. it's ::sysand_core::lock::Source if outside and ::crate::lock::Source if inside. If typed_path::Utf8UnixPath is part of the public API it should probably be reexported by sysand_core and then it's the same problem there.

One could try to add them as associated types to ProjectRead but then you introduce the additional complexity that it can vary with trait implementations. I'm unsure if thats desirable or necessary.

Another solution would be to refactor the crates so that they are always used outside of the crate where they are exported. Would essentially mean that all the project stuff goes into it's own crate (except for AnyProject which is derived using the macro).

Edit: I think the appropriate course of action is to factor out the projects into their own crate, but this would be a pretty wide reaching change so best to do as a separate PR after this is merged.

victor-linroth-sensmetry · 2026-01-20T12:22:04Z

Unsure if core/tests/project_no_derive.rs has any value left now that I have it working.

victor-linroth-sensmetry · 2026-01-20T20:58:14Z

Still left to make sure Windows paths are handled properly (don't think they are at the moment).

Edit: Since the sources only have paths that are Utf8UnixPaths, I think this should be covered.

core/src/env/memory.rs

sysand/src/cli.rs

Signed-off-by: victor.linroth.sensmetry <victor.linroth@sensmetry.com>

sysand/src/commands/clone.rs

sysand/src/cli.rs

andrius-puksta-sensmetry · 2026-02-11T07:49:13Z

sysand/src/commands/info.rs

+                MemoryResolver::from(overrides),
+                standard_resolver(
+                    cwd,
+                    if local_env_path.is_dir() {


This is eating errors. Maybe we should add is_file() and is_dir() to wrapfs, but make them e.g. fn is_file() -> Result<bool, SomeError>? This is mostly a nice-to-have, so can be done later.

What error? You can run sysand info on an IRI without having an env.

sysand/src/commands/env.rs

andrius-puksta-sensmetry · 2026-02-11T09:23:15Z

sysand/src/commands/add.rs

+    Ok(src_path)
+}
+
+fn relativize(path: &Utf8Path, root: &Utf8Path) -> Utf8PathBuf {


Suggested change

fn relativize(path: &Utf8Path, root: &Utf8Path) -> Utf8PathBuf {

/// Create a relative path from `root` to `path`. Both of them must either

/// be absolute or otherwise start with the same prefix.

fn relativize(path: &Utf8Path, root: &Utf8Path) -> Utf8PathBuf {

Also maybe return Option<Utf8PathBuf> so that caller can decide what to do in case relative path cannot be constructed.

This function is useful (i.e. should be used, but currently isn't) in lots of places, so it should be moved to some utils file.

andrius-puksta-sensmetry · 2026-02-11T09:23:41Z

sysand/src/commands/add.rs

+    // If prefixes (e.g. C: vs D: on Windows) differ, no relative path is possible.
+    if path.components().next() != root.components().next() {
+        return path.to_path_buf();
+    }


Not sure what to do about Windows UNC vs non-UNC paths, as they always have different first components. Relevant here is that std's canonicalize returns UNC paths on Windows, but calling that on both inputs here would be kind of wasteful.

Windows root-relative paths are also not handled correctly here.

It may be a good idea after all to canonicalize both paths (or ensuring only canonical paths are given as args here) to avoid subtle bugs due to symlinks and Windows weirndess.

core/src/project/any.rs

core/src/resolve/combined.rs

andrius-puksta-sensmetry · 2026-02-11T13:20:37Z

docs/src/config/dependencies.md

+
+Sometimes you may wish to use a project that isn't resolvable through an
+available index or you want to override the dependency resolution for other
+reasons. In any case you can do this by adding the appropriate IRI and `Source`


Suggested change

reasons. In any case you can do this by adding the appropriate IRI and `Source`

reasons. You may also just want to give the usage a different identifier for readability. In any case you can do this by adding the appropriate IRI and `sources`

I'm not sure this is a great way of doing general aliasing, as then you may want to still want to go through regular dependency resolution. Maybe there's a way to add that as a separate feature in the future?

docs/src/config/dependencies.md

andrius-puksta-sensmetry · 2026-02-11T13:25:35Z

docs/src/config/dependencies.md

+file, but it is also possible to do through the command line interface with the
+[`sysand add`](../commands/add.md) command.
+
+## Local projects


Maybe mention the exact CLI flag to use for each type here?

docs/src/config/dependencies.md

andrius-puksta-sensmetry · 2026-02-11T13:36:14Z

docs/src/config/dependencies.md

+the following entry to your `sysand.toml`.
+
+```toml
+[[project]]


Maybe rename this to [[source]] to give us space to put the current project metadata here (if we ever decide to do this)?

Yeah, I thought about that a too, but naming it something else breaks the symmetry with lockfile format. There is potentially more configurations we want to add here like e.g. aliasing (having multiple identifiers) and disabling checksums (in case the project is expected to change a lot like editable projects and sources pointing to git branches). I think having the lockfile and config mirror each other here would be nice.

andrius-puksta-sensmetry · 2026-02-11T13:39:53Z

macros/src/lib.rs

+#[proc_macro_derive(ProjectRead)]
+pub fn project_read_derive(input: TokenStream) -> TokenStream {


Add a doc comment describing what this derive generates (i.e. what the derived impl does given an example arg).

andrius-puksta-sensmetry · 2026-02-11T13:40:35Z

macros/src/lib.rs

+#[proc_macro_derive(ProjectMut)]
+pub fn project_mut_derive(input: TokenStream) -> TokenStream {


Description also needed here

sysand/src/commands/add.rs

andrius-puksta-sensmetry · 2026-02-11T14:40:43Z

sysand/src/commands/add.rs

+    for r in root_iter {
+        if let Utf8Component::Normal(_) = r {
+            result.push("..");
+        }
+    }


This is not always correct due to symlinks, unless both paths are already canonical.

This will also return an empty path if path and base are the same.

And the paths will use platform-dependent separator: \ on Windows and / everywhere? else. Is config.toml intended to be portable?

This is not always correct due to symlinks, unless both paths are already canonical.

Guess it's easiest to just canonicalize them both.

This will also return an empty path if path and base are the same.

No it will be .? (Maybe some warning is appropriate here though since this doesn't make much sense for actual usage.)

And the paths will use platform-dependent separator: \ on Windows and / everywhere? else. Is config.toml intended to be portable?

I'm pretty sure all paths get transformed to Utf8UnixPaths before going into any file.

Signed-off-by: victor.linroth.sensmetry <victor.linroth@sensmetry.com>

victor-linroth-sensmetry force-pushed the feat/config-sources branch from 67ca989 to 4173804 Compare December 8, 2025 15:53

victor-linroth-sensmetry force-pushed the feat/config-sources branch from 4173804 to a418bd2 Compare December 19, 2025 10:33

victor-linroth-sensmetry changed the title ~~Feat/config sources~~ feat: configuration of project sources Dec 29, 2025

victor-linroth-sensmetry force-pushed the feat/config-sources branch from a418bd2 to bf73d16 Compare January 7, 2026 14:18

victor-linroth-sensmetry self-assigned this Jan 7, 2026

victor-linroth-sensmetry force-pushed the feat/config-sources branch from bd2ad17 to 0fa5369 Compare January 20, 2026 10:53

victor-linroth-sensmetry requested review from andrius-puksta-sensmetry, tilowiklundSensmetry and vytautas-astrauskas-sensmetry January 20, 2026 12:22

andrius-puksta-sensmetry mentioned this pull request Jan 27, 2026

feat: allow adding/removing usages by path #154

Open

tilowiklundSensmetry reviewed Jan 28, 2026

View reviewed changes

core/src/env/memory.rs Show resolved Hide resolved

tilowiklundSensmetry reviewed Jan 28, 2026

View reviewed changes

core/src/env/memory.rs Outdated Show resolved Hide resolved

victor-linroth-sensmetry force-pushed the feat/config-sources branch 2 times, most recently from 49299d9 to 7a835d1 Compare January 29, 2026 12:53

victor-linroth-sensmetry marked this pull request as ready for review January 29, 2026 12:55

victor-linroth-sensmetry mentioned this pull request Jan 30, 2026

feat(cli): rudimentary support for http authentication #165

Merged

andrius-puksta-sensmetry reviewed Feb 6, 2026

View reviewed changes

sysand/src/cli.rs Show resolved Hide resolved

victor-linroth-sensmetry added 5 commits February 9, 2026 13:49

Added derive macro for ProjectRead trait.

513ace0

Signed-off-by: victor.linroth.sensmetry <victor.linroth@sensmetry.com>

Added derive macro for ProjectMut trait.

fb04953

Signed-off-by: victor.linroth.sensmetry <victor.linroth@sensmetry.com>

Added generic CachedProject.

e3b98fa

Signed-off-by: victor.linroth.sensmetry <victor.linroth@sensmetry.com>

Make derive macros work both inside and outside of sysand_core.

22d8099

Signed-off-by: victor.linroth.sensmetry <victor.linroth@sensmetry.com>

Makes ProjectRead and ProjectMut derive macros support generics.

d14ca10

Signed-off-by: victor.linroth.sensmetry <victor.linroth@sensmetry.com>

andrius-puksta-sensmetry reviewed Feb 10, 2026

View reviewed changes

sysand/src/commands/clone.rs Outdated Show resolved Hide resolved

andrius-puksta-sensmetry reviewed Feb 10, 2026

View reviewed changes

sysand/src/cli.rs Outdated Show resolved Hide resolved