Depending on a subset of modules in a git repo

[JackBister]

Hello! I am in a situation where I have a git repository containing a fairly large number of modules, structured like this

buf.yaml
accounts/v1/accounts.proto
banners/v1/banners.proto
users/v1/users.proto
... etc.

with buf.yaml containing the following

version: 2
modules:
  - path: accounts/v1
  - path: banners/v1
  - path: users/v1
...

and different clients depend on different subsets of these modules. I would prefer to avoid compiling all modules in all the different client builds since it is wasteful and slow. My solution originally was to include each module as its own input in buf.gen.yaml like this:

inputs:
  - git_repo: https://...
    subdir: accounts/v1
  - git_repo: https://...
    subdir: banners/v1

While this works fine, it still ends up being slow. As I understand it, the repository is cloned once for each input so git operations end up taking most of the time when calling buf generate.

Another option I have been testing is to do the git clone outside of buf and then use directory inputs to reference the cloned repository, but I would prefer not to go this route since it adds an extra step before buf generate that developers have to be aware of.

So my question is, have I missed some other way of accomplishing this without cloning the repo multiple times and without compiling all modules in the repo?

Thank you in advance!

[doriable]

Thank you for the detailed issue, and yes, the slowness is due to the fact that each input is treated independently, and thus would be cloned separately. I've outlined a couple of options below:

Option 1: push to the BSR

This is the most straightforward, and a single push action from the repository containing all the modules would be sufficient to publish all of them. And then from there, clients can have the option of using generated SDKs or reference the modules in their buf.gen.yamls, which would be much faster.

That being said, I understand that there are situations where pushing to the BSR might not be an option, so I might have a solution for you.

Option 2: use the paths key in buf.gen.yaml for inputs

So each input in buf.gen.yaml has access to the paths key, which allows you to provide a list of paths for that input. This is equivalent to using the --path flag in-line with buf generate.

However, this is a little bit complex, because the intention of the paths key/--path flag is to filter paths within a module, and cannot be used to target a module's root. So you cannot provide something like:

inputs:
  - git_repo: https://...
    paths:
      - accounts/v1
      - banners/v1
      - ...

You'll end up with an error like:

Failure: module "accounts/v1" was specified with --path, specify this module path directly as an input

So you'll need to configure your buf.gen.yaml such that you're targeting subsets of modules. But basically, if you want to target multiple paths within a single input, the paths key is available.