[rule based toolchain] Supporting exec tools as part of cc_args

[cerisier]

clang and gcc both can both delegate various part of the build to external tools. ld and as are the most popular.

In the case of CUDA, extra binaries such as ptxas come at play.

By default clang expects to find it inside a bin folder under --cuda-path=.
Or at a custom path defined by --ptxas-path=

The issue is that cc_args to pass --ptxas-path is evaluated under cc_toolchain.args which is target configured, whereas ptxas needs to match the platform resolved for the toolchain.

An option is to wrap those binaries into rule that reexports the binary provided as cfg = "exec" but from our own experience, exec cfg isn't guaranteed to match the resolved platform for the toolchain.

Arguably, the ability to pass a custom --ld-path would suffer from the same issue.

From my understanding, only the tool_map would be guaranteed to match the resolved platform for the toolchain, and that only after #555 lands.

Has this use case ever been considered ?

cc @dzbarsky and @fmeum with whom we have regular discussion about this.

[dzbarsky]

clang and gcc both can both delegate various part of the build to external tools. ld and as are the most popular.

In the case of CUDA, extra binaries such as ptxas come at play.

By default clang expects to find it inside a bin folder under --cuda-path=. Or at a custom path defined by --ptxas-path=

The issue is that cc_args to pass --ptxas-path is evaluated under cc_toolchain.args which is target configured, whereas ptxas needs to match the platform resolved for the toolchain.

An option is to wrap those binaries into rule that reexports the binary provided as cfg = "exec" but from our own experience, exec cfg isn't guaranteed to match the resolved platform for the toolchain.

Arguably, the ability to pass a custom --ld-path would suffer from the same issue.

From my understanding, only the tool_map would be guaranteed to match the resolved platform for the toolchain, and that only after #555 lands.

Has this use case ever been considered ?

cc @dzbarsky and @fmeum with whom we have regular discussion about this.

There's an in-flight PR adding env to cc_tool precisely for exec-configured deps, I think we need to also support args on cc_tool directly for the same reason, to handle this use case. @armandomontanez @trybka Does that sound right to yall?

[armandomontanez]

I'd expect this to come in some way through a cc_tool. In the very early days of the rule-based toolchains, you could bind arguments directly to tools. We moved away from that because having any flags come from tools ended up making ordering more difficult.

I expect as a user, folks might expect something like this to work: set PTXAS_PATH as an env var on the tool, then pass a flag like --ptxas-path=$PTXAS_PATH. That sounds like the easiest path forward here.

For a stricter solution, maybe it's time we pursue the creation and injection of custom toolchain variables.

[dzbarsky]

I'd expect this to come in some way through a cc_tool. In the very early days of the rule-based toolchains, you could bind arguments directly to tools. We moved away from that because having any flags come from tools ended up making ordering more difficult.

I expect as a user, folks might expect something like this to work: set PTXAS_PATH as an env var on the tool, then pass a flag like --ptxas-path=$PTXAS_PATH. That sounds like the easiest path forward here.

For a stricter solution, maybe it's time we pursue the creation and injection of custom toolchain variables.

I wonder if we supply format = {PTXAS: "@ptxas"}, data = ["@ptxas"] on the cc_tool and have those available to expansion on the args?