Added install pkgs extension

[agyoungs]

I've been using this extension for a while and figured it was generic enough to be added to the main repo. If it's redundant in some way please let me know as I find this extension handy to add additional developer packages

[tfoote]

In general I'm not super enthusiastic about enabling arbitrary package strings here. In particular that's very platform specific and potentially version specific. Installing a large number of packages at startup time can be very costly as well. And this content isn't cached effectively.

At the first level I'd like to see something with an interface that is more generic and uses something like the rosdep keys to be cross platform. You mention in your comments about adding standard groupings. I think those standard groupings will be much more powerful for individuals, but they're going to be pretty user specific. I'd like to find a way to enable people to make those groupings and share them such that they can be batched up . An example is the --dev-helpers extension where those are my personal simple tools that I used as a test case. But that approach doesn't really scale well. If there's a group that wants to share and distribute an extension with that batching that gets to a level of management that might work. But also if you're looking to add a bunch of packages it's likely a better solution to build a new base image before using rocker. Raw packages are not what rocker is designed for, they don't require being responsive to the local environment.

[tfoote]

This should at least validate the environment here is debian if this will only work for debian packages.

[agyoungs]

Does validate_environment get called anywhere? It seems like it's not used unless I'm missing something

[tfoote]

Well that's a regression. It should be being validated at the build stage. It looks like that was lost along the way.

[tfoote]

This template only works for debian like systems. It should be documented.

[agyoungs]

Raw packages are not what rocker is designed for, they don't require being responsive to the local environment.

@tfoote I'm just now working through a backlog of things on my personal todo list and I wanted to come back to this. I understand that rocker isn't designed to handle raw packages, but would you be opposed to it if care was put into how it's implemented?

I have a workflow where it's useful to be able to quickly setup a dev/test environment (sometimes in ROS, but usually at least based on a repo or set of repos for a given project) in a target environment. Many times a docker image doesn't exist for that project (and it's not a project I control), so I'm stuck building things from scratch on my end. I've definitely abused the intent/design of Rocker by modifying it personally to suit these needs, but it's been very effective when I'm testing in these different environments. I find it faster than creating a new docker image for every iteration where I realize I'm missing something. In the long run I always push for setting up CI jobs to setup a docker container registry, but I do think a first line tool like this is useful.

I'm thinking that it may be useful to have extensions be able to advertise packages they want to install that will let core make more intelligent decisions about when to allow that installation. Or potentially this extension can be a special case where core handles the installation of other packages at a specific time, allowing the cache to work effectively if other runtime options are changed around.

I do like the idea of having defined package groups (but I also like the flexibility of specifying raw packages), but I'm not really sure the best way to have that maintained. I suppose implementing in such a way that's similar to how the extensions work could be serviceable. Other people/organizations could maintain their own repos containing their mapping of useful packages and add a link on the README if they care to share publicly.

[tfoote]

There's definitely some potential here. I've used custom extensions to do this idea of grouping and asserting specific packages are pre installed.

I could see a core capability which is a package manager interface. And individual extensions could request that the package manager assert those packages are available in the image. This could then be collapsed into a single layer as you mentioned.

It would be even better if the packages were resolved through the rosdep database as well so that you could get it on whatever platform you're running on and arguments wouldn't be target platform specific. Thus the extension or user just asks for rosdep keys and it resolves onto the target platform.

[agyoungs]

@tfoote I've given it some more thought. Specifically, I'm interested in incorporating changes I've made to make installing packages and setting up ROS (both ROS1 and ROS2) workspaces seamlessly. I'll focus on those use cases in this comment, but I do think we should aim to capture a more generally useful workflow if possible. At the end of the day, this boils down to templating common tasks in Dockerfiles to provide convenience to the end user. Rocker of course is built to solve a different problem, but it does have all the required functionality for templating Dockerfiles built in. So I see two possibly good approaches:

1. Create a completely separate tool that creates these docker containers off base images. For now it could simply handle the use cases I've mentioned (package installation and ROS workspace setup). If I were to develop this tool, I would probably heavily re-use some of the core components from Rocker since it already does the kind of necessary templating efficiently.
2. Take the approach we've lightly been talking about here where we create some higher level package manager and generic task groups. Maybe it doesn't always make sense to do all the installations in one place for instance. Maybe there are dependencies that should all be installed as early as possible, but some specific environment packages that should be installed at a later time to try to encourage cache re-use if environment level flags are changed. And maybe the same applies to other kinds of docker setup tasks. Maybe there are some tasks that should be done as early as possible while others that are more likely to be swapped out should be done later. The mechanism to handle this already exists with root and user snippets, but perhaps there are other circumstances where this is useful.

Anyway, I think option 1 fits the intended use case of Rocker better. Rocker wasn't designed to be a Dockerfile generator. For instance I could see this tool being useful if you wanted to test a ROS workspace out on several versions of ROS. You could simply have this tool build the workspace and install all required deps (and even specify the entrypoint to start your software). This tool might even be useful in CI scripts that are simply building containers for supporting a workspace in several ROS versions. Then you can use Rocker if you need to run things on your local machine on a GPU and another machine without a GPU for instance.

However, I do think there are a few things that are coupled between option 1 and 2. For instance I think it might be nice to have the option to mount the workspace or build it in user space. Some of these cases probably lend themselves better to having everything in one tool (Rocker). I think the tool from option one will require a lot of code re-use from Rocker anyway. If we implement it all in Rocker, CI scripts could still use Rocker to build their ROS workspace docker containers, and allow end-users to use Rocker to set those generated images up for their environment as they see fit. You could potentially even just use Rocker to spit out the Dockerfile you want for building a ROS workspace.

Anyway, all this to say I already get heavy use out of the ROS ws extension and package installation extension, but they're definitely not production ready. I'd like your feedback on whether you think we can get this added to Rocker as I've outlined in option 2, or if you think another tool is going to be the way to go. I'm happy to do the work to refactor things as needed and put together a PR, but I just want to make sure it's not wasted effort if what I'm proposing is going to be too much of a scope shift for Rocker.

Side note: I haven't given too much thought about handling the rosdeps. In my current ros-ws-extension I just have a separate process lookup all the rosdeps and create an installation command using apt and pip as needed. However, I think it would be more effective to handle another way.

[tfoote]

I think that there's a combination of your two approaches. I think that the tool/capability to add packages generically potentially via rosdep keys would be very helpful. That would automatically generate snippets like this: https://github.com/osrf/rocker/blob/main/src/rocker/templates/rmw_snippet.Dockerfile.em but programmatically and potentially have more flexibility. And it could be reused like case inside other plugins.

We can provide a direct command line interface too, but probably not recommend it.

However I think that your case 1 is also a potentially valuable tool. Especially for CI and other things it would be great to have a tool that can create and publish images automatically. What I think would be really cool for this image to do would be to embed metadata into the container which encodes the plugins that are built into the image. Such that if you built an image with --rmw zenoh it would have built the image and installed the zenoh packages. But it wouldn't try to do the runtime settings it would just register that rocker::rmw=fastrtps has been built in. And then when that image is invoked by rocker it will be able to introspect which plugins are built into the image and then invoke the appropriate runtime arguments without needing to pass them on the command line a second time.

If we go for this sort of thing it would make sense to probably build that as a separate entry point inside of rocker.

There's some fun things that you'll have to register plugins as able to be preinstalled or not. For example the --user plugin has to be run on the runtime machine, otherwise the usernames won't match. But it could still be registered with the container's metadata so that the rocker invocation doesn't need to pass the argument.

I think that Labels provide the metadata storage we need for this: https://docs.docker.com/engine/manage-resources/labels/