Unsure how `arg` is meant to be used

[damonge]

I have a pipeline stage that looks like this:

  filter_PL:
    exec: python3
    script: pipeline/filterer_mcut
    globals:
      - global
    options:
      num_sims: 10
      m_cut: 30
      sim_sorter: pl_sim_names
    arg:
      - 0
      - 10
      - 20

When running through mbatch, mbatch itself fails with a rather obscure error (see below). It seems I can only get this to work if arg is a string or a list of strings. Is it never meant to be integers?

The error message is:

  File "/home/damonge/.local/bin/mbatch", line 33, in <module>
    sys.exit(load_entry_point('mbatch==0+untagged.56.g41f6fbf', 'console_scripts', 'mbatch')())
  File "/home/damonge/.local/lib/python3.10/site-packages/mbatch-0+untagged.56.g41f6fbf-py3.10.egg/mbatch/mbatch.py", line 575, in main
    if len(shlex.split(arg))>1: raise Exception("Argument should not be interpretable as multiple arguments.")
  File "/usr/lib/python3.10/shlex.py", line 315, in split
    return list(lex)
  File "/usr/lib/python3.10/shlex.py", line 300, in __next__
    token = self.get_token()
  File "/usr/lib/python3.10/shlex.py", line 109, in get_token
    raw = self.read_token()
  File "/usr/lib/python3.10/shlex.py", line 140, in read_token
    nextchar = self.instream.read(1)
AttributeError: 'int' object has no attribute 'read'. Did you mean: 'real'?

[damonge]

I guess, associated with the title of this issue, I thought all outputs of a "loop" stage would be saved in the same folder, but it seems that's not the case (i.e. each loop creates its own folder with the name stage_name_arg1 etc.).

I had assumed the role of arg was to serve to e.g. loop over many simulations in separate parallel runs. I'm guessing that's not the case, right? If so, is this a use case that would be worth supporting?

[damonge]

OK, I think I understand. The logic is that such loops should be handled at the MPI level within each stage.
It still may be worth documenting that arg is meant to be a string only.