Questions on the inference code

[ld-lab-pulsia]

Hello, thank you for sharing all this, I had a few questions about the inference.

1 -> Is it normal that the prompt use all this indentation in the docstring? Did you really use this prompt in the training or is there a missing textwrap.dedent?

2 -> max_model_len=32768 max_tokens=8192 -> Maybe you want to reduce 32768 as your prompt is not all thaty big, it could save cache space?

3 - > Could you explain why you run multiprocessing over vllm? Won't it duplicate the model? Usually the standard is rather to use a separate server and to do asynchronous calls.

4 -> limit_mm_per_prompt={"image": 50}, -> does that mean you intend to run the model over multiple pages simultaneously?

I am trying to integrate your model in this inference library: vlmparse, and I want to make sure I am doing it right.

[CherryLemon]

1 -> I believe this is how the model was trained originally, maybe just for pretty(indentation make it looks better), changing the prompt might suffer from performance drop.
2 -> Images count towards input token usage, so max_model_len needs to be large enough to hold all tokens from your input image + associated prompts, especially for large image.
3 -> The 2B model is CPU-heavy on powerful GPUs like H800: multiprocessing will lift throughput, but it's entirely optional. Setting the process count to 1 works perfectly fine for most casual use cases.
4 -> I don't think this can be used in multi-image according to the prompt, so set to {"image": 1} is enough.

[ld-lab-pulsia]

Thanks