Packing support in mcore-adapter VL models

[JCode012]

I see vl models in mcore-adapter currently does not seem to support packing. Are there any plans to support it in the future?

[Kuangdd01]

Could you clarify why you think packing isn't supported?

[JCode012]

@Kuangdd01
具体应该是 CP + Packing 的数据处理逻辑有点疑问，以 qwen3vl 为例。
这里没传 packed_seq_params，感觉没法正确拿到 thd 格式的 cp_batch，我看llm是传了的。

def get_batch_on_this_cp_rank(self, batch, dim3_keys: List[str] = ["attention_mask"]):
    # VLM need to view all input_ids and media features
    loss_needed_items = {
        "labels": batch.pop("labels", None),
    }
    loss_needed_items = super().get_batch_on_this_cp_rank(loss_needed_items, dim3_keys=dim3_keys)
    batch.update(loss_needed_items)
    return batch

get_rope_index 和 get_input_ranges 函数和也没考虑 packed_seq_params。

还有这里针对 mrope，我看 mcore016 传进来的是 packed_seq_params.cp_group，但是 adapter 里定义的 PackedSeqParams 好像没对 cp_group 属性赋值。

def forward(
    self,
    position_ids: torch.Tensor,
    mrope_section: list[int],
    cp_group: Optional[torch.distributed.ProcessGroup] = None,
) -> torch.Tensor:
    """Forward pass of multimodal RoPE embedding.
    Args:
        position_ids (torch.Tensor): A position_id tensor with shape [3, batchsize, seqlens]
        mrope_section (list[int]): Multimodal rope section is for channel dimension of temporal,
            height and width in rope calculation.
    Returns:
        Tensor: Embeddings after applying RoPE.
    """
    seq = position_ids.to(device=self.inv_freq.device, dtype=self.inv_freq.dtype)

    if self.seq_len_interpolation_factor is not None:
        seq *= 1 / self.seq_len_interpolation_factor

    inv_freq_expanded = self.inv_freq[None, None, :, None].expand(3, seq.shape[1], -1, 1)  # shape (3, bs, dim, 1)
    seq_expanded = seq[:, :, None, :].float()  # shape (3, bs, 1, seq_length)
    freqs = (inv_freq_expanded @ seq_expanded).transpose(2, 3)  # shape (3, bs, seq_length, dim)
    freqs = self.apply_interleaved_mrope(freqs, mrope_section)  # shape (bs, seq_length, dim)

    # first part even vector components, second part odd vector components,
    #  2 * dim in dimension size
    # sin, sin, ..., cos, cos, ...
    emb = torch.cat((freqs, freqs), dim=-1)  # shape (bs, seq_length, 2 * dim)

    # shape (seq_length, bs, 1, 2 * dim)
    emb = emb[..., None, :].transpose(0, 1).contiguous()
    if cp_group is None:
        cp_group = self.cp_group
    if cp_group is not None and cp_group.size() > 1:
        # slice rotary_pos_emb along sequence dimension and select the parition of the current
        # CP rank
        emb = get_pos_emb_on_this_cp_rank(emb, 0, cp_group)
    return emb

有可能我看的不仔细，

[Kuangdd01]

不考虑neat_packing的话，我理解是trainer内部对vl模型只做labels的cp切分，forward里面再做其他key的切分。
考虑neat_packing的话，现在的trainer里的assertion非常严格，很难进入主循环得到PackedSeqParams。

还有这里针对 mrope，我看 mcore016 传进来的是 packed_seq_params.cp_group，但是 adapter 里定义的 PackedSeqParams 好像没对 cp_group 属性赋值。

这里可不可以看做cp_group已经赋值了

class Qwen3VLMultimodalRotaryEmbedding(nn.Module):
     def __init__(self, ..):
          ....
          self.cp_group = (
            cp_group if cp_group is not None else parallel_state.get_context_parallel_group(check_initialized=False)
        )