feat(hexagon): implement Q4_1 support for MUL_MAT ops#56
feat(hexagon): implement Q4_1 support for MUL_MAT ops#56max-krasnyansky wants to merge 4 commits into
Conversation
- Add HTP_TYPE_Q4_1 and HTP_TYPE_Q8_1 definitions to htp-ops.h - Implement repacking logic for Q4_1 into Q4_1x4x2 format in ggml-hexagon.cpp - Implement dynamic Q8_1 quantization for F32 activations with block sums in matmul-ops.c - Add vec_dot kernels for Q4_1 weights and Q8_1 activations in matmul-ops.c - Add Q4_1 dequantizers and support for HMX fallback in hmx-matmul-ops.c Co-authored-by: max-krasnyansky <1380796+max-krasnyansky@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
- Add HTP_TYPE_Q4_1 and HTP_TYPE_Q8_1 definitions to htp-ops.h - Implement repacking logic for Q4_1 into Q4_1x4x2 format in ggml-hexagon.cpp - Implement dynamic Q8_1 quantization for F32 activations with block sums in matmul-ops.c - Add vec_dot kernels for Q4_1 weights and Q8_1 activations in matmul-ops.c - Add Q4_1 dequantizers and support for HMX fallback in hmx-matmul-ops.c Co-authored-by: max-krasnyansky <1380796+max-krasnyansky@users.noreply.github.com>
- Add HTP_TYPE_Q4_1 and HTP_TYPE_Q8_1 definitions to htp-ops.h - Implement repacking logic for Q4_1 into Q4_1x4x2 format in ggml-hexagon.cpp - Implement dynamic Q8_1 quantization for F32 activations with block sums in matmul-ops.c - Add vec_dot kernels for Q4_1 weights and Q8_1 activations in matmul-ops.c - Add Q4_1 dequantizers and support for HMX fallback in hmx-matmul-ops.c Co-authored-by: max-krasnyansky <1380796+max-krasnyansky@users.noreply.github.com>
- Add HTP_TYPE_Q4_1 and HTP_TYPE_Q8_1 definitions to htp-ops.h - Implement repacking logic for Q4_1 into Q4_1x4x2 format in ggml-hexagon.cpp - Implement dynamic Q8_1 quantization for F32 activations with block sums in matmul-ops.c - Add vec_dot kernels for Q4_1 weights and Q8_1 activations in matmul-ops.c - Add Q4_1 dequantizers and support for HMX fallback in hmx-matmul-ops.c Co-authored-by: max-krasnyansky <1380796+max-krasnyansky@users.noreply.github.com>
1 participant
This PR introduces full support for the Q4_1 data type in the Hexagon backend.
Key changes:
htp-ops.h: AddedHTP_TYPE_Q4_1andHTP_TYPE_Q8_1with their respective constants.ggml-hexagon.cpp: Added repacking logic for Q4_1 format usingrepack_row_q4x4x2_q4_1and unpacked functions.matmul-ops.c: Implemented Q8_1 dynamic quantization containing sums used for the affine min offset, plus the associatedvec_dotkernels for Q4_1x4x2 layout.hmx-matmul-ops.c: Extended HMX dequantization and matmul tasks to gracefully handle the additional min (m) scale values.PR created automatically by Jules for task 8647656785856433991 started by @max-krasnyansky