[quantization] Refactor: Remove float32 assumption and follow model dtype

[mhs4670go]

Background

The current PTQ pipeline assumes float32 as the default dtype for model execution and intermediate tensors. This assumption originates from legacy constraints in the CNN-era toolchain (e.g., ONE), where models were expected to be in float32 format.

However, this assumption is no longer suitable for modern Transformer/LLM workflows:

• Most pretrained models are in fp16 or bf16
• Converting to float32 increases memory usage significantly
• The dependency on float32 is not algorithmically required, but historically inherited

Problem

• Hardcoded float32 conversions (.float(), torch.float32) exist across the pipeline
• Model dtype is ignored even when explicitly defined in the checkpoint or load configuration
• This leads to:
  • Unnecessary memory overhead
  • Redundant dtype casts
  • Reduced flexibility for modern models

Proposal

Refactor the PTQ pipeline to remove the implicit float32 assumption and instead follow the original model dtype.

Key Changes

1. Remove float32 hardcoding

   • Eliminate .float() and torch.float32 assumptions in:
     • model wrapping
     • calibration
     • observer inputs
     • export paths

2. Adopt model dtype as default execution dtype

   • Use the actual model parameter dtype as the primary source of truth
   • Suggested precedence:
     1. Explicit user override
     2. Model parameter dtype
     3. Model config dtype
     4. Fallback to float32

3. Validate via regression

   • Run calibration, quantization, and evaluation flows
   • Compare against current float32-based baseline
   • Identify any numerical instability or accuracy degradation

4. Introduce selective fp32 promotion only if needed

   • If regression reveals instability, apply fp32 only to specific components:
     • observer statistics
     • scale/zero-point computation
     • error metrics
   • Avoid global dtype overrides

Expected Benefits

• Reduced memory footprint for large models
• Better alignment with modern LLM/VLM checkpoints
• Cleaner and more general PTQ design
• Decoupling from legacy backend assumptions (e.g., ONE)

[mhs4670go]

Enable Mixed-Precision Evaluation by Casting Around Fake Quantization

In the current PTQ evaluation pipeline, fake quantization is applied using PyTorch APIs such as:

• torch.fake_quantize_per_tensor_affine
• torch.fake_quantize_per_channel_affine

And, unfortunately, directly applying fake quantization in float16 is not supported by PyTorch and may lead to incorrect quantization behavior due to reduced numerical precision. These APIs are designed to operate on float32 inputs to ensure numerically accurate simulation of quantization behavior (e.g., rounding, clamping, and dequantization).

Related: #487

Proposal

Introduce a mixed-precision evaluation mode where:

• Model computation (node) runs in the original model dtype (e.g., float16 / bfloat16)
• Fake quantization is applied in float32 only

Conceptually:

node (fp16)
-> cast to fp32
-> fake_quant (fp32)
-> cast back to fp16
-> node (fp16)
-> ...

Expected Benefits

• Improved evaluation performance by leveraging lower-precision compute for most operations
• Better alignment with real deployment scenarios
• Retains correctness of quantization simulation by keeping fake quantization in float32

Considerations

• Casting overhead (fp16 <-> fp32) may reduce performance gains depending on graph structure
• Accuracy differences are expected to be minimal if the baseline model also runs in float16

Next Steps

• Implement utility functions or patch to wrap fake quantization with dtype casting
• Benchmark performance vs. current fp32 evaluation
• Compare accuracy against fp16 baseline model