Harden OFDM CFO path and add deterministic CFO verification tools

Author: secup

README.md

@@ -207,6 +207,23 @@ cd build
 ctest
 ```
 
+**CFO Verification (Internal Pre/Post Correction Chain):**
+
+```bash
+# From repository root (one-command gate)
+./tests/verify_cfo_chain.sh --cfo 50 --channel awgn --snr 20 --seed 42
+
+# Optional: capture raw TX/RX audio for offline analysis
+./build/cli_simulator --snr 20 --channel awgn --waveform ofdm_chirp \
+  --mod dqpsk --rate r1_2 --tx-cfo 50 \
+  --save-signals 1 --save-prefix /tmp/cfo_probe --test
+```
+
+Useful `cli_simulator` flags for CFO diagnostics:
+- `--tx-cfo` (alias `--cfo`) injects TX CFO in Hz across the full transmitted signal.
+- `--save-signals [N]` writes raw TX/RX captures for up to `N` messages.
+- `--save-prefix <path>` and `--save-max-samples <N>` control capture naming and size.
+
 **Manual Modulation Selection:**
 
 The `--mod` flag allows testing specific modulations:

docs/CFO_CORRECTION_FLOW.md

@@ -1,14 +1,14 @@
 # CFO Correction Flow - Complete Reference
 
-This document describes the complete CFO (Carrier Frequency Offset) correction flow, including the fading channel correction and feedback loop added 2026-02-03.
+This document describes the complete CFO (Carrier Frequency Offset) correction flow, including the fading-channel correction loop and the 2026-02-12 verification tooling updates.
 
 **Status:** Working and verified (100% CW on AWGN, 100% CW on good fading, ~83% on moderate)
 
 ---
 
 ## Overview
 
-CFO correction has THREE stages:
+CFO correction has THREE runtime stages:
 1. **Chirp-based coarse estimation** — during handshake (dual chirp preamble)
 2. **LTS-based residual correction** — per frame (training symbol phase comparison)
 3. **Pilot-based tracking** — per symbol (pilot carrier phase differences)
@@ -181,6 +181,61 @@ Frame N+1 processing:
 
 ---
 
+## Stage 5: Simulator TX CFO Injection and Internal Chain Proof
+
+The simulator now supports deterministic CFO stress testing and direct internal validation of the correction path.
+
+### TX CFO injection (simulator)
+
+**File:** `tools/cli_simulator.cpp`
+
+- `--tx-cfo <Hz>` (alias `--cfo`) injects a transmitter CFO shift in the simulator path.
+- Injection is applied with analytic-signal single-sideband shifting and per-direction phase continuity.
+- This avoids image artifacts and preserves realistic stream behavior for long transfers.
+
+Example:
+
+```bash
+./build/cli_simulator --snr 20 --channel awgn --waveform ofdm_chirp \
+  --mod dqpsk --rate r1_2 --tx-cfo 50 --seed 42 --test
+```
+
+### Internal pre/post correction dumps
+
+**File:** `src/ofdm/channel_equalizer.cpp` (inside `toBaseband()`)
+
+Set environment variables before running simulator:
+
+```bash
+ULTRA_DUMP_CFO_PREFIX=/tmp/cfo_chain_dump
+ULTRA_DUMP_CFO_CALLS=6
+```
+
+For each dump index `i`, the demod path writes:
+- `<prefix>_<i>_pre.cf32` (after mixer/downconversion, before CFO correction)
+- `<prefix>_<i>_post.cf32` (after CFO correction)
+- `<prefix>_<i>_meta.txt` (sample rate, CFO, phase info)
+
+### One-command verification harness
+
+**Files:**
+- `tests/verify_cfo_chain.sh`
+- `tools/verify_cfo_chain_dump.py`
+
+Run:
+
+```bash
+./tests/verify_cfo_chain.sh --cfo 50 --channel awgn --snr 20 --seed 42
+```
+
+The Python verifier estimates applied correction from:
+- `post * conj(pre)` phase slope
+
+Expected result:
+- Applied correction near `-expected_cfo` (within tolerance), for all dumped frames.
+
+---
+
 ## StreamingDecoder CFO Drift Limiting
 
 **File:** `src/gui/modem/streaming_decoder.cpp` (lines 425-440)
@@ -273,6 +328,9 @@ diff = eq_data × conj(ref) = TX_data × e^{-jφ} × conj(e^{-jφ}) = TX_data  
 | `src/waveform/ofdm_chirp_waveform.cpp` | `process()` CFO feedback, initial phase calculation |
 | `src/gui/modem/streaming_decoder.cpp` | `last_cfo_` caching, drift limiting, feedback update |
 | `src/ofdm/ofdm_sync.cpp` | `estimateCFOFromTraining()` (fallback when no chirp) |
+| `tools/cli_simulator.cpp` | TX CFO injection (`--tx-cfo`) and signal capture flags |
+| `tests/verify_cfo_chain.sh` | End-to-end CFO verification gate command |
+| `tools/verify_cfo_chain_dump.py` | Numeric verification of internal pre/post CFO correction |
 
 ---
 
@@ -284,6 +342,7 @@ diff = eq_data × conj(ref) = TX_data × e^{-jφ} × conj(e^{-jφ}) = TX_data  
 4. **Feedback loop must update cached CFO** — After demodulation, `cfo_hz_` and `last_cfo_` in the waveform AND `last_cfo_` in StreamingDecoder must be updated with the pilot-corrected value.
 5. **Drift limiting threshold: 1 Hz** — Reject chirp CFO measurements that differ by >1 Hz from cached value when connected. Real oscillator drift is slow.
 6. **Re-process training after CFO correction** — If LTS residual correction changes `freq_offset_hz`, the training symbols must be re-processed to get a clean channel estimate.
+7. **Pilot-CFO feedback is OFDM-only** — Do not apply OFDM pilot-based CFO updates to MC-DPSK frames.
 
 ---
 
@@ -301,4 +360,5 @@ diff = eq_data × conj(ref) = TX_data × e^{-jφ} × conj(e^{-jφ}) = TX_data  
 
 *Document created: 2026-01-26*
 *Major update: 2026-02-03 — Added fading CFO correction, LTS residual estimation, feedback loop*
+*Major update: 2026-02-12 — Added simulator TX CFO injection, internal pre/post dump hooks, and one-command verification harness*
 *Verified: 100% CW on good fading (3/3 runs, 120/120 CWs), 100% on AWGN*

docs/CFO_PHASE_HARDENING_PLAN.md

@@ -0,0 +1,106 @@
+# CFO Phase Hardening Plan (Pre-Demod Correction Path)
+
+Date: 2026-02-12
+Status: Completed (Phase 1, Phase 2, and Phase 3 implemented and validated)
+Owner: DSP/Modem path
+
+## Objective
+
+Harden CFO correction for connected OFDM flows so chirp/LTS-derived CFO is applied with the correct phase reference before demodulation, and maintain consistency when residual CFO triggers LTS reprocessing.
+
+Primary target: reduce sync-induced decode failures and control-path ACK loss driven by CFO/phase mismatch on fading channels.
+
+## Current Findings (Code Reality)
+
+1. CFO is already corrected before demod in OFDM paths:
+   - `OFDMChirpWaveform::process()` calls `setFrequencyOffsetWithPhase()` before `processPresynced()`.
+   - `OFDMDemodulator::Impl::toBaseband()` applies per-sample phase rotation using `freq_offset_hz` and `freq_correction_phase`.
+
+2. The absolute training-position hook exists but is not wired:
+   - `IWaveform::setAbsoluteTrainingPosition(size_t)` exists.
+   - `StreamingDecoder` never calls it.
+   - OFDM waveforms currently compute initial phase from buffer-relative training offsets.
+
+3. Residual CFO reprocess path resets phase origin to zero:
+   - In `estimateChannelFromLTS()`, if residual CFO is detected, code reprocesses training after `mixer.reset(); freq_correction_phase = 0.0f;`
+   - This can lose the initial phase baseline for the current frame.
+
+## Implementation Plan
+
+### Phase 1: Absolute Training Position Plumbing (Implement now)
+
+1. `StreamingDecoder`:
+   - On successful sync, compute absolute sample index for `sync_position_`.
+   - Call `waveform_->setAbsoluteTrainingPosition(abs_sync_training_start)`.
+   - Keep this per-frame and independent from ring-buffer wrap.
+
+2. `OFDMChirpWaveform` and `OFDMNvisWaveform`:
+   - Override `setAbsoluteTrainingPosition(size_t)`.
+   - Store absolute training start for phase initialization.
+   - In `process()`, use absolute training sample for initial CFO phase.
+   - Fallback to legacy relative offset only if absolute position is not provided.
+
+Acceptance:
+- Build succeeds.
+- Logs show absolute phase reference being used in OFDM process path.
+Implementation note (2026-02-12):
+- Done. `StreamingDecoder` now maps ring index -> absolute sample index at sync and calls `waveform_->setAbsoluteTrainingPosition(...)`.
+- Done. `OFDMChirpWaveform` and `OFDMNvisWaveform` now consume absolute training position for initial CFO phase.
+
+### Phase 2: Residual CFO Reprocess Phase Baseline (Implement now)
+
+1. `channel_equalizer.cpp`:
+   - Capture `freq_correction_phase` at training-start before first LTS pass.
+   - If residual CFO correction triggers reprocess, reset mixer and restore that captured phase baseline (not zero).
+
+Acceptance:
+- Build succeeds.
+- Residual CFO correction path preserves phase continuity assumptions.
+Implementation note (2026-02-12):
+- Done. LTS residual CFO reprocess now restores the captured training-start phase baseline instead of forcing phase=0.
+
+### Phase 3: Connected LTS Sync Metric Upgrade (Next step)
+
+1. Replace/augment current real-only training autocorrelation in `detectDataSync()` with CFO-aware metric (complex/phase-compensated), rather than cosine-only normalization.
+2. Validate detection robustness under ±20..30 Hz CFO and fading.
+
+Acceptance:
+- Lower sync reject streaks on fading + nonzero CFO.
+- No regression in clean AWGN runs.
+Implementation note (2026-02-12):
+- Done. `OFDMChirpWaveform::detectDataSync()` now uses Hilbert/analytic complex correlation for LTS detection instead of real-only autocorrelation.
+- Done. Burst-marker sign detection now uses CFO-phase-compensated complex correlation (`best_p * exp(-j*phi_cfo)`), improving robustness when CFO is non-zero.
+- Quick sanity validation (5 seeds, SNR=20, good fading, R2/3): 5/5 PASS.
+
+## Validation Matrix (Post-Implementation)
+
+1. Simulator (connected OFDM-CHIRP):
+   - SNR 20, fading good/moderate, rate DQPSK R2/3, seeds 30.
+   - Add nonzero CFO scenario (fixed and random, if exposed in harness).
+
+2. Metrics to compare:
+   - First-attempt frame success.
+   - ACK reception success.
+   - Retransmissions/timeouts.
+   - Sync reject streak count.
+   - Logged CFO evolution (chirp -> residual -> pilot corrected).
+
+3. OTA sanity check:
+   - Confirm `[MODE]` / logs do not show unstable CFO jumps frame-to-frame under stable channel.
+
+## Risks / Notes
+
+1. Do not apply a second external pre-rotation in front of OFDM demod unless carefully gated; OFDM demod already performs CFO correction internally.
+2. MC-DPSK path is separate and should not be changed by this phase plan.
+3. Keep behavior backward-compatible for paths that do not provide absolute training position.
+
+## Completion Evidence (2026-02-12)
+
+1. Deterministic chain verification:
+   - `./tests/verify_cfo_chain.sh --cfo 50 --channel awgn --snr 20 --seed 42`
+   - Result: pass, with applied correction matching expected `-CFO` within tolerance across all dumps.
+2. Regression sanity:
+   - `./build/cli_simulator --snr 20 --fading good --rate r1_2 --seed 42 --test`
+   - Result: pass (no regression from hardening changes).
+3. Internal proof path:
+   - `ULTRA_DUMP_CFO_PREFIX` + `ULTRA_DUMP_CFO_CALLS` dump pre/post corrected samples from `toBaseband()` for direct offline validation.

docs/README.md

@@ -17,6 +17,7 @@ This folder now separates **active, authoritative docs** from **historical refer
 - `docs/GUI_ARCHITECTURE.md`: GUI structure and components.
 - `docs/INVARIANTS.md`: Critical engineering invariants.
 - `docs/CFO_CORRECTION_FLOW.md`: CFO handling pipeline details.
+- `docs/CFO_PHASE_HARDENING_PLAN.md`: CFO hardening implementation and validation notes.
 - `docs/ADDING_NEW_WAVEFORM.md`: Guide for adding waveform implementations.
 - `docs/RESEARCH_DIRECTIONS.md`: Long-term R&D ideas (non-blocking).
 

src/gui/modem/streaming_decoder.cpp

@@ -501,6 +501,22 @@ void StreamingDecoder::searchForSync() {
 
         sync_position_ = (search_start + sync_result.start_sample) % MAX_BUFFER_SAMPLES;
 
+        // Convert ring-buffer index to absolute sample index.
+        // Needed so waveform CFO phase starts from true stream time, not local window offset.
+        auto ringPosToAbsolute = [this](size_t ring_pos) -> size_t {
+            if (total_fed_ < MAX_BUFFER_SAMPLES) {
+                // Buffer has not wrapped yet: ring index == absolute sample index.
+                return ring_pos;
+            }
+
+            const size_t oldest_abs = total_fed_ - MAX_BUFFER_SAMPLES;
+            const size_t oldest_pos = write_pos_;  // write_pos_ points to oldest sample after wrap
+            const size_t offset = (ring_pos >= oldest_pos)
+                ? (ring_pos - oldest_pos)
+                : (MAX_BUFFER_SAMPLES - oldest_pos + ring_pos);
+            return oldest_abs + offset;
+        };
+
         // Anti-replay: reject sync at same position as last decoded frame (circular distance)
         if (last_decoded_sync_pos_ != SIZE_MAX) {
             size_t d1 = (sync_position_ >= last_decoded_sync_pos_)
@@ -516,6 +532,12 @@ void StreamingDecoder::searchForSync() {
             }
         }
 
+        // Provide absolute training position to waveform so initial CFO phase is aligned.
+        if (waveform_) {
+            const size_t abs_training_pos = ringPosToAbsolute(sync_position_);
+            waveform_->setAbsoluteTrainingPosition(abs_training_pos);
+        }
+
         // CFO handling: On fading channels, chirp-based CFO measurement can be corrupted
         // by multipath (peaks shift differently for up vs down chirp).
         // When connected, trust the established CFO and limit drift.
@@ -939,26 +961,29 @@ void StreamingDecoder::decodeCurrentFrame() {
         return;  // processBuffer() will call accumulateBurstFrames() on next iteration
     }
 
-    // Feed back pilot-corrected CFO to cached value
-    float corrected_cfo = waveform_->estimatedCFO();
-    float current_cfo = last_cfo_.load();
+    // Feed back pilot-corrected CFO to cached value (OFDM only).
+    // MC-DPSK does not have pilot-based tracking; keep chirp-derived CFO.
+    if (is_ofdm) {
+        float corrected_cfo = waveform_->estimatedCFO();
+        float current_cfo = last_cfo_.load();
 
-    if (connected_) {
-        constexpr float MAX_PILOT_CFO_DRIFT_HZ = 2.0f;
-        float drift = corrected_cfo - current_cfo;
-        if (std::abs(drift) > MAX_PILOT_CFO_DRIFT_HZ) {
-            LOG_MODEM(WARN, "[%s] Pilot CFO drift clamped: %.2f → %.2f Hz (drift=%.2f, max=%.1f)",
-                      log_prefix_.c_str(), current_cfo, corrected_cfo, drift, MAX_PILOT_CFO_DRIFT_HZ);
-            corrected_cfo = current_cfo + std::copysign(MAX_PILOT_CFO_DRIFT_HZ, drift);
+        if (connected_) {
+            constexpr float MAX_PILOT_CFO_DRIFT_HZ = 2.0f;
+            float drift = corrected_cfo - current_cfo;
+            if (std::abs(drift) > MAX_PILOT_CFO_DRIFT_HZ) {
+                LOG_MODEM(WARN, "[%s] Pilot CFO drift clamped: %.2f → %.2f Hz (drift=%.2f, max=%.1f)",
+                          log_prefix_.c_str(), current_cfo, corrected_cfo, drift, MAX_PILOT_CFO_DRIFT_HZ);
+                corrected_cfo = current_cfo + std::copysign(MAX_PILOT_CFO_DRIFT_HZ, drift);
+            }
         }
-    }
 
-    if (std::abs(corrected_cfo - current_cfo) > 0.1f) {
-        LOG_MODEM(INFO, "[%s] CFO updated: %.2f → %.2f Hz (pilot-corrected)",
-                  log_prefix_.c_str(), current_cfo, corrected_cfo);
+        if (std::abs(corrected_cfo - current_cfo) > 0.1f) {
+            LOG_MODEM(INFO, "[%s] CFO updated: %.2f → %.2f Hz (pilot-corrected)",
+                      log_prefix_.c_str(), current_cfo, corrected_cfo);
+        }
+        last_cfo_.store(corrected_cfo);
+        sync_cfo_ = corrected_cfo;
     }
-    last_cfo_.store(corrected_cfo);
-    sync_cfo_ = corrected_cfo;
 
     constexpr size_t LDPC_BLOCK = v2::LDPC_CODEWORD_BITS;
     CodeRate rate = connected_ ? code_rate_ : CodeRate::R1_4;