Conversation
…er_fe_mc Adds optional `Rcpp::Nullable<Rcpp::NumericMatrix> fit_init = R_NilValue` parameter at the end of the IFE/MC C++ entry points and the inner EM workhorse functions. When non-null and shape matches Y, the EM loop seeds `fit` from it instead of the default `fit = Y0` (cold-start). NULL (default) preserves pre-2.4.2 behavior exactly. Both `fit = Y0` initialization sites in fe_ad_inter_iter and fe_ad_inter_covar_iter are routed through the warm_init dispatch (initial init AND post-burnin restart under use_weight=1). Sites touched: - src/ife_sub.cpp: fe_ad_inter_iter, fe_ad_inter_covar_iter - src/ife.cpp: inter_fe_ub (forwards to both inner functions) - src/mc.cpp: inter_fe_mc (forwards to fe_ad_inter_iter via mc=1) - src/fect.h: forward declarations updated CFE warm-start (cfe_iter / complex_fe_ub) deferred to v2.5.0 --- the additional gamma/kappa FE structures complicate the conditional init logic and the marginal coverage gain is small. Statistical justification (Yiqing 2026-05-01): the prediction surface Y_hat = F * Lambda^T is the unique global minimizer (Eckart-Young for unconstrained low-rank; convexity for nuclear-norm MC; convexity of the entropy-regularized projection for simplex bound). Factorization is non-unique by rotation, but Y_hat is uniquely identified --- so warm-starting won't bias the bootstrap distribution of any estimand. Full reasoning: statsclaw-workspace/fect/ref/ warm-start-audit-2026-05-01.md. This commit is the engine-side plumbing only. R wrappers and boot.R activation come in subsequent commits. Existing test suite (estimand + book-claims subset, 196 expectations) passes unchanged --- by construction since fit_init = NULL throughout. Refs: v2.4.2 release plan; statsclaw-workspace/fect/runs/ 2026-05-01-v242-inference-infra.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `fit.init = NULL` parameter to fect_fe() and fect_mc() R wrappers; threads it through to the inter_fe_ub / inter_fe_mc C++ calls as the `fit_init` argument. NULL preserves pre-2.4.2 cold-start behavior; bootstrap callers will pass the main-fit prediction surface (boot.R activation in next commit). Note: the r.cv = 0 sub-fit inside fect_fe (no factors, no EM) explicitly passes NULL even when fit.init is provided --- the warm-start matrix is shaped for a factor-model EM, not a no-factor additive-FE solver. Scope: - IFE: fect_fe (R/fe.R) - MC: fect_mc (R/mc.R) Deferred (separate commit or later release): - GSC path via fect_nevertreated (.estimate_co helper); needs centering-aware fit.init handling - CFE path via fect_cfe (deferred to v2.5.0 with cfe_iter) Tests: existing suite passes (fit.init = NULL throughout, behavior unchanged). Refs: v2.4.2 release plan; runs/2026-05-01-v242-inference-infra.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous message "F-test Failed. The estimated covariance matrix is singular." conflated two distinct inferential statements: (a) the test failed to reject the null, vs (b) the test could not be computed (singular covariance). Standard hypothesis-test usage reserves "Failed to reject" for case (a). Case (b) is a numerical-failure message and should be worded accordingly. Two occurrences in R/diagtest.R updated. Refs: Yiqing 2026-05-01 hypothesis-test-language feedback; test-language memory feedback_test_language_failed.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…1000
ci.method enum: c("basic", "percentile") -> c("basic", "percentile", "bc", "normal").
New methods:
- "bc": bias-corrected percentile (Efron 1987 minus acceleration).
z0 = Phi^-1(P(boot < est)); cutoffs shift by 2*z0 to compensate
for bootstrap-median bias relative to the point estimate. Free
(just one Phi^-1 call); uniform across all vartype values.
- "normal": Wald (theta +- z * SE). What fit$est.att already uses
internally; textbook for jackknife.
Per-type ci.method defaults via NULL trigger (ci.method = NULL):
- att -> "normal" (matches what fit$est.att actually contains;
corrects v2.4.1's mislabeled "basic" passthrough)
- att.cumu -> "percentile" (matches what att.cumu() does internally)
- aptt -> "bc" (ratio estimator; basic CI flips off the point
under bootstrap skew)
- log.att -> "bc" (log estimator; same skew rationale)
Existing v2.4.1 explicit ci.method values still work; the new default
only fires when ci.method is omitted.
Implementation: new internal helper .compute_ci() centralizes all four
methods. Refactored call sites: .estimand_att_overall,
.compute_aptt_event_time, .compute_log_att_event_time. att fast path
updated: triggers on ci.method == "normal" instead of "basic" so
default att queries hit the byte-equality passthrough from fit$est.att.
nboots default 200 -> 1000 in R/default.R (all three signatures
fect / fect.formula / fect.default), plus the misspecified-nboots
error message updated. Existing scripts that pass nboots = ...
explicitly are unaffected.
Note: bc CI may degenerate (ci.lo == ci.hi) when the point estimate
is far outside the bootstrap distribution --- this is the cell-drop
pathology in log.att / aptt; addressed by the hard-error in the
next commit.
Refs: statsclaw-workspace/fect/ref/v242-vartype-cimethod-design.md;
runs/2026-05-01-v242-inference-infra.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…t / aptt When a cell used in the point estimate has Y0_b <= 0 in a non-trivial fraction of bootstrap replicates, log(Y0_b) is undefined and colMeans(..., na.rm = TRUE) silently averages over fewer cells in that replicate. The point estimate uses N cells; bootstrap averages may use N, N-1, ..., 0 cells. This breaks the basic bootstrap principle that the resampled estimator should compute over the same data structure as the point estimator, contaminating the bootstrap distribution and yielding meaningless inference. v2.4.2 detects this and hard-errors with actionable guidance: 1. Filter out unstable cells via `cells = ~ Y0_hat > <threshold>` 2. Transform the outcome before fect: log(Y + c) 3. Use a different estimand (att doesn't have this pathology) Two detection paths: - log.att (.compute_log_att_event_time): trigger when the WORST cell has Y0_b <= 0 in > 5% of replicates. Sub-threshold dropping is tolerated as benign (small noise at the bootstrap-distribution scale). - aptt (.compute_aptt_event_time): trigger when E(Y0_b) ~ 0 (within 1e-10) in any replicate, since the APTT denominator blows up there. The 5% threshold is a heuristic; can be made tunable via an argument in a follow-up release if user feedback warrants. Also reverts an inadvertent suppressWarnings around the inner log(Y0_b_ok) call --- with the hard-error gate, suppressWarnings is no longer needed (any surviving log call has positive args). Refs: statsclaw-workspace/fect/ref/v242-vartype-cimethod-design.md hard-error section; runs/2026-05-01-v242-inference-infra.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New file `tests/testthat/test-estimand-ci-methods.R` (10 tests, 36
expectations) covers the v2.4.2 ci.method extensions:
- CI.1: enum accepts basic / percentile / bc / normal
- CI.2: NULL trigger gives per-type defaults (att=normal etc)
- CI.3: normal CI is symmetric around the point estimate
- CI.4: bc collapses to percentile when bootstrap median = point
- CI.5: bc shifts cutoffs when bootstrap is biased relative to point
- CI.6: vartype column reports fit-time vartype regardless of ci.method
- CI.7: log.att on simdata triggers cell-drop hard-error
- CI.7b: log.att works on a positive-Y panel
- CI.8: fect() / fect.formula / fect.default nboots default = 1000
Updated existing fixtures:
- `tests/testthat/test-estimand-log-att.R`:
- .fit_positive_Y(): bumped intercept 0.5 -> 2.0 and reduced
noise sd 0.2 -> 0.1 to keep Y comfortably positive (so the
new cell-drop hard-error doesn't fire on benign bootstrap noise).
- LA.4 now expects the hard-error instead of a warning.
- `tests/testthat/test-estimand-parametric.R`:
- log.att-on-parametric test now expects the hard-error
(sim_linear has many negative Y cells; v2.4.1's silent warning
was producing meaningless inference).
All 6 estimand test files pass: 33 tests, 97 expectations, 0 failures.
Refs: statsclaw-workspace/fect/runs/2026-05-01-v242-inference-infra.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…workers
Per Yiqing 2026-05-01: parametric bootstrap (and any parallel path
that uses mvtnorm / future / etc.) emitted N warnings ("package
'mvtnorm' was built under R version X.Y.Z") --- one per worker on
first package use --- because workers loaded packages in user-visible
contexts.
Two-pronged fix:
1. `.fect_make_future_cluster()` now runs `clusterEvalQ` at cluster
creation to pre-load mvtnorm / future / future.apply / doParallel /
foreach via `requireNamespace(quietly = TRUE)` inside
suppressPackageStartupMessages + suppressWarnings. This loads
packages SILENTLY at startup so subsequent worker code doesn't
re-trigger the version-mismatch warning.
2. New helper `.fect_with_quiet_pkg_warnings(expr)` wraps an
expression with a `withCallingHandlers` that targets ONLY the
"was built under R version" warning class (other warnings pass
through normally). Used at the boot.R parametric path's
`future_lapply` call as belt-and-suspenders.
Refactored fect() and did_wrapper() to route through
`.fect_make_future_cluster()` instead of `future::multisession`
directly, so all parallel paths benefit from the silent pre-load.
CV (cv.R), nevertreated (fect_nevertreated.R), and the bootstrap
foreach loop in boot.R already use the helper, so they automatically
benefit. fittest.R + permutation.R use `parallel::makeCluster()`
directly --- can route through the helper in a follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The book-claims test asserted fect()'s nboots default is 200, which was true through v2.4.1. v2.4.2 raises the default to 1000 (improves percentile/bc tail estimates). Update the assertion to match. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…TURE - DESCRIPTION: Version 2.4.1 -> 2.4.2; Date 2026-04-29 -> 2026-05-01 - NEWS.md: full v2.4.2 entry covering ci.method extensions (bc + normal + per-type defaults), cell-drop hard-error in log.att / aptt, nboots default raised 200 -> 1000, internal warm-start infrastructure (not user-visible; activation deferred to v2.5.0 with empirical justification), diagtest message reword, parallel-worker package warning suppression - vignettes/bb-updates.Rmd: same changelog entry, condensed for the user-facing book - man/estimand.Rd: ci.method default updated NULL with per-type default explanation - R/po-estimands.R: roxygen comment for ci.method updated to match - ARCHITECTURE.md: version 2.4.1 -> 2.4.2; po-estimands.R row reflects ci.method extensions and hard-error addition; estimand() function-table row updated. Manual patches noted; full scriber regen deferred until next material module change. Validation gates run on this branch: - Full testthat suite: 1190 expectations, 0 failures, 0 errors - estimand subset: 6 files, 33 tests, 97 expectations passing - ci-methods test file (new): 10 tests, 36 expectations passing - Coverage simulation (200 sims, all 4 ci.methods on att/overall): basic 92.5%, bc 91.0%, normal 91.5%, percentile 91.0% (nominal 95%; all within MC noise band; details in statsclaw-workspace/fect/ref/v242-coverage-study/findings.md) - Quarto book: rendered cleanly - R CMD check: blocked by a pre-existing macOS cp/xattr issue unrelated to v2.4.2 changes; CI on Ubuntu will verify on PR Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v2.4.2: inference infrastructure (ci.methods + cell-drop hard-error + warm-start C++ infra)
Owner
Author
|
Closing per Yiqing's authorization rule (master merge requires explicit OK; opening this PR was overreach in the autonomous overnight run). Work is on dev (PR #133 merged). Reopen / re-create when ready to ship v2.4.2. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v2.4.2
Release-prep PR. All commits squashed in PR #133 (now on dev).
Headline changes
ci.method = "bc"and"normal"added to estimand(); per-type defaults via NULL trigger.nbootsdefault raised 200 -> 1000 in fect() / fect.formula() / fect.default().Validation
feat/v242-inference-infrabranchStatsclaw artifacts (workbench-side)
statsclaw-workspace/fect/runs/2026-05-01-v242-inference-infra.mdstatsclaw-workspace/fect/ref/v242-vartype-cimethod-design.mdstatsclaw-workspace/fect/ref/warm-start-audit-2026-05-01.mdstatsclaw-workspace/fect/ref/v242-coverage-study/After merge
🤖 Generated with Claude Code