Skip to content

circuit-tracer integration for CrossLayerTranscoders#8

Open
jiito wants to merge 104 commits into
Goreg12345:masterfrom
jiito:conversion-to-circuit-tracer
Open

circuit-tracer integration for CrossLayerTranscoders#8
jiito wants to merge 104 commits into
Goreg12345:masterfrom
jiito:conversion-to-circuit-tracer

Conversation

@jiito
Copy link
Copy Markdown
Contributor

@jiito jiito commented Nov 6, 2025

Changes

  1. Added ModelConverter and concrete CircuitTracerConverter that is used to turn a CLTModule into circuit tracer compatible format.
  2. Added callbacks for hugging face upload and model conversion.
  3. Added integration test to verify model conversion + loading in circuit-tracer.

Questions

  1. When is the best time to save the circuit-tracer compatible model? Which callback to attach it to in the checkpoint? I have made this configurable with config arguments
  2. Do we have to adapt how we save the non-linearity threshold? The circuit tracer expects a [1, d_features] tensor for each layer's non-linearity. Right now, I am using model.nonlinearity.theta[:, layer_idx, :] but this doesn't account for the bandwidth...
  3. Any better suggestions for the ModelConverter interface? This is maybe better framed as a ModuleSaver? I am planning for using this same interface for delphi and other integrations.

Verification

  • integration test with an untrained model
  • tested the ModelConversionCallback manually
  • Functional E2E test with training (circuit-tracer.yaml) and loading the saved model.
  • Double check non-lin settings

Resolves #4

Future Work

  • Because circuit-tracer only supports relu and jump relu activation functions, additional work needs to be done to address compatibility with topK and other types of CLTs.
  • Take a look at composability of yaml config files to greatly simplify / modularize aspects of the training process.

@jiito jiito marked this pull request as ready for review November 18, 2025 17:29
@jiito jiito mentioned this pull request Nov 18, 2025
@jiito jiito changed the title circuit-tracer integration circuit-tracer integration for CrossLayerTranscoders Jan 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Circuit-tracer Compatibility

1 participant