a clean and readable implementation of Tiny Recursive Models by Samsung SAIL from scratch in pytorch. The TRM paper builds on ideas introduced in the earlier Hierarchical Reasoning Model paper.
Tiny Recursive Models (TRMs) work by refining their predictions step by step instead of producing an answer in a single forward pass like standard Transformers. The model maintains two internal states: a prediction state and a latent state. Starting from learnable initial values, the latent state is repeatedly refined using the input and current prediction, and then the prediction itself is updated using the refined latent state. The same small Transformer block is reused at every step, allowing the model to iteratively improve its understanding while keeping the number of parameters low. This recursive refinement process is inspired by how humans revise their thoughts over multiple passes rather than reasoning all at once.