Repository intelligence and context optimization toolkit for Claude Code workflows.
TokenMaxxing helps developers reduce unnecessary token consumption through repository analysis, duplicate detection, context compression, prompt optimization, and context prioritization.
TokenMaxxing is a high-performance Rust CLI and developer tools suite designed for codebase profiling, serving as a smart Claude tokenizer assistant and complete repository intelligence utility. Specially engineered for Claude Code workflows, it guarantees maximum token efficiency, precise token optimization, and rigorous context optimization by delivering automated repository analysis, telegraphic prompt optimization, and advanced context compression (such as whitespace minification and semantic deduplication).
Note
This project is currently in Public Beta. While fully functional and verified, APIs and options may evolve based on real-world usage.
- Token Counting Engine: Fast, Accurate, and Hybrid subword token estimation using parallel directory walkers.
- Smart Repository Heuristics: Detects build artifacts (e.g., target, build, dist), package locks, duplicate files, cache temp folders, and oversized structures.
- Context Quality Ranker: Separates file contexts into logical blocks and ranks them by semantic value.
- Context Simulator: Models repository size ratio against typical LLM context bounds (200k tokens) to alert users of context overflow risks.
- Prompt Rewrite Engine: Compresses conversational noise using mode-specific telegraphic filters.
- Memory Generator: Caches dependency schemas and codebase architecture summaries to avoid repeated file walks.
- Visual Dashboards: Renders output metrics cleanly into Markdown, JSON, or static HTML layouts.
TokenMaxxing CLI can be built and installed locally via Cargo:
cargo install --path tokenmaxxing-cliInitialize a quick scan of your current directory:
# Run the end-to-end auto pipeline
tokenmaxxing auto .
# Analyze repository waste
tokenmaxxing waste .
# Evaluate context window space utilization
tokenmaxxing simulate .| Command | Action |
|---|---|
count <PATH> |
Counts tokens in a file or directory using fast or accurate modes. |
optimize <PATH> |
Minifies Markdown, JSON, or XML formatting. |
compress <PATH> |
Trims redundant whitespace and boilerplate segments. |
analyze <PATH> |
Logs oversized structures and suspected duplicated files. |
stats <PATH> |
Outputs total estimated tokens using the hybrid algorithm. |
report <PATH> |
Compiles an optimization report as Markdown dashboard. |
benchmark <PATH> |
Runs local execution speed benchmarks and saves metrics. |
context <PATH> |
Renders quality metrics, noise ratio, and optimization opportunities. |
rank <PATH> |
Lists chunk prioritizations sorted by semantic logic. |
waste <PATH> |
Tallies cache files, target artifacts, and duplicate content bytes. |
simulate <PATH> |
Models footprint against LLM limits and alerts overflow risk. |
auto <PATH> |
Executes a sequential pipeline scan, waste test, and scoring pass. |
rewrite <PATH> |
Condenses prompt politeness blocks (Balanced, Aggressive flags). |
summarize <PATH> |
Evaluates active module types and component footprints. |
memory <PATH> |
Tracks parsed cargo dependency vectors and workspace layouts. |
dashboard <PATH> |
Outputs full metrics to terminal, --html file, or --json parser. |
TokenMaxxing segregates logic across:
tokenmaxxing-core: The main processing and estimation engine.tokenmaxxing-cli: Interactive CLI.- Multi-language bindings (
napi-rsfor JS/TS,pyo3for Python).
Evaluated on the local repository structure:
| Metric | Performance |
|---|---|
| Counting Speed | 43 ms |
| Compression Ratio | 67.74% (lower is better) |
| Optimization Ratio | 68.15% (lower is better) |
| Repository Analysis Speed | 4 ms |
Verified against standard tests:
- Unit Tests: 15 tests (all passing).
- Quality Checks:
cargo clippyandcargo fmtverified (0 warnings, 0 errors). - Release Compilation: Successful.
- Context Quality scoring and chunk ranking algorithms.
- Smart repository analysis (Target builds, lockfiles, cache paths).
- Interactive Prompt Rewriter modes.
- Dashboard generation (JSON/HTML/Markdown).
- Tree-sitter advanced AST compression support.
- Pre-commit git hooks integration.
Please refer to CONTRIBUTING.md for local setup and testing standards.
This project is licensed under the MIT License - see the LICENSE file for details.