arch_eval-v0.2.0

arch_eval v0.2.0

First Stable Release of the arch_eval library 🔥

• Fixed All Bugs
• Now the library is completely compatible with all kinds of datasets and models
• Fully Compatible with the transformer library.
• Added datasets for Text, Vision and VisionLanguage Tasks.
• Added new exceptions
• New hook decorator for plugins

Documentation 📖

Documentation available at This Page

arch_eval-v0.1.0

arch_eval-v0.1.0 -- Initial Release

This is the first release of arch_eval – a high-level library for efficient and fast architecture evaluation and comparison of machine learning models. It provides a unified interface for training, benchmarking, and hyperparameter optimization, designed to accelerate ML research and development workflows.

Key Features

• Unified Training Interface – Train single models with a clean, configuration-driven API. One Trainer class handles everything from data loading to distributed training.
• Multi-Model Benchmarking – Compare multiple architectures sequentially or in parallel (thread/process-based) with automatic metric extraction and result aggregation.
• Distributed Training – Built-in support for DataParallel, DistributedDataParallel (DDP), and FSDP – scale from one GPU to multi-node clusters.
• Mixed Precision – AMP with float16, bfloat16, and experimental FP8 support (requires NVIDIA Transformer Engine).
• Gradient Checkpointing – Reduce memory footprint for large models with minimal code changes.
• Rich Visualization – Real-time training windows with metric curves and system resource monitoring; video recording of metric evolution; publication-ready final plots.
• Logging – Native integration with Weights & Biases.
• Hyperparameter Optimization – Grid search and random search out-of-the-box, with extensible search strategies.
• Extensible Plugin System – Custom hooks and callbacks for maximum flexibility; global plugins that can be discovered automatically.
• Data Handling – Supports PyTorch Datasets, synthetic data (scikit-learn generators), torchvision datasets, Hugging Face datasets, and streaming data.
• Production-Ready – Configurable timeouts, retry logic, deterministic execution, and comprehensive checkpointing.

What's Included?

Core Components

• Trainer – Single-model training loop with full lifecycle hooks
• Benchmark – Multi-model comparison with parallel execution
• HyperparameterOptimizer – Grid and random search
• PluginManager – Dynamic plugin discovery and hook execution

Configuration Classes

• TrainingConfig – All parameters for a single training run
• BenchmarkConfig – Parameters for model comparison
• BaseConfig – Common fields (data, device, logging, etc.)

Callbacks

• EarlyStopping – Stop when metrics plateau
• ModelCheckpoint – Save best models automatically
• LRSchedulerLogger – Log learning rates
• TensorBoardLogger – TensorBoard integration

Data Utilities

• DatasetHandler – Automatic dataset conversion to PyTorch DataLoaders
• SyntheticDataset – Wrapper for generated data
• create_synthetic_dataset – Factory for scikit-learn synthetic datasets

Distributed Utilities

• init_distributed / cleanup_distributed – Process group management
• get_wrapped_model – Automatic model wrapping for DDP/FSDP

Visualization

• RealtimeWindow – Live training plots with system stats
• VideoRecorder – Metric evolution videos (requires ffmpeg)
• PlotSaver – Final metric plots (PNG)

Metrics

• MetricCalculator – Task-aware metrics (classification, regression, language modeling) with confusion matrix support

Profiling

• profiler_context – PyTorch profiler integration with TensorBoard trace export

Utilities

• auto_device decorator – Automatic tensor device placement
• memory_summary – Human-readable memory usage
• setup_logging – Configurable logging with file rotation

Documentation

Full documentation is available at: arch_eval 0.1.0 documentation

Installation

# From GitHub (latest development)
git clone --depth=1 https://github.com/lof310/arch_eval.git
cd arch_eval
pip install -e .

Requirements

• Python ≥ 3.8
• PyTorch ≥ 1.9
• pandas, numpy, scikit-learn, psutil, matplotlib, seaborn
• Optional: wandb, transformer-engine (for FP8), ffmpeg (for video)

License

Distributed under the Apache License 2.0. See LICENSE for more information.

Citation

If you use arch_eval in your research, please cite:

@software{arch_eval2026,
  author = {Leinier Orama Fernández},
  title = {arch_eval: High-level Library for Architecture Evaluation of ML Models},
  year = {2026},
  publisher = {GitHub},
  url = {https://github.com/lof310/arch_eval}
}