Parent: #80
Problem
pytest-benchmark includes warmup rounds by default, which hides first-call overhead: module import time, regex compilation, lookup table initialization, etc.
For zerodep's use case (single-file modules copied into projects), cold start is the real user experience in:
- CLI tools that import a module, do one operation, and exit
- Serverless functions (Lambda/Cloud Functions) with fresh containers
- Short-lived scripts
Proposal
- Add a
test_cold_start.py (project-level, not per-module) that measures import + first-call time for each module
- Use
subprocess to get true cold import time (no cached bytecode from the test runner)
- Alternatively, use
importlib.invalidate_caches() + importlib.reload() to simulate cold import within the test process
Implementation sketch
import subprocess, time, json
MODULES = ["yaml", "multipart", "httpclient", "soup", "protobuf", ...]
def test_cold_import(benchmark):
"""Measure import time via subprocess to avoid cached state."""
def run():
subprocess.run(
["python", "-c", "import yaml"],
capture_output=True, check=True,
)
benchmark(run)
Reporting
- Separate section in benchmark report: "Cold Start Times"
- Compare against reference libraries (e.g.,
import yaml vs import PyYAML)
Cost
Low — single test file, straightforward subprocess measurements.
Parent: #80
Problem
pytest-benchmark includes warmup rounds by default, which hides first-call overhead: module import time, regex compilation, lookup table initialization, etc.
For zerodep's use case (single-file modules copied into projects), cold start is the real user experience in:
Proposal
test_cold_start.py(project-level, not per-module) that measures import + first-call time for each modulesubprocessto get true cold import time (no cached bytecode from the test runner)importlib.invalidate_caches()+importlib.reload()to simulate cold import within the test processImplementation sketch
Reporting
import yamlvsimport PyYAML)Cost
Low — single test file, straightforward subprocess measurements.