Skip to content

Benchmark: measure cold start / first-call overhead #87

@Oaklight

Description

@Oaklight

Parent: #80

Problem

pytest-benchmark includes warmup rounds by default, which hides first-call overhead: module import time, regex compilation, lookup table initialization, etc.

For zerodep's use case (single-file modules copied into projects), cold start is the real user experience in:

  • CLI tools that import a module, do one operation, and exit
  • Serverless functions (Lambda/Cloud Functions) with fresh containers
  • Short-lived scripts

Proposal

  • Add a test_cold_start.py (project-level, not per-module) that measures import + first-call time for each module
  • Use subprocess to get true cold import time (no cached bytecode from the test runner)
  • Alternatively, use importlib.invalidate_caches() + importlib.reload() to simulate cold import within the test process

Implementation sketch

import subprocess, time, json

MODULES = ["yaml", "multipart", "httpclient", "soup", "protobuf", ...]

def test_cold_import(benchmark):
    """Measure import time via subprocess to avoid cached state."""
    def run():
        subprocess.run(
            ["python", "-c", "import yaml"],
            capture_output=True, check=True,
        )
    benchmark(run)

Reporting

  • Separate section in benchmark report: "Cold Start Times"
  • Compare against reference libraries (e.g., import yaml vs import PyYAML)

Cost

Low — single test file, straightforward subprocess measurements.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Priority 3 - Medium-LowenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions