perf: lazy page rendering in convert_pdf_to_image

[KRRT7]

Summary

• Render and save each PDF page inside the page loop instead of accumulating all images in memory first, reducing peak memory usage
• When path_only=True, images are no longer retained after saving to disk

Peak memory drops from O(N pages) to O(1 page).

Benchmark

Azure Standard_D8s_v5 — 8 vCPU Intel Xeon Platinum 8473C, 32 GiB RAM, Python 3.12.12

50-page PDF (sample-docs/pdf2image-memory-error-test-400p.pdf, first 50 pages), rendered at 200 DPI with output_folder:

Memory (peak per test)

Ref	Peak Memory	Allocations	Delta
6a085c621f1c (base)	248.7 MiB	695
692f6a17840b (head)	74.8 MiB	780	-70%

Peak memory drops 174 MiB (-70%) by rendering and saving each page inside the loop instead of accumulating all PIL images before writing. Timing is neutral (+2% within noise).

Timing — convert_pdf_to_image (output_folder path, 50 pages)

Ref	Min	Median	Mean	OPS	Rounds
6a085c621f1c (base)	1.7s	1.7s	1.7s	0.58 ops/s	5
692f6a17840b (head)	1.7s	1.8s	1.8s	0.57 ops/s	5
Speedup	0.99x	0.97x	0.98x	0.98x

End-to-end pipeline benchmark (10 PDFs, 2,905 total pages)

	Before	After
Avg memory	7.5 GB	4.6 GB
Steady-state	~7.9 GB (never released)	~4.8 GB

---

Generated by codeflash optimization agent

Reproduce the benchmark locally

# Full comparison (timing + memory):
uv run codeflash compare 6a085c621f1c 692f6a17840b --memory \
  --functions 'unstructured_inference/inference/layout.py::convert_pdf_to_image' \
  --inject benchmarks/test_benchmark_convert_pdf.py \
  --inject benchmarks/__init__.py \
  --inject pyproject.toml \
  --inject sample-docs/pdf2image-memory-error-test-400p.pdf

Benchmark test source

"""Benchmark for convert_pdf_to_image() lazy page rendering optimization.

Uses an existing multi-page sample PDF to measure peak memory of
convert_pdf_to_image() with output_folder. The key optimization renders
and saves each page inside the loop instead of accumulating all PIL images
before saving -- reducing peak from O(N pages) to O(1 page).

Uses 50 pages from the 400-page test PDF to exercise the memory difference.
"""

import tempfile
from pathlib import Path

from unstructured_inference.inference.layout import convert_pdf_to_image

# 400-page PDF from the repo's sample-docs -- use first 50 pages
_SAMPLE_PDF = str(Path(__file__).resolve().parent.parent / "sample-docs" / "pdf2image-memory-error-test-400p.pdf")
_N_PAGES = 50


def run_convert_with_output_folder():
    """Convert 50 PDF pages to images, saving to disk (the optimized path)."""
    with tempfile.TemporaryDirectory() as tmpdir:
        result = convert_pdf_to_image(
            filename=_SAMPLE_PDF,
            output_folder=tmpdir,
            dpi=200,
            last_page=_N_PAGES,
        )
    return result


def test_benchmark_convert_pdf_to_image(benchmark):
    benchmark(run_convert_with_output_folder)

Test plan

• Added tests for convert_pdf_to_image covering the no-output-folder and output-folder-with-images return paths
• codeflash compare confirms -70% peak memory (248.7 MiB -> 74.8 MiB) on 50-page PDF
• Timing neutral (+2% within noise)