perf: replace pdfium global lock with process pool for concurrent rendering

[KRRT7]

Summary

• Replace the global threading.Lock around pypdfium2 rendering with a ProcessPoolExecutor
• Each worker process has its own pdfium instance (separate address space), enabling true parallel PDF rendering
• Pool size defaults to cpu_count / 2, configurable via PDF_RENDER_POOL_SIZE env var
• No changes to the convert_pdf_to_image public API

Why

pypdfium2's C library is not thread-safe, so PR #458 added a global _pdfium_lock = Lock() that serializes all PDF rendering. When multiple documents are processed concurrently (via asyncio.to_thread in core-product), each one queues behind the others — destroying throughput.

Benchmark

Rendering loremipsum_multipage.pdf (10 pages, 200 DPI) with N concurrent threads:

Threads	Lock (before)	Pool (after)	Speedup
1	0.78s	0.81s	1.0x
2	1.11s	1.06s	1.1x
4	4.56s	1.66s	2.7x
8	5.27s	2.80s	1.9x

Single-document latency unchanged (~66ms/page after warmup).

Rendering accounts for 23-43% of total per-page pipeline time (depending on model inference speed), so this translates to meaningful end-to-end throughput gains under concurrency.

Benchmark script

"""Benchmark: pdfium process pool concurrency."""
import os, sys, time, statistics, tempfile, threading

INFERENCE_ROOT = os.path.expanduser("~/Desktop/work/unstructured_org/unstructured-inference")
sys.path.insert(0, INFERENCE_ROOT)
PDF = os.path.join(INFERENCE_ROOT, "sample-docs/loremipsum_multipage.pdf")

from unstructured_inference.inference.layout import convert_pdf_to_image, _get_render_pool

# Warm up all pool workers
pool = _get_render_pool()
threads = []
for _ in range(pool._max_workers):
    def _w():
        with tempfile.TemporaryDirectory() as tmp:
            convert_pdf_to_image(filename=PDF, dpi=200, output_folder=tmp, path_only=True)
    t = threading.Thread(target=_w); t.start(); threads.append(t)
for t in threads: t.join()
for _ in range(3):
    with tempfile.TemporaryDirectory() as tmp:
        convert_pdf_to_image(filename=PDF, dpi=200, output_folder=tmp, path_only=True)

# Baseline
times = []
for _ in range(5):
    with tempfile.TemporaryDirectory() as tmp:
        t0 = time.perf_counter()
        convert_pdf_to_image(filename=PDF, dpi=200, output_folder=tmp, path_only=True)
        times.append(time.perf_counter() - t0)
single = statistics.median(times)

for n in [1, 2, 4, 8]:
    walls = []
    for _ in range(3):
        per = [None]*n
        def worker(i):
            with tempfile.TemporaryDirectory() as tmp:
                t0 = time.perf_counter()
                convert_pdf_to_image(filename=PDF, dpi=200, output_folder=tmp, path_only=True)
                per[i] = time.perf_counter() - t0
        t0 = time.perf_counter()
        ts = [threading.Thread(target=worker, args=(i,)) for i in range(n)]
        for t in ts: t.start()
        for t in ts: t.join()
        walls.append(time.perf_counter() - t0)
    w = statistics.median(walls)
    print(f"{n} threads: wall={w:.3f}s  speedup_vs_lock={n*single/w:.1f}x")

Test plan

• All 247 existing tests pass (pytest test_unstructured_inference/ -x -q — 247 passed, 1 skipped)
• Benchmarked with real PDFs (loremipsum_multipage.pdf, layout-parser-paper.pdf, recalibrating-risk-report.pdf)
• Verified single-document latency is not regressed
• Verified pool handles path_only=True (primary production path) and PIL image return