Skip to content

feat: add OpenAI-compatible LLM backend#220

Open
pntech20 wants to merge 1 commit into
vecna-labs:mainfrom
pntech20:codex/openai-compatible-backend
Open

feat: add OpenAI-compatible LLM backend#220
pntech20 wants to merge 1 commit into
vecna-labs:mainfrom
pntech20:codex/openai-compatible-backend

Conversation

@pntech20
Copy link
Copy Markdown

Summary

  • add OpenAICompatibleBackend for Chat Completions-compatible HTTP providers
  • support text and JSON-schema requests, API key/env auth, extra headers, timeouts, and HTTP error mapping
  • export the backend from the public API and document the openai-compatible optional install target
  • add mocked HTTP tests for text, structured JSON, HTTP errors, and timeout handling

Verification

  • uv lock --check
  • uv run ruff format src\openrange\llm.py src\openrange\__init__.py tests\test_openai_compatible_backend.py
  • uv run ruff check README.md pyproject.toml src\openrange\llm.py src\openrange\__init__.py tests\test_openai_compatible_backend.py
  • uv run mypy src\openrange\llm.py tests\test_openai_compatible_backend.py
  • uv run pytest tests\test_openai_compatible_backend.py -q
  • git diff --check

Note: I also tried uv run pytest tests\test_openai_compatible_backend.py tests\test_llm_and_dashboard.py -q; the new tests passed, but existing test_llm_and_dashboard.py cases fail on Windows because Python scripts are executed directly as Win32 binaries and os.getpgid is unavailable on Windows.

Refs #188

@larstalian
Copy link
Copy Markdown
Collaborator

Thanks for putting this together — the test setup with a real local HTTPServer is nicer than the mock-heavy version I'd had in mind.

Quick heads-up on scope: I'm updating #188 to lead with a LiteLLMBackend (one library covers ~100 providers and handles the structured-output quirks for each), with this OpenAICompatibleBackend kept around as a zero-dep escape hatch — for folks who don't want to pull in pydantic / httpx / tiktoken and just need to point at Ollama, vLLM, or any other Chat Completions–shaped server. So this PR is still in scope, just framed a bit differently. The README copy should probably reflect that it's the lightweight stdlib option rather than the catch-all.

A few things while you're in there:

  • openai-compatible = [] in pyproject.toml doesn't actually install anything, so uv sync --extra openai-compatible is a no-op. Either drop the extra entirely (the backend is stdlib, no install step needed) or populate it.
  • The strict: True in the response_format payload — fine for OpenAI itself, but Ollama, vLLM, llama.cpp server, Together, and several OpenRouter models either reject it or silently ignore it. Since the README lists those as supported, either flip the default to False or expose it as a constructor option.
  • The api_key_env fallback path isn't covered by a test — every test passes api_key= explicitly. A small monkeypatch.setenv case would close that gap.
  • urlopen() accepts any scheme, including file:// and ftp://. Worth a scheme check in preflight() (http/https only), just so a typo in base_url doesn't quietly read a local file.
  • extra_headers can clobber Authorization — probably intentional for Azure-style api-key headers, but worth a docstring note.
  • The default OPENAI_COMPATIBLE_DEFAULT_MODEL = "gpt-4o-mini" is fine for api.openai.com but won't exist on most compatible servers. Either drop the default or document the caveat.

One small housekeeping thing: the commit on this branch is coming through as mac <mac@apple.com> rather than your GitHub identity, so it isn't linked to your profile in the contributor graph. If you don't mind amending the author (git commit --amend --author="Your Name <verified-email>" + force-push), it just makes git-blame / attribution cleaner down the line.

Happy to merge once these are sorted. Thanks again for picking this up.

@pntech20 pntech20 force-pushed the codex/openai-compatible-backend branch from a8b28c5 to 1414ef6 Compare May 19, 2026 00:40
@pntech20
Copy link
Copy Markdown
Author

Addressed the review items on this branch:

  • removed the empty openai-compatible extra and refreshed uv.lock
  • made JSON schema strictness opt-in with json_schema_strict=True
  • added a non-http scheme guard in preflight()
  • added coverage for api_key_env
  • documented the lightweight stdlib positioning, auth header override behavior, and model caveat
  • amended the commit author to pntech20 <pntech20@gmail.com>

Focused checks passed locally:

  • uv run pytest tests/test_openai_compatible_backend.py
  • uv run ruff check src/openrange/llm.py tests/test_openai_compatible_backend.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants