feat(proxy): add max_parallel_requests for model/user/team levels

[peterkc]

Summary

Implements per-model, per-user, and per-team max_parallel_requests limits in the proxy rate limiter, addressing three TODOs in parallel_request_limiter.py.

• Model-level: Configurable via key metadata (model_max_parallel_requests) or model_max_budget
• User-level: New user_max_parallel_requests field on UserAPIKeyAuth
• Team-level: New team_max_parallel_requests field on UserAPIKeyAuth

Changes

File	Change
auth_utils.py	Added get_key_model_max_parallel_requests() helper
_types.py	Added user_max_parallel_requests and team_max_parallel_requests fields
parallel_request_limiter.py	Wired all 3 TODO locations (v1 limiter)
parallel_request_limiter_v3.py	Added model-level support to v3 limiter
test_max_parallel_requests.py	10 unit tests

Documentation

📄 Model-Specific Max Parallel Requests - New doc with:

• Quick start guide
• Key, team, and priority override examples
• Mermaid flowchart showing priority resolution

Config Examples

Model-level (via API key metadata):

{
  "model_max_parallel_requests": {
    "gpt-4": 5,
    "gpt-3.5-turbo": 20
  }
}

User-level: Set user_max_parallel_requests on user record
Team-level: Set team_max_parallel_requests on team record

Test plan

• Unit tests pass (10/10)
• v3 limiter support added
• Documentation added
• CI passes

Closes #17824

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Review	Updated (UTC)
litellm	Error		Dec 13, 2025 11:27pm

[krrishdholakia]

@peterkc can you add a loom of key model max parallel requests working as expected?

[JiangJiaWei1103]

Over LGTM. I'll test this integration in our use case. Thanks!

[peterkc]

@krrishdholakia Happy to add a Loom demo! A couple of quick clarifications to make sure I cover what you're looking for:

v3 Limiter Support
While testing, I noticed the PR modifies parallel_request_limiter.py (v1), but the default is v3 (parallel_request_limiter_v3.py). I've drafted a small patch (~15 lines) to port model-level max_parallel_requests to v3 as well. Should I:

1. Include the v3 port in this PR? (Recommended—ensures feature works out of the box)
2. Demo with v1 only using LEGACY_MULTI_INSTANCE_RATE_LIMITING=true?

Demo Scope
Planning to show:

• Key-level model limits (metadata.model_max_parallel_requests)
• Model isolation (GPT-4 limit=2 doesn't affect GPT-3.5 limit=5)
• 429 responses when limits are exceeded

Want me to also include team-level limits and the priority override (key > team)?