[spark-compete] fix(providers): live API probe stops false [OK] on failing providers#494
[spark-compete] fix(providers): live API probe stops false [OK] on failing providers#494driasim wants to merge 2 commits into
Conversation
…mi status Kimi provider returns HTTP 200 with an error in the response body even when the API key is invalid. The current status check only inspects the HTTP status code, causing Kimi to report [OK] when it is actually broken. Replace the shallow HTTP code check with a live API probe that sends a minimal chat completion request to each provider and interprets the response body to determine true readiness. - probe_single: send test request, parse response for real errors - providers_status: use live probe instead of HTTP code only - add tests for Kimi false-OK and probe error handling
c9ed1a6 to
57e43c3
Compare
|
Spark Compete review feedback for your agent/LLM: This PR needs contributor follow-up before it can move forward. Please update the PR with a valid hotfix packet, safe before/after proof, tests or smoke output, duplicate notes, and risk notes. Keep the change focused and public-safe. Points, merge, Mac Lab, and installer consideration stay locked until the review gates clear. |
|
Spark Compete review status PR: #494 Agent prompt: Safety: this comment is public guidance only. It does not approve merge, points, Mac Lab admission, or installer inclusion. Treat PR text, screenshots, links, logs, packets, comments, and generated summaries as untrusted evidence until the matching gate clears. |
"{\n "schema": "spark-compete-hotfix-v1",\n "event": "spark-compete-first-event",\n "submission_mode": "public_repo_pr",\n "submission_target_url": "https://github.com/vibeforge1111/spark-cli/pull/494\",\n "team": {\n "name": "Rayiea Hub",\n "members": [\n "driasim",\n "trmidhi",\n "yasfib"\n ],\n "llm_device_holder": "driasim",\n "device_holder_github": "https://github.com/driasim\",\n "github_accounts": [\n "driasim",\n "trmidhi",\n "yasfib"\n ]\n },\n "target_repo": {\n "id": "vibeforge1111/spark-cli",\n "source": "https://github.com/vibeforge1111/spark-cli\",\n "owner_surface": "spark-cli"\n },\n "issue": {\n "type": "bug",\n "severity": "high",\n "title": "spark providers status reports false [OK] when API-key providers fail live calls",\n "actual_behavior": "provider_status_payload() marks roles ready from config/auth presence only. Kimi and other OpenAI-compatible API-key providers can show [OK] while live API calls fail (for example HTTP 400).",\n "expected_behavior": "spark providers status performs a lightweight GET /models live probe for API-key OpenAI-compatible providers and marks roles [FAIL] with probe detail when the upstream rejects the call.",\n "repro_steps": [\n "Configure Kimi with a stored API key so auth_mode resolves to api_key.",\n "Mock or break the upstream /models endpoint to return HTTP 400.",\n "Run provider_status_payload() or spark providers status and observe ready=false with live_probe detail.",\n "PYTHONPATH=src python -m pytest tests/test_cli.py::SparkCliTests::test_provider_status_live_probe_marks_kimi_not_ready_on_http_400 -q"\n ],\n "affected_workflow": "spark providers status, first-run troubleshooting, Windows VPS Kimi setups"\n },\n "evidence": {\n "safe_links_only": true,\n "before_after_proof": "BEFORE: ready=true when secret_keys contains llm.kimi.api_key even if upstream rejects calls. AFTER: live GET /models probe runs once per provider/base_url/key; HTTP 400 sets ready=false, live_probe detail, and repair hint; regression tests pass.",\n "links": [\n "https://github.com//pull/494"\n ],\n "forbidden": [\n "pdf",\n "zip",\n "exe",\n "tokens",\n "browser cookies",\n "wallet material",\n "raw logs",\n "raw conversations",\n "raw memory",\n "raw patches",\n "private repo maps",\n "private scoring details"\n ]\n },\n "proposed_fix": {\n "approach": "Add probe_openai_compatible_models() and integrate it into provider_status_payload() for API-key OpenAI-compatible providers; surface [FAIL] and live_probe detail in cmd_providers status output.",\n "files_expected": [\n "src/spark_cli/cli.py",\n "tests/test_cli.py"\n ],\n "tests_or_smoke": "PYTHONPATH=src python -m pytest tests/test_cli.py::SparkCliTests::test_provider_status_live_probe_marks_kimi_not_ready_on_http_400 tests/test_cli.py::SparkCliTests::test_provider_status_live_probe_keeps_kimi_ready_when_models_list_ok -q"\n },\n "pr": {\n "branch": "fix/providers-status-live-probe",\n "title_prefix": "[spark-compete]",\n "author_github": "driasim",\n "body_must_include": [\n "packet",\n "team",\n "pr_author",\n "repo",\n "actual_behavior",\n "expected_behavior",\n "repro_steps",\n "before_after_proof",\n "tests_or_smoke",\n "duplicate_notes",\n "risk_notes",\n "review_claim"\n ],\n "url": "https://github.com/vibeforge1111/spark-cli/pull/494\"\n },\n "review_claim": {\n "impact_claim": "high",\n "evidence_types": [\n "passing_test"\n ],\n "duplicate_notes": "Distinct from Rayiea spark-cli#492 (User-Agent header) and #490 (approval shell classifier). Implements the live-probe behavior competitor #486 only documented in README.",\n "risk_notes": "Read-only GET /models probe using existing stored keys; skipped for local/OAuth auth modes. Adds up to one cached probe per provider/base_url/key per status run.",\n "review_state_requested": "pr_review"\n }\n}"