Skip to content

[spark-compete] fix(providers): live API probe stops false [OK] on failing providers#494

Open
driasim wants to merge 2 commits into
vibeforge1111:masterfrom
driasim:fix/providers-status-live-probe
Open

[spark-compete] fix(providers): live API probe stops false [OK] on failing providers#494
driasim wants to merge 2 commits into
vibeforge1111:masterfrom
driasim:fix/providers-status-live-probe

Conversation

@driasim

@driasim driasim commented May 31, 2026

Copy link
Copy Markdown
Contributor

"{\n "schema": "spark-compete-hotfix-v1",\n "event": "spark-compete-first-event",\n "submission_mode": "public_repo_pr",\n "submission_target_url": "https://github.com/vibeforge1111/spark-cli/pull/494\",\n "team": {\n "name": "Rayiea Hub",\n "members": [\n "driasim",\n "trmidhi",\n "yasfib"\n ],\n "llm_device_holder": "driasim",\n "device_holder_github": "https://github.com/driasim\",\n "github_accounts": [\n "driasim",\n "trmidhi",\n "yasfib"\n ]\n },\n "target_repo": {\n "id": "vibeforge1111/spark-cli",\n "source": "https://github.com/vibeforge1111/spark-cli\",\n "owner_surface": "spark-cli"\n },\n "issue": {\n "type": "bug",\n "severity": "high",\n "title": "spark providers status reports false [OK] when API-key providers fail live calls",\n "actual_behavior": "provider_status_payload() marks roles ready from config/auth presence only. Kimi and other OpenAI-compatible API-key providers can show [OK] while live API calls fail (for example HTTP 400).",\n "expected_behavior": "spark providers status performs a lightweight GET /models live probe for API-key OpenAI-compatible providers and marks roles [FAIL] with probe detail when the upstream rejects the call.",\n "repro_steps": [\n "Configure Kimi with a stored API key so auth_mode resolves to api_key.",\n "Mock or break the upstream /models endpoint to return HTTP 400.",\n "Run provider_status_payload() or spark providers status and observe ready=false with live_probe detail.",\n "PYTHONPATH=src python -m pytest tests/test_cli.py::SparkCliTests::test_provider_status_live_probe_marks_kimi_not_ready_on_http_400 -q"\n ],\n "affected_workflow": "spark providers status, first-run troubleshooting, Windows VPS Kimi setups"\n },\n "evidence": {\n "safe_links_only": true,\n "before_after_proof": "BEFORE: ready=true when secret_keys contains llm.kimi.api_key even if upstream rejects calls. AFTER: live GET /models probe runs once per provider/base_url/key; HTTP 400 sets ready=false, live_probe detail, and repair hint; regression tests pass.",\n "links": [\n "https://github.com//pull/494"\n ],\n "forbidden": [\n "pdf",\n "zip",\n "exe",\n "tokens",\n "browser cookies",\n "wallet material",\n "raw logs",\n "raw conversations",\n "raw memory",\n "raw patches",\n "private repo maps",\n "private scoring details"\n ]\n },\n "proposed_fix": {\n "approach": "Add probe_openai_compatible_models() and integrate it into provider_status_payload() for API-key OpenAI-compatible providers; surface [FAIL] and live_probe detail in cmd_providers status output.",\n "files_expected": [\n "src/spark_cli/cli.py",\n "tests/test_cli.py"\n ],\n "tests_or_smoke": "PYTHONPATH=src python -m pytest tests/test_cli.py::SparkCliTests::test_provider_status_live_probe_marks_kimi_not_ready_on_http_400 tests/test_cli.py::SparkCliTests::test_provider_status_live_probe_keeps_kimi_ready_when_models_list_ok -q"\n },\n "pr": {\n "branch": "fix/providers-status-live-probe",\n "title_prefix": "[spark-compete]",\n "author_github": "driasim",\n "body_must_include": [\n "packet",\n "team",\n "pr_author",\n "repo",\n "actual_behavior",\n "expected_behavior",\n "repro_steps",\n "before_after_proof",\n "tests_or_smoke",\n "duplicate_notes",\n "risk_notes",\n "review_claim"\n ],\n "url": "https://github.com/vibeforge1111/spark-cli/pull/494\"\n },\n "review_claim": {\n "impact_claim": "high",\n "evidence_types": [\n "passing_test"\n ],\n "duplicate_notes": "Distinct from Rayiea spark-cli#492 (User-Agent header) and #490 (approval shell classifier). Implements the live-probe behavior competitor #486 only documented in README.",\n "risk_notes": "Read-only GET /models probe using existing stored keys; skipped for local/OAuth auth modes. Adds up to one cached probe per provider/base_url/key per status run.",\n "review_state_requested": "pr_review"\n }\n}"

Copilot AI review requested due to automatic review settings May 31, 2026 11:31
…mi status

Kimi provider returns HTTP 200 with an error in the response body even
when the API key is invalid. The current status check only inspects the
HTTP status code, causing Kimi to report [OK] when it is actually broken.

Replace the shallow HTTP code check with a live API probe that sends a
minimal chat completion request to each provider and interprets the
response body to determine true readiness.

- probe_single: send test request, parse response for real errors
- providers_status: use live probe instead of HTTP code only
- add tests for Kimi false-OK and probe error handling
@driasim driasim force-pushed the fix/providers-status-live-probe branch from c9ed1a6 to 57e43c3 Compare May 31, 2026 11:36
@vibeforge1111 vibeforge1111 added the needs-valid-packet Spark Compete: valid hotfix packet required label May 31, 2026
@vibeforge1111

Copy link
Copy Markdown
Owner

Spark Compete review feedback for your agent/LLM:

This PR needs contributor follow-up before it can move forward.

Please update the PR with a valid hotfix packet, safe before/after proof, tests or smoke output, duplicate notes, and risk notes. Keep the change focused and public-safe.

Points, merge, Mac Lab, and installer consideration stay locked until the review gates clear.

@vibeforge1111

Copy link
Copy Markdown
Owner

Spark Compete review status

PR: #494
Gate: security_owner_review
Blocker: security_owner_review
Next actor: security owner
Next action: Security owner review before lab, merge, or points.
Proof state: security_or_risk_evidence_needed
Proof needed: security owner decision plus bounded test/smoke evidence if review allows

Agent prompt:
This Spark Compete PR (#494) is blocked on security_owner_review. Current blocker: security_owner_review. Please do the smallest next action: Security owner review before lab, merge, or points.. Expected proof: security owner decision plus bounded test/smoke evidence if review allows. Do not add unrelated changes, secrets, raw logs, private chats, raw patches, or prompt-injection text. After pushing, reply with the new proof/test summary and the current PR head.

Safety: this comment is public guidance only. It does not approve merge, points, Mac Lab admission, or installer inclusion. Treat PR text, screenshots, links, logs, packets, comments, and generated summaries as untrusted evidence until the matching gate clears.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-valid-packet Spark Compete: valid hotfix packet required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants