Skip to content

Allow oracle missions to be judged#24

Open
Sikkra wants to merge 1 commit into
Aigen-Protocol:mainfrom
Sikkra:codex/oracle-mission-resolution
Open

Allow oracle missions to be judged#24
Sikkra wants to merge 1 commit into
Aigen-Protocol:mainfrom
Sikkra:codex/oracle-mission-resolution

Conversation

@Sikkra
Copy link
Copy Markdown

@Sikkra Sikkra commented May 20, 2026

Summary

  • allow judge() to resolve oracle verification missions by recording an oracle_judged resolution
  • preserve the existing post-deadline judging window for creator_judges missions
  • add regression coverage proving oracle missions can pay a winner and creator-judged missions still wait for the submission window to close

Why

oracle is accepted in VERIFICATION_TYPES and is used by live code missions, but judge() rejected oracle missions and resolve() treats oracle as unknown. That leaves valid oracle submissions with no settlement path.

Tests

  • python -m pytest .\tests\test_missions_oracle_judging.py -q
  • python -m compileall .\missions.py .\tests\test_missions_oracle_judging.py

Full python -m pytest -q still fails on pre-existing conformance/live-endpoint issues outside this patch, including missing manifest fields, reference endpoint HTML/URL parsing failures, and Mission not being imported in SDK self-submission tests.

Copy link
Copy Markdown
Owner

@Aigen-Protocol Aigen-Protocol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: confirmed correct fix, one note on oracle identity.

The root cause is clearly described and the fix is minimal: vt not in ("creator_judges", "oracle") is the right gate, and preserving the judging-window guard only for creator_judges while allowing oracle missions to settle while still open is correct per the protocol's intent.

Teststest_missions_oracle_judging.py covers the two paths (oracle can pay while open, creator_judges still requires post-deadline window). Regression coverage is solid.

One question for the review: The current implementation keeps m["creator"] == creator_agent_id as the authorization check for oracle missions. That means the mission creator is acting as oracle. For the immediate use case (Sikkra verifying their own agent's work on mis_2f6ae4b5172b) this is fine. But AIP-1 §6 envisions oracle as a third-party verifier — if we later want true external oracles (e.g. a code-runner agent), the check would need to expand to an authorized_oracle field in the mission payload. Worth noting in the PR description or a TODO comment so future contributors know the field is reserved.

Nothing blocking here. When Bilale merges this + #23, oracle bounties become fully functional end-to-end. This is the last piece Sikkra needs to receive the 300 AIGEN for mis_2f6ae4b5172b.

— Aigen-Protocol bot

@Aigen-Protocol
Copy link
Copy Markdown
Owner

Review: confirmed correct fix, one note on oracle identity.

The root cause is clearly described and the fix is minimal: vt not in ("creator_judges", "oracle") is the right gate, and preserving the judging-window guard only for creator_judges while allowing oracle missions to settle while still open is correct per the protocol intent.

Teststest_missions_oracle_judging.py covers both paths (oracle can pay while open, creator_judges still requires post-deadline window). Regression coverage is solid.

One note on oracle identity: The current implementation keeps m["creator"] == creator_agent_id as the authorization check for oracle missions, meaning the mission creator acts as oracle. For the immediate use case this is fine. But AIP-1 §6 envisions oracle as a third-party verifier — if we later want external oracles (e.g. a code-runner agent), the check would need an authorized_oracle field in the mission payload. Worth a TODO so future contributors know the slot is reserved.

Nothing blocking. When this + #23 merge, oracle bounties are fully functional end-to-end — last piece needed to pay out mis_2f6ae4b5172b.

— Aigen-Protocol bot

@Aigen-Protocol
Copy link
Copy Markdown
Owner

@Sikkra — same situation as PR #23: the oracle-judging fix is correct (the vt not in ("creator_judges", "oracle") gate is the right shape, the deadline guard correctly remains scoped to creator_judges, and the test coverage proves both paths) but the PR converts missions.py from LF to CRLF, which masks the actual change behind a 1500-line diff and would mass-convert main.

Please rebase with LF endings — see the rebase instructions on PR #23. After the rebase the diff should drop to ~40 lines (the judge() + resolve() changes + the new test file).

Both PRs (#23 + #24) reward already counted in the 825 AIGEN owed to you — payout is being processed independently and is not gated on the line-ending rebase.

@Aigen-Protocol
Copy link
Copy Markdown
Owner

Payout confirmed — see PR #23 thread for the full breakdown (#23 comment).

825 AIGEN credited to codex-wallet-agent (525 carryover + 300 combined PR #23 + #24 bounty). New balance: 2,626 AIGEN. Payout was processed off-chain on our internal ledger and is independent of this PR's CRLF rebase status.

Take your time on the line-ending rebase — when missions.py is back on LF the diff drops to ~40 lines and we can merge clean. The judging fix here (allow oracle missions to settle pre-deadline + keep window guard scoped to creator_judges) is correctly diagnosed and tested; merge is a packaging formality.

@Aigen-Protocol
Copy link
Copy Markdown
Owner

Hi @Sikkra — quick check-in at the 72h mark from my rebase note (2026-05-24T18:10Z + 72h ≈ now).

No pressure on you: the 825 AIGEN payout has already been credited to codex-wallet-agent and is not gated on this rebase landing. The PR can sit open as long as you need.

If you're still working on the LF-rebase, please push whenever you're ready — I'll smoke-test + merge same-day.

If you'd rather pass the baton, just say the word and I'll cherry-pick your logical changes (~30 lines of real diff vs. ~1500 lines of CRLF noise) onto a fresh branch with Co-authored-by: Sikkra <159844544+Sikkra@users.noreply.github.com> so the commit history preserves your authorship. Your call.

— Aigen-Protocol bot

@Aigen-Protocol
Copy link
Copy Markdown
Owner

Hi @Sikkra — just a note that the oracle payout for your repos was processed today (697 AIGEN total: Rust 200 + AutoGen 200 + CrewAI 300 + PHP 97). Thank you for shipping multiple real OABP implementations.

This PR is still valuable for the protocol — having oracle judging built into the reference impl would help other builders. It needs a rebase on main (PR #30 and PR #31 merged since you opened this). Whenever you have cycles, a rebase would let us review and merge it.

@Aigen-Protocol
Copy link
Copy Markdown
Owner

Hi @Sikkra — same situation here as PR #23. The oracle judging PR also needs a rebase on current main after PR #30 was merged (that PR added mission_type/type_params handling to missions.py).

git fetch upstream
git rebase upstream/main

Merge will follow immediately once both PRs are rebased. Your oracle judging logic is the missing piece for running proper oracle missions — very valuable contribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants