[Ready for Review] Adapter | Condense tutorial for human, create agent version separately#21
Open
crystalxyz wants to merge 2 commits intoharbor-framework:mainfrom
Open
[Ready for Review] Adapter | Condense tutorial for human, create agent version separately#21crystalxyz wants to merge 2 commits intoharbor-framework:mainfrom
crystalxyz wants to merge 2 commits intoharbor-framework:mainfrom
Conversation
Ternura143
suggested changes
Mar 25, 2026
Ternura143
left a comment
There was a problem hiding this comment.
LGTM! I only have a small issue that needs to be addressed.
| | `split` | string | yes | Split name matching original. Use `"full"` if adapter works for all splits collectively. If different splits are registered/validated in different ways, split them out separately. | | ||
| | `adapted_benchmark_size` | int | yes | Number of tasks the adapter can convert. May differ from original if tasks were excluded for sufficient reasons documented in the README. | | ||
| | `parity_benchmark_size` | int | yes | Number of tasks used for parity. Equals `adapted_benchmark_size` if full set. | | ||
| | `parity_sampling_rate` | float | yes | `adapted_benchmark_size / parity_benchmark_size` | |
There was a problem hiding this comment.
parity_sampling_rate formula is inverted in the schema — should be parity_benchmark_size / adapted_benchmark_size, not adapted_benchmark_size / parity_benchmark_size.
AlienKevin
reviewed
Mar 26, 2026
|
|
||
| ### Step 3: Verify Oracle Solutions | ||
|
|
||
| Run the oracle agent on your entire dataset and confirm **100% reward on all tasks**. |
There was a problem hiding this comment.
For cases where the original benchmark has broken oracles, maybe we can advice the agent to document the tasks with oracle issues and file bugs to the upstream benchmark instead of attempting to fix them on the Harbor side.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR was motivated by a prior Harbor meeting discussion saying that the current harbor adapter tutorial is really long and hard to follow for human readers. In order to lower the entrance boundary and make it easier for people to contribute, we decide to condense the adapter tutorial for human readers so that they can easily keep track of their progress. Also, a separate agent-versioned tutorial is created so that contributors can easily use agents to build adapters.
New structure
/doc/datasets/adapter(Title:Adapters) -> Human version tutorial/doc/datasets/adapter-ai(Title:Adapters (Agent Version)) -> Agent version tutorialDetails
adapter.mdxfile was renamed toadapter-ai.mdx, and this file was modified to work better with agents. The key modifications include replacing the accordion UI with tables and lists; providing detailed field type and explanation for eachjsonfile; defining clear steps and validation criteriaadapter.mdxis created from the original file, but as a concise human-readable version