Skip to content

[Ready for Review] Adapter | Condense tutorial for human, create agent version separately#21

Open
crystalxyz wants to merge 2 commits intoharbor-framework:mainfrom
crystalxyz:adapter-tutorial-fix
Open

[Ready for Review] Adapter | Condense tutorial for human, create agent version separately#21
crystalxyz wants to merge 2 commits intoharbor-framework:mainfrom
crystalxyz:adapter-tutorial-fix

Conversation

@crystalxyz
Copy link

@crystalxyz crystalxyz commented Mar 25, 2026

Summary

This PR was motivated by a prior Harbor meeting discussion saying that the current harbor adapter tutorial is really long and hard to follow for human readers. In order to lower the entrance boundary and make it easier for people to contribute, we decide to condense the adapter tutorial for human readers so that they can easily keep track of their progress. Also, a separate agent-versioned tutorial is created so that contributors can easily use agents to build adapters.

New structure

  • /doc/datasets/adapter (Title: Adapters) -> Human version tutorial
  • /doc/datasets/adapter-ai (Title: Adapters (Agent Version)) -> Agent version tutorial
  • A block added at the beginning of the human tutorial to point agents to the other page

Details

  • The original adapter.mdx file was renamed to adapter-ai.mdx, and this file was modified to work better with agents. The key modifications include replacing the accordion UI with tables and lists; providing detailed field type and explanation for each json file; defining clear steps and validation criteria
  • A new adapter.mdx is created from the original file, but as a concise human-readable version

@crystalxyz crystalxyz changed the title Condense adapter tutorial for human readers, create agent version separately [Ready for Review] Adapter | Condense tutorial for human, create agent version separately Mar 25, 2026
Copy link

@Ternura143 Ternura143 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I only have a small issue that needs to be addressed.

| `split` | string | yes | Split name matching original. Use `"full"` if adapter works for all splits collectively. If different splits are registered/validated in different ways, split them out separately. |
| `adapted_benchmark_size` | int | yes | Number of tasks the adapter can convert. May differ from original if tasks were excluded for sufficient reasons documented in the README. |
| `parity_benchmark_size` | int | yes | Number of tasks used for parity. Equals `adapted_benchmark_size` if full set. |
| `parity_sampling_rate` | float | yes | `adapted_benchmark_size / parity_benchmark_size` |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parity_sampling_rate formula is inverted in the schema — should be parity_benchmark_size / adapted_benchmark_size, not adapted_benchmark_size / parity_benchmark_size.


### Step 3: Verify Oracle Solutions

Run the oracle agent on your entire dataset and confirm **100% reward on all tasks**.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For cases where the original benchmark has broken oracles, maybe we can advice the agent to document the tasks with oracle issues and file bugs to the upstream benchmark instead of attempting to fix them on the Harbor side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants