Skip to content

Create evaluation matrix mapping approaches to requirements #47

@olaservo

Description

@olaservo

Background

Approaches documents six approaches with community input, and experimental findings captures implementation results. But there is no structured way to compare approaches against a shared set of requirements. This makes it difficult to identify where approaches genuinely diverge, where they overlap, and which gaps remain.

This issue depends on having a requirements document (#46) but can be started in parallel using draft requirements.

Scope

Create a section in docs/approaches.md (or a standalone docs/evaluation-matrix.md) containing:

  1. A matrix table with:

    • Rows: Requirements (from docs/requirements.md)
    • Columns: Approaches 1–6
    • Cells: ✅ (satisfies), ⚠️ (partially / with caveats), ❌ (does not satisfy), ❓ (untested)
  2. Notes per cell explaining the rating, especially for ⚠️ and ❌ — what's missing or what caveat applies.

  3. A summary section identifying:

    • Requirements satisfied by all approaches (consensus areas — these can be convention-ized now)
    • Requirements where approaches diverge (these are the real decision points)
    • Requirements no approach currently satisfies (gaps that need new work)
  4. Mapping to experimental evidence — where findings from Compare skill delivery mechanisms: file-based vs MCP-based #37, Survey client resource-loading support across major MCP clients #38, Build client-side reference implementation for model-driven resource loading #40, Build server-side reference implementation for skills-as-resources #41, Test skills-via-sampling approach (Approach 3 variant) #42 provide data, reference them.

Acceptance Criteria

  • Matrix covering at least 10 requirements × 6 approaches
  • Each ⚠️ or ❌ cell has a brief rationale
  • Summary section identifies consensus areas, divergence points, and gaps
  • Reviewed by at least 2 IG facilitators
  • Referenced from the convention draft (Draft MCP Skills Convention v0.1 #43) as decision support

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    conventionRelates to MCP Skills Convention v0.1 effortdocumentationImprovements or additions to documentation

    Type

    No type

    Projects

    Status

    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions