Skip to content

Feat: Add model-compatibility.json registry for self-hostable models #329

@fuleinist

Description

@fuleinist

Problem

Issue #235 asks which smaller self-hostable models work with browser-harness (e.g. qwen3.6 35b-a3b). There is no documented compatibility matrix. Users deploying on limited hardware or air-gapped environments have no way to know what will actually work before spending hours on setup.

Proposal

Add a model-compatibility.json file at the root of the repository that lists tested models with compatibility metadata:

[
  {
    "model": "qwen3.6 35b-a3b",
    "provider": "ollama",
    "status": "verified",
    "notes": "Works well for simple click/type tasks. Struggles with complex multi-step flows.",
    "last_tested": "2026-05-01"
  },
  {
    "model": "mistral-small",
    "provider": "anyscale",
    "status": "verified",
    "notes": "Good balance of speed and capability for headless deployments.",
    "last_tested": "2026-05-01"
  }
]

Status values: verified | works | unknown | broken

Also add a lightweight CLI command:

browser-harness models list          # show all models
browser-harness models list --min-size 35b  # filter by parameter size
browser-harness models list --status verified  # only verified

And a --model-info <name> flag to get details on a specific model.

Alternatives considered

  • Wiki/docs page: harder to keep in sync, no machine-readable format
  • Just answering Supported/verified models lists #235 inline: does not scale, the question will keep being asked
  • Rely on user reports: no structured record of what has been tested

Before submitting

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions