Skip to content

feat: BaseScenario — abstract interface for all domain benchmark scenarios #22

@Neal006

Description

@Neal006

Background

MemoryLens currently has two benchmark scenarios: default (generic tech Q&A) and edtech (student-tutor). Both are implemented as module-level constants. As new domain scenarios are added, we need a shared abstract interface so every scenario is structurally consistent, IDE-friendly (type-checked), and easy for contributors to discover and extend.

Key file to read first:

  • simulator/scenarios/edtech.py — the pattern this class will formalise

What to build

simulator/scenarios/base.py — new file

from abc import ABC, abstractmethod
from typing import List
from simulator.facts import Fact


class BaseScenario(ABC):
    """
    Abstract base class for all MemoryLens domain benchmark scenarios.

    A scenario bundles three things needed to run a benchmark in a specific domain:
    - facts       : the canonical set of facts to inject and test recall on
    - persona_pool: multiple fact-sets (one per persona) for multi-seed runs
    - filler_turns: domain-specific filler questions between fact injections

    To implement a new scenario, inherit from this class and add the three
    abstract properties.  See simulator/scenarios/edtech.py for a reference.
    """

    name: str = "base"
    description: str = ""

    @property
    @abstractmethod
    def facts(self) -> List[Fact]:
        """Return the canonical fact list for this scenario."""
        ...

    @property
    @abstractmethod
    def persona_pool(self) -> List[List[Fact]]:
        """Return 5 persona fact lists for multi-seed statistical validation."""
        ...

    @property
    @abstractmethod
    def filler_turns(self) -> List[str]:
        """Return >= 20 domain-specific filler questions."""
        ...

simulator/scenarios/__init__.py

  • Import and expose BaseScenario so contributors can do from simulator.scenarios import BaseScenario

Acceptance criteria

  • simulator/scenarios/base.py exists with BaseScenario ABC
  • BaseScenario has facts, persona_pool, filler_turns as abstract properties
  • Importable: from simulator.scenarios.base import BaseScenario
  • Exposed from package: from simulator.scenarios import BaseScenario
  • Existing edtech.py is not modified (backward compat)
  • All existing tests still pass

Technical hints

  • Use abc.ABC and @property @abstractmethod — same pattern as memory/base.py
  • The class body is ~30 lines including docstrings — keep it minimal
  • Fact is imported from simulator.facts; avoid circular imports by not importing scenario subclasses here

Getting started

pip install -e ".[dev]"
pytest tests/ -v   # confirm baseline

# Create simulator/scenarios/base.py
# Run:
python -c "from simulator.scenarios.base import BaseScenario; print('BaseScenario OK')"

This is a ~30-line file and a perfect standalone first contribution to the project.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions