Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,31 @@ Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

---

## [v1.3.3] — 2026-04-05

### Added

- **Assignment Score Telemetry (`src/ticket_assigner.py`)**: The ticket assigner now records per-assignment scoring breakdowns (skill, stress, centrality, composite) into a MongoDB `assignment_scores` collection for downstream analysis.
- **Department Expertise Defaults (`src/utils/persona_utils.py`)**: Added a `DEPARTMENT_EXPERTISE_DEFAULTS` map so personas without explicit expertise lists fall back to sensible department-level defaults rather than a generic "general engineering" placeholder.
- **Departure Department Field (`config/config.yaml`)**: `org_lifecycle` departure entries now include a `dept` field, making ownership handoff tracking more precise during engineer exits.
- **Selective Artifact Regeneration (`eval/export_to_hf.py`)**: Added an `--only` CLI flag to selectively regenerate specific artifact types (`nps`, `invoices`, `datadog`) without a full corpus rebuild.

### Changed

- **On-Call Capacity Propagation (`src/ticket_assigner.py`)**: On-call status is now passed dynamically into `_compute_capacity` via an `on_call` parameter instead of being read from a static config key, keeping capacity calculations consistent with the runtime rotation.
- **Persona Fallback Logic (`src/utils/persona_utils.py`)**: Switched persona lookup from `PERSONAS.get(name, DEFAULT_PERSONA)` to `PERSONAS.get(name) or DEFAULT_PERSONA` to correctly handle personas with falsy but present entries.

### Removed

- **Agentic Eval Harness (`eval/agentic_eval_harness.py`)**: Removed the standalone agentic evaluation harness (2,676 lines). Evaluation logic is now consolidated elsewhere.
- **Standalone Eval Docs (`EVAL.md`, `README.md`)**: Removed the `EVAL.md` file and the corresponding Evaluation & Benchmarking section from `README.md`. Evaluation documentation will be maintained separately.

### Fixed

- **External Email Drop Test (`tests/test_external_email.py`)**: Refactored the customer email drop probability test to use a fully-specified source fixture and a mocked `_derive_customer_email_signals`, replacing the brittle direct `_sources` injection.

---

## [v1.3.2] — 2026-03-31

### Added
Expand Down
103 changes: 0 additions & 103 deletions EVAL.md

This file was deleted.

15 changes: 2 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@

### A deterministic corporate simulator for generating ground-truth ecosystems and evaluating enterprise AI agents

![OrgForge corpus overview](orgforge_hero.png)

OrgForge simulates weeks of realistic enterprise activity — Confluence pages, JIRA tickets, Slack threads, Git PRs, Zoom transcripts, Zendesk tickets, Salesforce records, emails, and server telemetry — grounded in an event-driven state machine so LLMs can't hallucinate facts out of sequence.

The dataset is the exhaust of a living simulation. Engineers leave mid-sprint, forcing deterministic incident handoffs, ticket reassignments, and CRM ownership lapses. Knowledge gaps surface when under-documented systems break. New hires build their internal network through simulated collaboration. Stress propagates through a live, weighted social graph. Every artifact reflects the exact state of the org at the moment it was written.
Expand Down Expand Up @@ -35,7 +37,6 @@ The dataset is the exhaust of a living simulation. Engineers leave mid-sprint, f
- [How the Event Bus Works](#how-the-event-bus-works)
- [Memory Requirements](#memory-requirements)
- [Project Structure](#project-structure)
- [Evaluation & Benchmarking](#-evaluation--benchmarking)
- [Roadmap](#roadmap)
- [Adding a New Artifact Type](#adding-a-new-artifact-type)
- [Contributing](#contributing)
Expand Down Expand Up @@ -380,18 +381,6 @@ orgforge/

---

### 🧪 Evaluation & Benchmarking

OrgForge includes a full-stack evaluation harness to measure how well AI agents retrieve and reason over the generated corporate data.

- **Deterministic Ground Truth**: All answers are derived from the simulation’s state machine, not LLM hallucinations.
- **Multi-Hop Reasoning**: Test agents on causal, temporal, and gap-detection questions.
- **End-to-End Testing**: Use `eval_e2e.py` to run full RAG pipelines against providers like AWS Bedrock, OpenAI, and Cohere.

For detailed instructions on generating eval sets, running benchmarks, and interpreting scores, see **[EVAL.md](#EVAL.md)**.

---

## Roadmap

- [x] Native integrations for Zoom, Zendesk, and Salesforce CRM
Expand Down
2 changes: 2 additions & 0 deletions config/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ org_lifecycle:
day: 12
reason: "voluntary" # voluntary | layoff | performance
role: "Senior Backend Engineer"
dept: "Engineering_Backend"
knowledge_domains:
- "auth-service"
- "redis-cache"
Expand All @@ -164,6 +165,7 @@ org_lifecycle:
day: 24
reason: "layoff"
role: "DevOps Engineer"
dept: "Engineering_Backend"
knowledge_domains:
- "kubernetes-deploy"
- "terraform-infra"
Expand Down
Loading
Loading