Skip to content

Conversation

@ltiao
Copy link
Contributor

@ltiao ltiao commented Feb 10, 2026

Summary:
We have a method get_opt_trace_by_steps that was used extensively during our Ax 1.0 benchmarking campaign. It duplicates the basic logic of get_trace but differs in that it operates along (trial_index, MAP_KEY) pairs and respects ordering by timestamp (i.e. chronological order). However, it is limited to single-objective unconstrained problems, and our current needs (multi-objective and/or constrained) have outgrown it. We reconcile the two by extracting three core building blocks of get_trace:

  1. _pivot_data_with_feasibility: Pivots data to wide format with feasibility information and metric completeness checks.
  2. _compute_trace_values: Computes per-observation trace values (hypervolume for MOO, objective value for SOO), with cumulative best support.
  3. _aggregate_and_cumulate_trace: Aggregates values by groups and computes the cumulative best across groups.

These are implemented in a more general way that respects arbitrary groupings and orderings. We then refactor get_trace (and its helpers _prepare_data_for_trace and get_trace_by_arm_pull_from_data) to use these building blocks, and leverage them in get_opt_trace_by_steps to extend its support to multi-objective and constrained problems.

Additionally:

  • The timestamp-based sorting in get_opt_trace_by_steps is preserved, which is critical for correct cumulative hypervolume computation (without this, observations would be processed in (trial_index, arm_name, MAP_KEY) order instead of chronological order).
  • Tests are updated to replace NotImplementedError checks with actual MOO and constrained test cases that verify correctness of the new functionality.

Differential Revision: D79581270

@meta-cla meta-cla bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Feb 10, 2026
@meta-codesync
Copy link

meta-codesync bot commented Feb 10, 2026

@ltiao has exported this pull request. If you are a Meta employee, you can view the originating Diff in D79581270.

…rained (facebook#4884)

Summary:

We have a method `get_opt_trace_by_steps` that was used extensively during our Ax 1.0 benchmarking campaign. It duplicates the basic logic of `get_trace` but differs in that it operates along `(trial_index, MAP_KEY)` pairs and respects ordering by timestamp (i.e. chronological order). However, it is limited to single-objective unconstrained problems, and our current needs (multi-objective and/or constrained) have outgrown it. We reconcile the two by extracting three core building blocks of `get_trace`:

1. `_pivot_data_with_feasibility`: Pivots data to wide format with feasibility information and metric completeness checks.
2. `_compute_trace_values`: Computes per-observation trace values (hypervolume for MOO, objective value for SOO), with cumulative best support.
3. `_aggregate_and_cumulate_trace`: Aggregates values by groups and computes the cumulative best across groups.

These are implemented in a more general way that respects arbitrary groupings and orderings. We then refactor `get_trace` (and its helpers `_prepare_data_for_trace` and `get_trace_by_arm_pull_from_data`) to use these building blocks, and leverage them in `get_opt_trace_by_steps` to extend its support to multi-objective and constrained problems.

Additionally:
- The timestamp-based sorting in `get_opt_trace_by_steps` is preserved, which is critical for correct cumulative hypervolume computation (without this, observations would be processed in `(trial_index, arm_name, MAP_KEY)` order instead of chronological order).
- Tests are updated to replace `NotImplementedError` checks with actual MOO and constrained test cases that verify correctness of the new functionality.

Differential Revision: D79581270
@codecov-commenter
Copy link

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.76%. Comparing base (8a706b8) to head (e18d69f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4884   +/-   ##
=======================================
  Coverage   96.76%   96.76%           
=======================================
  Files         591      591           
  Lines       61955    61969   +14     
=======================================
+ Hits        59948    59962   +14     
  Misses       2007     2007           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed Do not delete this pull request or issue due to inactivity. fb-exported meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants