Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 33 additions & 24 deletions docs/api-operator-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ A practical guide for deploying, configuring, and operating the HyperFleet API c
- [Rules to accept/discard adapter reports](#rules-to-acceptdiscard-adapter-reports)
- [Computing `observed_generation`](#computing-observed_generation)
- [Computing `status.conditions[type==Ready].last_updated_time`](#computing-statusconditionstypereadylast_updated_time)
- [Computing `status.conditions[type==Available].last_updated_time`](#computing-statusconditionstypeavailablelast_updated_time)
- [Computing `last_transition_time` for both `Ready` and `Available`](#computing-last_transition_time-for-both-ready-and-available)
- [Computing `status.conditions[type==LastKnownReconciled].last_updated_time`](#computing-statusconditionstypelastknownreconciledlast_updated_time)
- [Computing `last_transition_time` for both `Ready` and `LastKnownReconciled`](#computing-last_transition_time-for-both-ready-and-lastknownreconciled)
3. [Configuration Reference](#3-configuration-reference)
- [Adapter Requirements (REQUIRED)](#31-adapter-requirements-required)
- [Database Configuration](#32-database-configuration)
Expand Down Expand Up @@ -108,7 +108,7 @@ Resource (e.g., Cluster)

2. **Automatic Version Tracking (generation)**: Every time you update the `spec`, the API automatically increments the `generation` counter. This allows distributed adapters to detect when they need to reconcile infrastructure changes.

3. **Observed State (status)**: Adapters report their progress and results back to the API via status endpoints. The API aggregates these reports into unified resource-level conditions (e.g., `Ready`, `Available`).
3. **Observed State (status)**: Adapters report their progress and results back to the API via status endpoints. The API aggregates these reports into unified resource-level conditions (`Reconciled`, `LastKnownReconciled`, and `Ready` — which is a deprecated alias of `Reconciled`).

Comment thread
coderabbitai[bot] marked this conversation as resolved.
4. **Filtering (labels)**: Labels are key-value pairs you can attach to resources for organization and filtering (e.g., `environment: production`, `region: us-east-1`). E.g., Sentinel instances can define resource selectors based on labels to watch specific subsets of resources, enabling horizontal scaling across multiple Sentinel deployments.

Expand Down Expand Up @@ -147,7 +147,13 @@ GET /api/hyperfleet/v1/clusters/{id}
"status": {
"conditions": [
{
"type": "Available",
"type": "Reconciled",
"status": "True",
"observed_generation": 1,
"last_transition_time": "2026-03-10T07:56:35Z"
},
{
"type": "LastKnownReconciled",
"status": "True",
"observed_generation": 1,
"last_transition_time": "2026-03-10T07:56:35Z"
Expand All @@ -164,7 +170,7 @@ GET /api/hyperfleet/v1/clusters/{id}
"updated_time": "2026-03-10T07:56:35Z"
}

→ API returns aggregated status with Available and Ready conditions
→ API returns aggregated status with Reconciled, LastKnownReconciled, and Ready conditions

# 3. View adapter statuses
GET /api/hyperfleet/v1/clusters/{id}/statuses
Expand Down Expand Up @@ -318,12 +324,13 @@ adapters:

### 2.3 Status Aggregation

HyperFleet API aggregates the condition values reported by adapters associated with a resource and adds two synthetic aggregated conditions.
HyperFleet API aggregates the condition values reported by adapters associated with a resource and adds three synthetic aggregated conditions.

| Condition | Meaning | When True |
|-----------|---------|-----------|
| **Ready** | Resource is fully reconciled at current spec | All registered adapters report `Available=True` at the **current** `resource.spec.generation` |
| **Available** | Resource is operational at any known good configuration | All registered adapters report `Available=True` (at any generation) |
| **Reconciled** | Resource is fully reconciled at current spec | All registered adapters report `Available=True` at the **current** `resource.spec.generation` |
| **LastKnownReconciled** | Resource is operational at any known good configuration | All registered adapters report `Available=True` for a common `observed_generation`, or sticky-true is preserved when adapters are transitioning to a new generation |
| **Ready** *(deprecated — use Reconciled)* | Alias of Reconciled | Same as Reconciled |
Comment on lines +327 to +333
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Coordinate this rename with the architecture repo too.

This guide is now aligned on LastKnownReconciled, but the linked openshift-hyperfleet/architecture ADRs and adapter-framework docs still describe aggregated Available. A companion update there will avoid publishing two different contracts to adapter authors.

As per coding guidelines, "Validate changes against HyperFleet architecture standards from the linked architecture repository."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/api-operator-guide.md` around lines 327 - 333, Docs were updated to use
the aggregated condition name LastKnownReconciled but related ADRs and
adapter-framework docs still reference Available; coordinate a rename across
repos by updating the architecture ADRs and adapter-framework docs to use
LastKnownReconciled (and mark Ready as deprecated/alias of Reconciled) so all
published contracts match; search for symbols/terms "LastKnownReconciled",
"Reconciled", "Ready", and "Available" in the openshift-hyperfleet/architecture
ADRs and adapter-framework docs, change wording and examples to the new names,
update any API/contract examples and tests to reference LastKnownReconciled, and
validate the changes against HyperFleet architecture standards before merging.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of scope for this PR. The architecture repo update will be coordinated as a follow-up after both this PR and the spec PR (#43) merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rafabene, understood — the architecture repo update will be a follow-up once this PR and the upstream spec PR (#43) are merged.

Would you like me to open a follow-up GitHub issue to track the architecture/adapter-framework documentation update (ADRs, adapter-framework docs) to align with the LastKnownReconciled rename?


**Note**: The meaning of the field `last_updated_time` for the aggregated conditions has special meaning. It doesn't reflect the last time it was updated from adapters but the OLDEST time it can be considered to be valid.

Expand All @@ -333,20 +340,22 @@ This is used by other components (Sentinel) to determine how "fresh" the status

The resource `status.conditions` array contains:

- **Ready** - The resource is reconciled to the latest resource spec
- **Reconciled** - The resource is reconciled to the latest resource spec
- `True`: All required adapters `conditions[type=Available].status==True` at current spec generation
- `False`: Any other combination of conditions

- **Available** - The resource is reconciled at a generation of the spec, current or past
- **Ready** *(deprecated — use Reconciled)* - Alias of Reconciled with identical semantics

- **LastKnownReconciled** - The resource is reconciled at a generation of the spec, current or past
- This condition is stateful meaning that is computed taking into account its previous values of `status` and `observed_generation`
- This condition is "best effort", since there are cases that can not be covered correctly.
- `True`:
- All required adapters `conditions[type=Available].status==True` for the same `observed_generation`
- Current value `status==True` and required adapters `conditions[type=Available]` at mixed `observed_generation`
- `False`: Any other combination of conditions
- e.g. `Available=True` for `observed_generation==1`
- One adapter reports `Available=False` for `observed_generation=1` `Available` transitions to `False`
- One adapter reports `Available=False` for `observed_generation=2` `Available` keeps its `True` status
- e.g. `LastKnownReconciled=True` for `observed_generation==1`
- One adapter reports `Available=False` for `observed_generation=1` `LastKnownReconciled` transitions to `False`
- One adapter reports `Available=False` for `observed_generation=2` `LastKnownReconciled` keeps its `True` status

- One **per-adapter** condition for each required adapter that has reported, mirroring the adapter's `conditions[type=Available]`:
- `type`: Derived from the adapter name — PascalCase with `Successful` suffix (e.g., `adapter1` → `Adapter1Successful`, `my-adapter` → `MyAdapterSuccessful`)
Expand Down Expand Up @@ -392,10 +401,10 @@ These are API examples for a resource and resource statuses:
"last_transition_time": "2021-01-01T10:00:00Z"
},
{
"type": "Available",
"type": "LastKnownReconciled",
"status": "True",
"reason": "All adapters reported Available True for the same generation",
"message": "All adapters reported Available True for the same generation",
"reason": "AllAdaptersReconciled",
"message": "All required adapters report Available=True for the tracked generation",
"observed_generation": 1,
"created_time": "2021-01-01T10:00:00Z",
"last_updated_time": "2021-01-01T10:00:00Z",
Expand Down Expand Up @@ -497,23 +506,23 @@ These are API examples for a resource and resource statuses:
When a resource is created:

- Initial `generation` is 1 and aggregated conditions are evaluated
- `observed_generation` for `Ready` and `Available` aggregated conditions is 1
- `last_updated_time` and `last_transition_time` for `Ready` and `Available` aggregated conditions is `resource.last_updated_time`
- `observed_generation` for `Ready` and `LastKnownReconciled` aggregated conditions is 1
- `last_updated_time` and `last_transition_time` for `Ready` and `LastKnownReconciled` aggregated conditions is `resource.last_updated_time`

When a resource is changed:

- `resource.generation` gets incremented and aggregated conditions are re-evaluated
- `status.conditions[type==Ready].observed_generation` always follows `resource.generation`
- `status.conditions[type==Available].observed_generation` changes when all required adapters `condition[type==Available].observed_generation==resource.generation` otherwise remains unchanged.
- `status.conditions[type==LastKnownReconciled].observed_generation` changes when all required adapters `condition[type==Available].observed_generation==resource.generation` otherwise remains unchanged.

##### Computing `observed_generation`

- For `Ready` it always matches `resource.generation`
- For `Available`:
- For `LastKnownReconciled`:
- If all required adapters have a common `observed_generation` it will match the common value
- If required adapters have mixed `observed_generation`
- If `Available` is `True`, `observed_generation` remains at its current value
- If `Available` is `False`, `observed_generation` will get the value of the `max(condition[type==Available].observed_generation)`
- If `LastKnownReconciled` is `True`, `observed_generation` remains at its current value
- If `LastKnownReconciled` is `False`, `observed_generation` will get the value of the `max(condition[type==Available].observed_generation)`

##### Computing `status.conditions[type==Ready].last_updated_time`

Expand All @@ -526,14 +535,14 @@ The meaning of `last_updated_time` in the aggregated conditions refers to the ne
- Why do we want to keep the "oldest" value? because if it is too old, we need to trigger a reconciliation
- When some required adapter conditions `condition[type==Available].observed_generation==resource.generation` then `last_updated_time=min(statuses[].conditions[type==Available && observed_generation==resource.generation].observed_time)`

##### Computing `status.conditions[type==Available].last_updated_time`
##### Computing `status.conditions[type==LastKnownReconciled].last_updated_time`

- If all required adapters have `condition[type==Available].observed_generation` at the same value then `last_updated_time=min(statuses[].conditions[type==Available].observed_time)`
- If not all required adapters have `condition[type==Available].observed_generation` at the same value:
- If any adapter at current `observed_generation==X` has `conditions[type==Available].status==False` then `last_updated_time=min(adapters[type==Available && observed_generation==X].observed_time`
- In any other case `last_updated_time` is kept unchanged

##### Computing `last_transition_time` for both `Ready` and `Available`
##### Computing `last_transition_time` for both `Ready` and `LastKnownReconciled`

- Meaning is last time this condition’s status (True / False) changed, regardless of the existing and new `observed_generation`
- This property is stateful since it relies on the existing value to determine if there has been a transition
Expand Down
88 changes: 63 additions & 25 deletions docs/api-resources.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,20 @@ POST /api/hyperfleet/v1/clusters/{cluster_id}/statuses
"status": {
"conditions": [
{
"type": "Available",
"type": "Reconciled",
"status": "False",
"reason": "AwaitingAdapters",
"message": "Waiting for adapters to report status",
"reason": "ReconciledMissingAdapters",
"message": "Required adapters have not yet reported status",
"observed_generation": 1,
"created_time": "2025-01-01T00:00:00Z",
"last_updated_time": "2025-01-01T00:00:00Z",
"last_transition_time": "2025-01-01T00:00:00Z"
},
{
"type": "LastKnownReconciled",
"status": "False",
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
"reason": "AdaptersMissingReports",
"message": "Required adapters have not yet reported status",
"observed_generation": 1,
"created_time": "2025-01-01T00:00:00Z",
"last_updated_time": "2025-01-01T00:00:00Z",
Expand All @@ -65,8 +75,8 @@ POST /api/hyperfleet/v1/clusters/{cluster_id}/statuses
{
"type": "Ready",
"status": "False",
"reason": "AwaitingAdapters",
"message": "Waiting for adapters to report status",
"reason": "ReconciledMissingAdapters",
"message": "Required adapters have not yet reported status",
"observed_generation": 1,
"created_time": "2025-01-01T00:00:00Z",
"last_updated_time": "2025-01-01T00:00:00Z",
Expand All @@ -79,7 +89,7 @@ POST /api/hyperfleet/v1/clusters/{cluster_id}/statuses

</details>

**Note**: Status initially has `Available=False` and `Ready=False` conditions until adapters report status.
**Note**: Status initially has `Reconciled=False`, `LastKnownReconciled=False`, and `Ready=False` conditions until adapters report status.

### Get Cluster

Expand Down Expand Up @@ -108,10 +118,20 @@ POST /api/hyperfleet/v1/clusters/{cluster_id}/statuses
"status": {
"conditions": [
{
"type": "Available",
"type": "Reconciled",
"status": "True",
"reason": "ResourceAvailable",
"message": "Cluster is accessible",
"reason": "ReconciledAll",
"message": "All required adapters reported Available=True or Finalized=True at the current generation",
"observed_generation": 1,
"created_time": "2025-01-01T00:00:00Z",
"last_updated_time": "2025-01-01T00:00:00Z",
"last_transition_time": "2025-01-01T00:00:00Z"
},
{
"type": "LastKnownReconciled",
"status": "True",
"reason": "AllAdaptersReconciled",
"message": "All required adapters report Available=True for the tracked generation",
"observed_generation": 1,
"created_time": "2025-01-01T00:00:00Z",
"last_updated_time": "2025-01-01T00:00:00Z",
Expand All @@ -120,8 +140,8 @@ POST /api/hyperfleet/v1/clusters/{cluster_id}/statuses
{
"type": "Ready",
"status": "True",
"reason": "ResourceReady",
"message": "All adapters report ready at current generation",
"reason": "ReconciledAll",
"message": "All required adapters reported Available=True or Finalized=True at the current generation",
"observed_generation": 1,
"created_time": "2025-01-01T00:00:00Z",
"last_updated_time": "2025-01-01T00:00:00Z",
Expand Down Expand Up @@ -304,10 +324,20 @@ POST /api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id}/statuses
"status": {
"conditions": [
{
"type": "Available",
"type": "Reconciled",
"status": "False",
"reason": "ReconciledMissingAdapters",
"message": "Required adapters have not yet reported status",
"observed_generation": 1,
"created_time": "2025-01-01T00:00:00Z",
"last_updated_time": "2025-01-01T00:00:00Z",
"last_transition_time": "2025-01-01T00:00:00Z"
},
{
"type": "LastKnownReconciled",
"status": "False",
"reason": "AwaitingAdapters",
"message": "Waiting for adapters to report status",
"reason": "AdaptersMissingReports",
"message": "Required adapters have not yet reported status",
"observed_generation": 1,
"created_time": "2025-01-01T00:00:00Z",
"last_updated_time": "2025-01-01T00:00:00Z",
Expand All @@ -316,8 +346,8 @@ POST /api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id}/statuses
{
"type": "Ready",
"status": "False",
"reason": "AwaitingAdapters",
"message": "Waiting for adapters to report status",
"reason": "ReconciledMissingAdapters",
"message": "Required adapters have not yet reported status",
"observed_generation": 1,
"created_time": "2025-01-01T00:00:00Z",
"last_updated_time": "2025-01-01T00:00:00Z",
Expand Down Expand Up @@ -361,17 +391,24 @@ POST /api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id}/statuses
"status": {
"conditions": [
{
"type": "Available",
"type": "Reconciled",
"status": "True",
"reason": "ReconciledAll",
"message": "All required adapters reported Available=True or Finalized=True at the current generation",
"observed_generation": 1
},
{
"type": "LastKnownReconciled",
"status": "True",
"reason": "ResourceAvailable",
"message": "NodePool is accessible",
"reason": "AllAdaptersReconciled",
"message": "All required adapters report Available=True for the tracked generation",
"observed_generation": 1
},
{
"type": "Ready",
"status": "True",
"reason": "ResourceReady",
"message": "All adapters report ready at current generation",
"reason": "ReconciledAll",
"message": "All required adapters reported Available=True or Finalized=True at the current generation",
"observed_generation": 1
}
]
Expand Down Expand Up @@ -420,7 +457,7 @@ All list endpoints support filtering using TSL (Tree Search Language) query synt

```bash
curl -G http://localhost:8000/api/hyperfleet/v1/clusters \
--data-urlencode "search=status.conditions.Ready='True' and labels.environment='production'"
--data-urlencode "search=status.conditions.Reconciled='True' and labels.environment='production'"
```

See **[search.md](search.md)** for complete documentation.
Expand All @@ -446,8 +483,9 @@ See **[search.md](search.md)** for complete documentation.
The status object contains synthesized conditions computed from adapter reports:

- `conditions` - Array of resource conditions, including:
- **Available** - Whether resource is running at any known good configuration
- **Ready** - Whether all adapters have processed current spec generation
- **Reconciled** - Whether all adapters have reconciled at the current spec generation
- **LastKnownReconciled** - Whether resource is running at any known good configuration
- **Ready** *(deprecated — alias of Reconciled)* - Same semantics as Reconciled; prefer `Reconciled` for new integrations
- Additional conditions from adapters (with `observed_generation`, timestamps)

### Condition Fields
Expand All @@ -464,7 +502,7 @@ The status object contains synthesized conditions computed from adapter reports:
- All above fields plus:
- `observed_generation` - Generation this condition reflects
- `created_time` - When condition was first created (API-managed)
- `last_updated_time` - When adapter last reported (API-managed, from AdapterStatus.last_report_time)
- `last_updated_time` - API-managed. For per-adapter conditions, taken from `AdapterStatus.last_report_time`. For aggregated conditions (`Reconciled`, `LastKnownReconciled`, `Ready`), computed as the oldest valid adapter report time within the relevant generation bucket — not the latest report time
- `last_transition_time` - When status last changed (API-managed)

## Parameter Restrictions
Expand Down
4 changes: 2 additions & 2 deletions docs/search.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,14 +98,14 @@ Label keys must contain only lowercase letters (a-z), digits (0-9), and undersco

Query resources by status conditions: `status.conditions.<Type>='<Status>'`

Condition types must be PascalCase (`Ready`, `Available`) and status must be `True` or `False` for resource conditions.
Condition types must be PascalCase (`Ready`, `LastKnownReconciled`) and status must be `True` or `False` for resource conditions.

**Note:** Only the `=` operator is supported for condition queries. Other operators (`!=`, `<`, `>`, `in`, etc.) will return an error. The `NOT` operator is not supported with condition queries (`status.conditions.<Type>` or `status.conditions.<Type>.<Subfield>`) and will return a `400 Bad Request` error. Use the inverse condition value instead (e.g., `status.conditions.Ready='False'` rather than `NOT status.conditions.Ready='True'`).

```bash
# Find available clusters
curl -G "http://localhost:8000/api/hyperfleet/v1/clusters" \
--data-urlencode "search=status.conditions.Available='True'"
--data-urlencode "search=status.conditions.LastKnownReconciled='True'"
Comment thread
coderabbitai[bot] marked this conversation as resolved.

# Find clusters that are not ready
curl -G "http://localhost:8000/api/hyperfleet/v1/clusters" \
Expand Down
Loading