Fix session processor crash when a queue item is deleted while running

[lstein]

Summary

Follow-up to #9350, addressing the root cause of the missing queue item.

Deleting a queue item while it is executing is a supported operation: clear and delete_queue_item remove the DB row immediately and rely on the processor noticing the cancel event afterwards. But the workflow-call lifecycle re-fetched the queue item (and its parents) unconditionally after the session runner returned, raising SessionQueueItemNotFoundError. Before #9350 this escalated to a fatal error that killed the session processor thread; after #9350 it still logs a scary non-fatal error and traceback on every cancel-via-delete.

Two fixes:

Backend — WorkflowCallQueueLifecycle now treats a missing queue item or parent as "deleted mid-run" and returns quietly instead of raising, matching the existing handling in _on_after_run_session:

• run_queue_item catches SessionQueueItemNotFoundError on its post-run re-fetch (the exact failure point in the reported traceback).
• _get_parent_queue_item returns None when the parent row is gone (e.g. queue cleared while a child workflow was running); the resume/fail/cancel parent handlers all check it, as do the parent re-fetches inside the failure cascades.

Frontend — the shift+x hotkey labeled "cancel queue item" was still calling the delete endpoint. 2ddcde1 intentionally migrated the queue UI from cancel to delete semantics, but 5baa4bd later moved the visible queue controls back to cancelation and missed the hotkey. The hotkey now uses useCancelCurrentQueueItem, aligning it with the CancelCurrentQueueItemIconButton it mirrors and removing the most common trigger of the delete-while-running race.

Related Issues / Discussions

• Handle missing queue items in session failure #9350 (merged backstop; this PR removes the underlying error)
• Discord report: https://discord.com/channels/1020123559063990373/1049495067846524939/1525012683211145306

QA Instructions

1. Enqueue several generations.
2. While one is in progress, press shift+x, use Queue → "Cancel and Clear All", or delete the current item from the queue list.
3. Before: log shows Non-fatal error in session processor SessionQueueItemNotFoundError: No queue item with id N (and pre-Handle missing queue items in session failure #9350, a fatal processor error). After: no error; processing continues with the next item.

Also covered by two new unit tests (item deleted mid-run; parent deleted while a child workflow runs, for completed/failed/canceled children). _DummySessionQueue.get_queue_item now raises the real SessionQueueItemNotFoundError instead of KeyError to match production behavior.

Merge Plan

None - straightforward merge.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• ❗Changes to a redux slice have a corresponding migration
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

🤖 Generated with Claude Code

[lstein]

Added a third commit fixing a follow-on UI issue: after canceling the current item, the progress bar stayed frozen at its last value instead of resetting.

Root cause: the canceled status event clears the progress store, but a denoise step callback can race the cancelation and emit invocation_progress after the terminal status event. The workflowExecutionCoordinator state machine exists to reject exactly these trailing events, but the terminal-status handler deleted the item's state entry at the moment it was needed, so the trailing event was treated as a fresh item and re-applied. The entry is now retained (item ids are never reused — retries insert new rows — and the LRU cache bounds memory).

Also: queue_cleared (Cancel and Clear All) deletes the in-progress item without emitting any per-item terminal status event, so nothing reset the bar on that path either — the listener now clears it explicitly.

Includes a regression test that fails against the previous coordinator behavior.

🤖 Generated with Claude Code

[JPPhoto]

• invokeai/frontend/web/src/services/events/setEventListeners.tsx:539 - The queue_cleared handler clears $lastProgressEvent but does not mark the deleted in-progress item terminal in workflowExecutionCoordinator. A trailing invocation_progress event is therefore accepted at invokeai/frontend/web/src/services/events/workflowExecutionCoordinator.ts:131 and written back to $lastProgressEvent at setEventListeners.tsx:204, recreating the frozen progress bar this change intends to fix. The retained terminal state only protects cancellation paths that emit queue_item_status_changed; the PR explicitly says queue clearing does not emit that event. To expose this issue, add a test that starts an item, handles queue_cleared, then delivers a trailing progress event and verifies that the coordinator rejects it and the progress store remains empty.

• invokeai/app/services/session_processor/workflow_call_runtime.py:192 - Parent deletion is handled only during the initial lookup and selected re-fetches, leaving time-of-check/time-of-use races throughout all three parent transitions. If clear() or delete_queue_item() removes the parent after _get_parent_queue_item() returns, set_queue_item_session() at lines 221 or 231, complete_queue_item() at line 235, resume_queue_item() at line 241, the failure path reached at line 259, or cancel_queue_item() at line 271 can still raise SessionQueueItemNotFoundError. The SQLite implementations confirm these mutations re-read the row and raise when it has disappeared. That exception returns to the session processor as the same non-fatal error and traceback the PR is intended to eliminate. The new test deletes the parent before the guarded lookup, so it cannot exercise these windows. To expose this issue, add tests whose queue doubles delete the parent immediately after get_queue_item() succeeds and before each completed, failed, and canceled parent mutation.

[lstein]

@JPPhoto Thanks for the careful review — both findings were real. Addressed in the two latest commits:

queue_cleared trailing progress (be3984d): the coordinator now exposes onQueueCleared(), which aborts pending reconciliations and marks every tracked non-terminal item canceled, so a trailing invocation_progress racing the clear is rejected instead of repopulating the progress bar. The queue_cleared listener calls it before resetting the progress store. Added the coordinator-level test you suggested: start an item, accept progress, handle queue-clear, then verify a trailing progress event is rejected (and the pending reconciliation aborted).

Parent-transition TOCTOU windows (6fdcd80): rather than chasing each window individually, run_queue_item now wraps the three parent transitions in a single except SessionQueueItemNotFoundError — any item vanishing between lookup and mutation means the chain is being torn down, and quietly returning is correct regardless of which mutation raised (the fine-grained lookup guards are kept for their early-return semantics). The dummy session queue gained a not_found_item_ids knob that makes mutations raise while lookups still succeed — faithfully simulating deletion-after-lookup — with one test per transition path (completed / failed / canceled). All three tests fail against the previous code.

🤖 Generated with Claude Code

[JPPhoto]

@lstein One more:

• invokeai/frontend/web/src/services/events/workflowExecutionCoordinator.ts:78 - onQueueCleared() unconditionally aborts every pending workflow reconciliation, but QueueClearedEvent is broadcast to the entire queue room because it contains no user identity (invokeai/app/api/sockets.py:309 and :449). In multiuser mode, a non-admin clear deletes only the requesting user's rows, while every connected user receives the event. If another user's completed or failed workflow item is currently being reconciled, this handler aborts that unaffected request; the item remains in the database, but no later event restarts reconciliation, so completed sibling outputs and node execution state can remain missing or stale. The new test verifies that reconciliation is aborted, but does not cover the user-scoped clear behavior that makes this incorrect. To expose this issue, add a test that starts reconciliation for user B's terminal workflow item, delivers a queue-cleared event caused by user A's user-filtered clear, and verifies that user B's reconciliation still completes.