Skip to content

scst_user: Fix infinite cleanup loop caused by stale SGV pool reference#378

Merged
lnocturno merged 1 commit into
SCST-project:masterfrom
smileusd:upstream_scst_D_state_issue
Jun 8, 2026
Merged

scst_user: Fix infinite cleanup loop caused by stale SGV pool reference#378
lnocturno merged 1 commit into
SCST-project:masterfrom
smileusd:upstream_scst_D_state_issue

Conversation

@smileusd

@smileusd smileusd commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

fix the issue #279

The full analize doc at https://smileusd.github.io/2026/06/01/2026-06-01-scst-infinite-loop-deadlock-analysis/

The device cleanup loop in dev_user_process_cleanup() spins at ~2 million iterations per second and never exits, ultimately triggering a kernel soft lockup. The previous workaround panicked the system after 10,000 iterations.

Root cause (confirmed by instrumentation):

A ucmd gets permanently stuck in ucmd_hash with:
state = UCMD_STATE_ON_FREE_SKIPPED (7)
cmd = NULL
ref = 1
sent_to_user = 0

The stuck ref=1 is the reference taken by dev_user_alloc_pages() via
ucmd_get() for the first scatter-gather page. It is released only by
dev_user_free_sg_entries() → ucmd_put(), which fires when the SGV pool
evicts a cached object. The sequence that prevents this eviction:

  1. dev_user_unjam_dev() finds an EXECING command (sent_to_user=1, ref=2: alloc + alloc_pages), bumps ref to 3 via ucmd_get_check(), then calls dev_user_unjam_cmd().

  2. dev_user_unjam_cmd() releases cmd_list_lock and calls scst_cmd_done(SCST_CONTEXT_THREAD), which synchronously runs the full SCST completion pipeline:

    dev_user_on_free_cmd()
    ucmd->cmd = NULL
    ucmd->state = UCMD_STATE_ON_FREE_SKIPPED (type == IGNORE)
    dev_user_process_reply_on_free()
    dev_user_free_sgv()
    sgv_pool_free(ucmd->sgv) ← returns SGV to pool LRU cache;
    dev_user_free_sg_entries() NOT called;
    alloc_pages ucmd_get() NOT balanced
    ucmd->sgv = NULL
    ucmd_put() ← ref: 3→2

  3. Back in dev_user_unjam_dev(): ucmd_put() ← ref: 2→1. ref != 0, so dev_user_free_ucmd() / cmd_remove_hash() are NOT called. ucmd remains in ucmd_hash.

  4. unjam_cmd also reset sent_to_user=0, so on every subsequent pass through dev_user_unjam_dev() the ucmd is counted (res++) but skipped (!sent_to_user → continue). dev_user_get_next_cmd() returns -EAGAIN (ucmd is not in ready_cmd_list). With cleanup_done=1 the while(1) loop has no exit condition.

The sgv_pool_flush() calls at the TOP of dev_user_unjam_dev() run
BEFORE any commands are unjammed. SGV objects cached during unjamming
are therefore never flushed; dev_user_free_sg_entries() never fires.

Fix:

Add sgv_pool_flush() for both pools at the BOTTOM of dev_user_unjam_dev(),
after the unjam loop exits and after the spinlock is released. This evicts
all SGV objects cached during unjamming, triggering:
dev_user_free_sg_entries() → ucmd_put() → dev_user_free_ucmd()
→ cmd_remove_hash()
removing the stuck ucmd from the hash. On the next cleanup-loop iteration
dev_user_unjam_dev() returns res=0 and dev_user_process_cleanup() breaks.

Also replace the panic() workaround with schedule() so the CPU yields
to let SGV free callbacks complete between iterations.

@smileusd smileusd force-pushed the upstream_scst_D_state_issue branch from 1b7c0d1 to 17a53bf Compare June 5, 2026 10:05
Comment thread scst/src/dev_handlers/scst_user.c Outdated
The device cleanup loop in dev_user_process_cleanup() spins at ~2 million
iterations per second and never exits, ultimately triggering a kernel soft
lockup. The previous workaround panicked the system after 10,000
iterations.

Root cause (confirmed by instrumentation):

  A ucmd gets permanently stuck in ucmd_hash with:
    state  = UCMD_STATE_ON_FREE_SKIPPED (7)
    cmd    = NULL
    ref    = 1
    sent_to_user = 0

  The stuck ref=1 is the reference taken by dev_user_alloc_pages() via
  ucmd_get() for the first scatter-gather page. It is released only by
  dev_user_free_sg_entries() → ucmd_put(), which fires when the SGV pool
  *evicts* a cached object. The sequence that prevents this eviction:

  1. dev_user_unjam_dev() finds an EXECING command (sent_to_user=1,
     ref=2: alloc + alloc_pages), bumps ref to 3 via ucmd_get_check(),
     then calls dev_user_unjam_cmd().

  2. dev_user_unjam_cmd() releases cmd_list_lock and calls
     scst_cmd_done(SCST_CONTEXT_THREAD), which synchronously runs the
     full SCST completion pipeline:

       dev_user_on_free_cmd()
         ucmd->cmd = NULL
         ucmd->state = UCMD_STATE_ON_FREE_SKIPPED  (type == IGNORE)
         dev_user_process_reply_on_free()
           dev_user_free_sgv()
             sgv_pool_free(ucmd->sgv)
               /* SGV cached on pool LRU; dev_user_free_sg_entries()
                * not called; alloc_pages ucmd_get() not balanced */
             ucmd->sgv = NULL
           ucmd_put()  ← ref: 3→2

  3. Back in dev_user_unjam_dev(): ucmd_put() ← ref: 2→1.
     ref != 0, so dev_user_free_ucmd() / cmd_remove_hash() are NOT called.
     ucmd remains in ucmd_hash.

  4. unjam_cmd also reset sent_to_user=0, so on every subsequent pass
     through dev_user_unjam_dev() the ucmd is counted (res++) but skipped
     (!sent_to_user → continue). dev_user_get_next_cmd() returns -EAGAIN
     (ucmd is not in ready_cmd_list). With cleanup_done=1 the while(1)
     loop has no exit condition.

  The sgv_pool_flush() calls at the TOP of dev_user_unjam_dev() run
  BEFORE any commands are unjammed. SGV objects cached during unjamming
  are therefore never flushed; dev_user_free_sg_entries() never fires.

Fix:

  Add sgv_pool_flush() for both pools at the BOTTOM of
  dev_user_unjam_dev(), after the spinlock is released. This evicts
  all SGV objects cached during unjamming, triggering:
    dev_user_free_sg_entries() → ucmd_put() → dev_user_free_ucmd()
      → cmd_remove_hash()
  removing the stuck ucmd from the hash. On the next cleanup-loop iteration
  dev_user_unjam_dev() returns res=0 and dev_user_process_cleanup() breaks.

  sgv_pool_flush() is fully synchronous (calls sgv_dtor_and_free() inline);
  by the time it returns the callbacks have already fired and the ucmd has
  already been removed from the hash. No schedule() or sleep is needed.
@smileusd smileusd force-pushed the upstream_scst_D_state_issue branch from 17a53bf to fe24d80 Compare June 7, 2026 07:20
@smileusd smileusd requested a review from lnocturno June 8, 2026 02:12
@lnocturno lnocturno merged commit 83745c0 into SCST-project:master Jun 8, 2026
54 checks passed
@lnocturno

Copy link
Copy Markdown
Contributor

Hi,

Thank you for the patch!

Gleb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants