Skip to content

sleepAfter alarm fires during long-running containerFetch() operations #162

@ghostwriternr

Description

@ghostwriternr

What happened

A container was killed by the sleepAfter alarm while containerFetch() was actively awaiting a response from the container. The operation (a matplotlib chart generation via Sandbox SDK's exec(), which uses containerFetch internally) had been running for approximately 6 seconds when the container received SIGTERM.

Client-side error:

Error: ReadableStream received over RPC disconnected prematurely.

Container-side error:

SessionDestroyedError: Session 'main' was destroyed during command execution

Note: the exact error messages come from Sandbox SDK, but sharing it here to be illustrative.

Reproduction

  1. Create a DO subclass extending Container
  2. Set sleepAfter to a short duration (e.g., "10s")
  3. Call containerFetch() with a request that takes longer than sleepAfter to complete (e.g., a command execution that runs for 15+ seconds)
  4. The alarm fires while containerFetch() is still awaiting the response, and onActivityExpired() kills the container
    This also affects back-to-back operations: if several quick operations consume most of the sleepAfter window, a subsequent longer operation can be killed even though it just started, because renewActivityTimeout() resets the deadline to now + sleepAfter, not now + sleepAfter from when the operation finishes.

Expected behaviour

The sleepAfter timer should not fire while containerFetch() is actively awaiting a response. The container is clearly in use when there's an in-flight request.

Current workaround

Before each long-running operation, extend the sleepAfter property and refresh the timeout:

this.sleepAfter = estimatedDurationSeconds + 30;
this.renewActivityTimeout();

This works but requires subclasses to predict operation duration and manage the timeout around every containerFetch() call.

Potentially related issues

Environment

Observed in production using v0.1.1 of @cloudflare/containers

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions