[Bug]: Random crashes running Llama 3.3 70B with lora

### Your current environment

```text
Collecting environment information...
PyTorch version: 2.4.0
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (conda-forge gcc 11.3.0-19) 11.3.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.11.9 | packaged by conda-forge | (main, Apr 19 2024, 18:36:13) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-6.8.0-52-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090
GPU 3: NVIDIA GeForce RTX 3090
GPU 4: NVIDIA GeForce RTX 3090
GPU 5: NVIDIA GeForce RTX 3090
GPU 6: NVIDIA GeForce RTX 3090
GPU 7: NVIDIA GeForce RTX 3090

Nvidia driver version: 565.57.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        43 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               48
On-line CPU(s) list:                  0-47
Vendor ID:                            AuthenticAMD
Model name:                           AMD EPYC 7402 24-Core Processor
CPU family:                           23
Model:                                49
Thread(s) per core:                   2
Core(s) per socket:                   24
Socket(s):                            1
Stepping:                             0
Frequency boost:                      enabled
CPU max MHz:                          2800,0000
CPU min MHz:                          1500,0000
BogoMIPS:                             5600.21
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es
Virtualization:                       AMD-V
L1d cache:                            768 KiB (24 instances)
L1i cache:                            768 KiB (24 instances)
L2 cache:                             12 MiB (24 instances)
L3 cache:                             128 MiB (8 instances)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-47
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow:   Mitigation; Safe RET
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] pyzmq==26.2.1
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.45.2
[pip3] triton==3.0.0
[conda] blas                      2.16                        mkl    conda-forge
[conda] libblas                   3.8.0                    16_mkl    conda-forge
[conda] libcblas                  3.8.0                    16_mkl    conda-forge
[conda] liblapack                 3.8.0                    16_mkl    conda-forge
[conda] liblapacke                3.8.0                    16_mkl    conda-forge
[conda] mkl                       2020.2                      256  
[conda] nccl                      2.25.1.1             ha44e49d_0    conda-forge
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] pytorch                   2.4.0           py3.11_cuda12.4_cudnn9.1.0_0    pytorch
[conda] pytorch-cuda              12.4                 hc786d27_7    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] pyzmq                     26.2.1                   pypi_0    pypi
[conda] torchtriton               3.0.0                     py311    pytorch
[conda] torchvision               0.19.0                   pypi_0    pypi
[conda] transformers              4.45.2                   pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
Aphrodite Version: 0.6.7
Aphrodite Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    NODE    NODE    NODE    NODE    NODE    NODE    0-47    0               N/A
GPU1    NODE     X      NODE    NODE    NODE    NODE    NODE    NODE    0-47    0               N/A
GPU2    NODE    NODE     X      PHB     PHB     NODE    NODE    NODE    0-47    0               N/A
GPU3    NODE    NODE    PHB      X      PHB     NODE    NODE    NODE    0-47    0               N/A
GPU4    NODE    NODE    PHB     PHB      X      NODE    NODE    NODE    0-47    0               N/A
GPU5    NODE    NODE    NODE    NODE    NODE     X      PHB     PHB     0-47    0               N/A
GPU6    NODE    NODE    NODE    NODE    NODE    PHB      X      PHB     0-47    0               N/A
GPU7    NODE    NODE    NODE    NODE    NODE    PHB     PHB      X      0-47    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks
```

### 🐛 Describe the bug

Random crashes with unclear cause. The log of the crash is shown below, this is with disable frontend multiprocessing and disable async output proc.

Start command is as shown here with a lot of lora loaded:

```
#!/bin/bash
export APHRODITE_RPC_TIMEOUT=120000
export APHRODITE_FORCE_P2P=1
python -m aphrodite.endpoints.openai.api_server \
--model /home/owen/models/Llama-3.3-70B-Instruct-W8A8-INT8 \
--swap-space 2 --gpu-memory-utilization 0.94 --max-model-len 24576 --max-seq-len-to-capture 24576 --port 8000 \
--enable-prefix-caching --preemption-mode swap \
--max-num-seqs 50 --served-model-name Llama-3.3-70B-Instruct -tp 8 --enforce-eager false \
--enable-lora --max-lora-rank 64 --max-loras 5 --lora-modules \
```

```
INFO:     Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 6 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.1%, CPU KV cache usage: 0.0%.
INFO:     Prefix cache hit rate: GPU: 41.65%, CPU: 0.00%
INFO:     192.168.20.72:50892 - "GET /health HTTP/1.1" 200 OK
INFO:     192.168.20.72:50902 - "GET /health HTTP/1.1" 200 OK
INFO:     192.168.20.72:50912 - "GET /health HTTP/1.1" 200 OK
INFO:     192.168.20.72:50918 - "GET /health HTTP/1.1" 200 OK
INFO:     192.168.20.72:50918 - "GET /health HTTP/1.1" 200 OK
ERROR:    Engine background task failed
Exception in callback functools.partial(<function _log_task_completion at 0x787abe3eb420>, error_callback=<bound method AsyncAphrodite._error_callback of <aphrodite.engine.async_aphrodite.AsyncAphrodite object at 0x787ab95bc210>>)
handle: <Handle functools.partial(<function _log_task_completion at 0x787abe3eb420>, error_callback=<bound method AsyncAphrodite._error_callback of <aphrodite.engine.async_aphrodite.AsyncAphrodite object at 0x787ab95bc210>>)>
Traceback (most recent call last):
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 54, in _log_task_completion
    return_value = task.result()
                   ^^^^^^^^^^^^^
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 809, in run_engine_loop
    result = task.result()
             ^^^^^^^^^^^^^
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 735, in engine_step
    request_outputs = await self.engine.step_async(virtual_engine)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 388, in step_async
    self._process_model_outputs(ctx=ctx)
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/aphrodite_engine.py", line 1008, in _process_model_outputs
    self.output_processor.process_outputs(
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/output_processor/single_step.py", line 91, in process_outputs
    return self._process_sequence_group_outputs(sequence_group, outputs[0],
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/output_processor/single_step.py", line 204, in _process_sequence_group_outputs
    scheduler.fork_seq(parent, seq)
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/scheduler.py", line 1333, in fork_seq
    self.block_manager.fork(parent_seq, child_seq)
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block_manager_v2.py", line 327, in fork
    self.block_tables[child_seq.seq_id] = src_block_table.fork()
                                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/block_table.py", line 208, in fork
    forked_blocks = self._allocator.fork(self._blocks[-1])
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/cpu_gpu_block_allocator.py", line 206, in fork
    return allocator.fork(last_block)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/prefix_caching_block.py", line 368, in fork
    source_blocks = get_all_blocks_recursively(last_block)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 355, in get_all_blocks_recursively
    recurse(last_block, all_blocks)
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    recurse(block.prev_block, lst)
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    recurse(block.prev_block, lst)
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    recurse(block.prev_block, lst)
  [Previous line repeated 977 more times]
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 350, in recurse
    if block.prev_block is not None:
       ^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
  File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 66, in _log_task_completion
    raise AsyncEngineDeadError(
aphrodite.engine.async_aphrodite.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for the actual cause.
ERROR:    AsyncAphrodite has failed, terminating server process
INFO:     192.168.20.72:58230 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
ERROR:    AsyncAphrodite has failed, terminating server process
INFO:     192.168.20.72:45800 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 268, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
    await func()
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 233, in listen_for_disconnect
    message = await receive()
              ^^^^^^^^^^^^^^^
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 563, in receive
    await self.message_event.wait()
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 787b4c84bdd0

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
  |     raise exc
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
  |     await self.app(scope, receive, _send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
  |     await self.app(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
  |     await route.handle(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
  |     await self.app(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
  |     await response(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 261, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 767, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
    |     await func()
    |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 245, in stream_response
    |     async for chunk in self.body_iterator:
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/endpoints/openai/serving_chat.py", line 319, in chat_completion_stream_generator
    |     async for res in result_generator:
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/common/utils.py", line 466, in iterate_with_cancellation
    |     item = await awaits[0]
    |            ^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 54, in _log_task_completion
    |     return_value = task.result()
    |                    ^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 809, in run_engine_loop
    |     result = task.result()
    |              ^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 735, in engine_step
    |     request_outputs = await self.engine.step_async(virtual_engine)
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 388, in step_async
    |     self._process_model_outputs(ctx=ctx)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/aphrodite_engine.py", line 1008, in _process_model_outputs
    |     self.output_processor.process_outputs(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/output_processor/single_step.py", line 91, in process_outputs
    |     return self._process_sequence_group_outputs(sequence_group, outputs[0],
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/output_processor/single_step.py", line 204, in _process_sequence_group_outputs
    |     scheduler.fork_seq(parent, seq)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/scheduler.py", line 1333, in fork_seq
    |     self.block_manager.fork(parent_seq, child_seq)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block_manager_v2.py", line 327, in fork
    |     self.block_tables[child_seq.seq_id] = src_block_table.fork()
    |                                           ^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/block_table.py", line 208, in fork
    |     forked_blocks = self._allocator.fork(self._blocks[-1])
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/cpu_gpu_block_allocator.py", line 206, in fork
    |     return allocator.fork(last_block)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/prefix_caching_block.py", line 368, in fork
    |     source_blocks = get_all_blocks_recursively(last_block)
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 355, in get_all_blocks_recursively
    |     recurse(last_block, all_blocks)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    |     recurse(block.prev_block, lst)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    |     recurse(block.prev_block, lst)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    |     recurse(block.prev_block, lst)
    |   [Previous line repeated 977 more times]
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 350, in recurse
    |     if block.prev_block is not None:
    |        ^^^^^^^^^^^^^^^^
    | RecursionError: maximum recursion depth exceeded
    +------------------------------------
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 268, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
    await func()
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 233, in listen_for_disconnect
    message = await receive()
              ^^^^^^^^^^^^^^^
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 563, in receive
    await self.message_event.wait()
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 787b4d29b750

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
  |     raise exc
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
  |     await self.app(scope, receive, _send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
  |     await self.app(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
  |     await route.handle(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
  |     await self.app(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
  |     await response(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 261, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 767, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
    |     await func()
    |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 245, in stream_response
    |     async for chunk in self.body_iterator:
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/endpoints/openai/serving_chat.py", line 319, in chat_completion_stream_generator
    |     async for res in result_generator:
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/common/utils.py", line 466, in iterate_with_cancellation
    |     item = await awaits[0]
    |            ^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 54, in _log_task_completion
    |     return_value = task.result()
    |                    ^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 809, in run_engine_loop
    |     result = task.result()
    |              ^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 735, in engine_step
    |     request_outputs = await self.engine.step_async(virtual_engine)
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 388, in step_async
    |     self._process_model_outputs(ctx=ctx)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/aphrodite_engine.py", line 1008, in _process_model_outputs
    |     self.output_processor.process_outputs(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/output_processor/single_step.py", line 91, in process_outputs
    |     return self._process_sequence_group_outputs(sequence_group, outputs[0],
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/output_processor/single_step.py", line 204, in _process_sequence_group_outputs
    |     scheduler.fork_seq(parent, seq)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/scheduler.py", line 1333, in fork_seq
    |     self.block_manager.fork(parent_seq, child_seq)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block_manager_v2.py", line 327, in fork
    |     self.block_tables[child_seq.seq_id] = src_block_table.fork()
    |                                           ^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/block_table.py", line 208, in fork
    |     forked_blocks = self._allocator.fork(self._blocks[-1])
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/cpu_gpu_block_allocator.py", line 206, in fork
    |     return allocator.fork(last_block)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/prefix_caching_block.py", line 368, in fork
    |     source_blocks = get_all_blocks_recursively(last_block)
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 355, in get_all_blocks_recursively
    |     recurse(last_block, all_blocks)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    |     recurse(block.prev_block, lst)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    |     recurse(block.prev_block, lst)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    |     recurse(block.prev_block, lst)
    |   [Previous line repeated 977 more times]
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 350, in recurse
    |     if block.prev_block is not None:
    |        ^^^^^^^^^^^^^^^^
    | RecursionError: maximum recursion depth exceeded
    +------------------------------------
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 268, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
    await func()
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 233, in listen_for_disconnect
    message = await receive()
              ^^^^^^^^^^^^^^^
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 563, in receive
    await self.message_event.wait()
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 787b4e1c0790

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
  |     raise exc
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
  |     await self.app(scope, receive, _send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
  |     await self.app(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
  |     await route.handle(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
  |     await self.app(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
  |     await response(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 261, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 767, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
    |     await func()
    |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 245, in stream_response
    |     async for chunk in self.body_iterator:
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/endpoints/openai/serving_chat.py", line 319, in chat_completion_stream_generator
    |     async for res in result_generator:
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/common/utils.py", line 466, in iterate_with_cancellation
    |     item = await awaits[0]
    |            ^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 54, in _log_task_completion
    |     return_value = task.result()
    |                    ^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 809, in run_engine_loop
    |     result = task.result()
    |              ^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 735, in engine_step
    |     request_outputs = await self.engine.step_async(virtual_engine)
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 388, in step_async
    |     self._process_model_outputs(ctx=ctx)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/aphrodite_engine.py", line 1008, in _process_model_outputs
    |     self.output_processor.process_outputs(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/output_processor/single_step.py", line 91, in process_outputs
    |     return self._process_sequence_group_outputs(sequence_group, outputs[0],
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/output_processor/single_step.py", line 204, in _process_sequence_group_outputs
    |     scheduler.fork_seq(parent, seq)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/scheduler.py", line 1333, in fork_seq
    |     self.block_manager.fork(parent_seq, child_seq)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block_manager_v2.py", line 327, in fork
    |     self.block_tables[child_seq.seq_id] = src_block_table.fork()
    |                                           ^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/block_table.py", line 208, in fork
    |     forked_blocks = self._allocator.fork(self._blocks[-1])
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/cpu_gpu_block_allocator.py", line 206, in fork
    |     return allocator.fork(last_block)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/prefix_caching_block.py", line 368, in fork
    |     source_blocks = get_all_blocks_recursively(last_block)
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 355, in get_all_blocks_recursively
    |     recurse(last_block, all_blocks)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    |     recurse(block.prev_block, lst)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    |     recurse(block.prev_block, lst)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    |     recurse(block.prev_block, lst)
    |   [Previous line repeated 977 more times]
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 350, in recurse
    |     if block.prev_block is not None:
    |        ^^^^^^^^^^^^^^^^
    | RecursionError: maximum recursion depth exceeded
    +------------------------------------
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 268, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
    await func()
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 233, in listen_for_disconnect
    message = await receive()
              ^^^^^^^^^^^^^^^
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 563, in receive
    await self.message_event.wait()
  File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 787b4e133e90

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
  |     raise exc
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
  |     await self.app(scope, receive, _send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
  |     await self.app(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
  |     await route.handle(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
  |     await self.app(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
  |     await response(scope, receive, send)
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 261, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 767, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
    |     await func()
    |   File "/home/owen/miniconda3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 245, in stream_response
    |     async for chunk in self.body_iterator:
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/endpoints/openai/serving_chat.py", line 319, in chat_completion_stream_generator
    |     async for res in result_generator:
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/common/utils.py", line 466, in iterate_with_cancellation
    |     item = await awaits[0]
    |            ^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 942, in generate
    |     async for output in await self.add_request(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 112, in generator
    |     raise result
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 54, in _log_task_completion
    |     return_value = task.result()
    |                    ^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 809, in run_engine_loop
    |     result = task.result()
    |              ^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 735, in engine_step
    |     request_outputs = await self.engine.step_async(virtual_engine)
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 388, in step_async
    |     self._process_model_outputs(ctx=ctx)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/aphrodite_engine.py", line 1008, in _process_model_outputs
    |     self.output_processor.process_outputs(
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/output_processor/single_step.py", line 91, in process_outputs
    |     return self._process_sequence_group_outputs(sequence_group, outputs[0],
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/engine/output_processor/single_step.py", line 204, in _process_sequence_group_outputs
    |     scheduler.fork_seq(parent, seq)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/scheduler.py", line 1333, in fork_seq
    |     self.block_manager.fork(parent_seq, child_seq)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block_manager_v2.py", line 327, in fork
    |     self.block_tables[child_seq.seq_id] = src_block_table.fork()
    |                                           ^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/block_table.py", line 208, in fork
    |     forked_blocks = self._allocator.fork(self._blocks[-1])
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/cpu_gpu_block_allocator.py", line 206, in fork
    |     return allocator.fork(last_block)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/prefix_caching_block.py", line 368, in fork
    |     source_blocks = get_all_blocks_recursively(last_block)
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 355, in get_all_blocks_recursively
    |     recurse(last_block, all_blocks)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    |     recurse(block.prev_block, lst)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    |     recurse(block.prev_block, lst)
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 351, in recurse
    |     recurse(block.prev_block, lst)
    |   [Previous line repeated 977 more times]
    |   File "/home/owen/aphro-latest/aphrodite-engine/aphrodite/processing/block/common.py", line 350, in recurse
    |     if block.prev_block is not None:
    |        ^^^^^^^^^^^^^^^^
    | RecursionError: maximum recursion depth exceeded
    +------------------------------------
INFO:     Shutting down
INFO:     Waiting for connections to close. (CTRL+C to force quit)
INFO:     Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 6 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.1%, CPU KV cache usage: 0.0%.
INFO:     Prefix cache hit rate: GPU: 41.65%, CPU: 0.00%
INFO:     Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 6 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.1%, CPU KV cache usage: 0.0%.
INFO:     Prefix cache hit rate: GPU: 41.65%, CPU: 0.00%
INFO:     Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 6 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.1%, CPU KV cache usage: 0.0%.
INFO:     Prefix cache hit rate: GPU: 41.65%, CPU: 0.00%
```

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Bug]: Random crashes running Llama 3.3 70B with lora #1277

Your current environment

🐛 Describe the bug

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Uh oh!

[Bug]: Random crashes running Llama 3.3 70B with lora #1277

Description

Your current environment

🐛 Describe the bug

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions