Skip to content

Fix Tenacity errors which are happening in CHECK_COMMAND_LINE_STATUS #243

@sivkovic

Description

@sivkovic

Traceback:

Traceback (most recent call last):
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/srv/services/ridgeback/code/ridgeback/orchestrator/tasks.py", line 143, in command_processor
    check_status_of_command_line_jobs(job)
  File "/srv/services/ridgeback/code/ridgeback/orchestrator/tasks.py", line 381, in check_status_of_command_line_jobs
    command_line_status = submiter.get_commandline_status(track_cache_str)
  File "/srv/services/ridgeback/code/ridgeback/submitter/toil_submitter/toil_jobsubmitter.py", line 81, in get_commandline_status
    toil_track_obj.check_status()
  File "/srv/services/ridgeback/code/ridgeback/submitter/toil_submitter/toil_track_utils.py", line 654, in check_status
    job_store, root_job = _resume_job_store(self.job_store_path, self.total_attempts)
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/tenacity/__init__.py", line 339, in wrapped_f
    return self(f, *args, **kw)
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/tenacity/__init__.py", line 430, in __call__
    do = self.iter(retry_state=retry_state)
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/tenacity/__init__.py", line 379, in iter
    six.raise_from(retry_exc, fut.exception())
  File "<string>", line 3, in raise_from
tenacity.RetryError: <Future at 0x7fe9d6ac96d8 state=finished raised KeyError>
[2021-08-17 11:43:35,153: ERROR/MainProcess] Task handler raised error: <MaybeEncodingError: Error sending result: '"(1, <ExceptionInfo: RetryError('<Future at 0x7fe9d6ac96d8 state=finished raised KeyError>')>, None)"'. Reason: ''PicklingError("Can\'t pickle <class \'tenacity.RetryError\'>: it\'s not the same object as tenacity.RetryError")''.>
Traceback (most recent call last):
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/billiard/pool.py", line 366, in workloop
    put((READY, (job, i, result, inqW_fd)))
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/billiard/queues.py", line 366, in put
    self.send_payload(ForkingPickler.dumps(obj))
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/billiard/reduction.py", line 56, in dumps
    cls(buf, protocol).dump(obj)
billiard.pool.MaybeEncodingError: Error sending result: '"(1, <ExceptionInfo: RetryError('<Future at 0x7fe9d6ac96d8 state=finished raised KeyError>')>, None)"'. Reason: ''PicklingError("Can\'t pickle <class \'tenacity.RetryError\'>: it\'s not the same object as tenacity.RetryError")''.```

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions