RQ and Logtail/BetterStack issue

[mistalaba]

Hi!

I was having an issue with RQ logs not being picked up by Logtail. Here's my Django setup (sorry for the formatting):

settings.py:

LOGGING = {'version': 1,
 'disable_existing_loggers': False,
 'formatters': {'verbose': {'format': '[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s',
   'datefmt': '%d/%b/%Y %H:%M:%S %Z'},
  'simple': {'format': '%(levelname)s %(message)s'}},
 'filters': {'require_debug_false': {'()': 'django.utils.log.RequireDebugFalse'},
  'require_debug_true': {'()': 'django.utils.log.RequireDebugTrue'}},
 'handlers': {
  'logtail': {'class': 'logtail.LogtailHandler',
   'source_token': 'xxx',
   'host': 'https://xxx.betterstackdata.com',
   'level': 'INFO',
   'filters': ['require_debug_false']}},
 'loggers': {'': {'handlers': ['console', 'file', 'logtail'],
   'level': 'DEBUG'},
  'django': {'handlers': ['console', 'file', 'logtail'],
   'level': 'INFO',
   'propagate': False},
  'rq': {'handlers': ['console', 'file', 'logtail'],
   'level': 'DEBUG',
   'propagate': False},
  }} 

core/tasks.py:

import logging
from django_rq import job

logger = logging.getLogger(__name__)

# An experiment for routing logging via RQ
@job("default")
def log_via_rq(message):
    from core.rq_workers import log_experiment

    log_experiment()
    logger.info(f"📬 Logging via RQ: {message}")

core/rq_workers.py:

import logging

def log_experiment():
    logger = logging.getLogger(__name__)
    logger.info("📧 This is a log message inherited by log_via_rq function in core.tasks.py")

Then, in python manage.py shell:

In [1]: from core.tasks import log_via_rq

In [2]: log_via_rq.delay('hellojsan')
Out[2]: Job('1bad5e06-219f-49d2-8fc3-e452b36e0769', enqueued_at=datetime.datetime(2026, 3, 23, 8, 11, 54, 858738, tzinfo=datetime.timezone.utc))

And finally, the output in BetterStack:

Here's BetterStack (couldn't copy formatted):

2026-03-23 09:10:49.407 CET
myapp
info
Worker cc6750f9ea7a429ab37a43f74e8dba85: cleaning registries for queue: default ../.venv/lib/python3.14/site-packages/rq/worker/base.py:434
rq.worker
2026-03-23 09:11:54.872 CET

myapp
info
default: core.tasks.log_via_rq('hellojsan') (1bad5e06-219f-49d2-8fc3-e452b36e0769) ../.venv/lib/python3.14/site-packages/rq/worker/base.py:1095
rq.worker
2026-03-23 09:13:03.815 CET

myapp
info
pikepdf C++ to Python logger bridge initialized ../lib/python3.14/site-packages/pikepdf/__init__.py:13
pikepdf._core

After a lot of debugging, I managed to work around it by creating a custom RQ Worker class and assigning that to RQ:

class LoggingWorker(Worker):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        logging.getLogger(__name__).info("✅ LoggingWorker instantiated")

    def main_work_horse(self, job, queue):
        for handler in logging.getLogger().handlers:
            if isinstance(handler, LogtailHandler):
                handler.ensure_flush_thread_alive()
        super().main_work_horse(job, queue)

    def perform_job(self, job, queue):
        try:
            return super().perform_job(job, queue)
        finally:
            for handler in logging.getLogger().handlers:
                if isinstance(handler, LogtailHandler):
                    handler.flush()
                    if hasattr(handler, "flush_thread") and handler.flush_thread is not None:
                        handler.flush_thread.join(timeout=2)

But, it would be nice if Logtail could handle this. Is there a way?

Thanks! (And please let me know if you need more info!)