Archiving really big mailboxes

[DunklerPhoenix]

Bug Report

• I have checked that there is no existing issue dealing with this problem.

Summary

It seems that Eonvelope tries to pull all emails from a mailbox at once. This result in a timeout or in an systemexit (maybe because of too much used ressources).

Eonvelope version

0.5.2

Steps to reproduce *

1. have a really big mailbox (in my case archive with 6.9GB and 200k+ mails)
2. try to get it once via ALL MAIL creteria
3. get flooded by system exit and timouts :D

Bug behaviour *

Expected behaviour *

Maybe it would be better to pull mailboxes only in background via worker with multiple tries e.g. 200 mails per call.

Logs *

2026-02-01 15:10:47,604 ERROR - core.utils.fetchers.IMAP4_SSL_Fetcher.__exit__: An error <class 'SystemExit'> occurred exiting Fetcher!
Traceback (most recent call last):
  File "/opt/src/core/models/Mailbox.py", line 177, in test
    fetcher.test(self)
    ~~~~~~~~~~~~^^^^^^
  File "/opt/src/core/utils/fetchers/IMAP4Fetcher.py", line 182, in test
    self.safe_select(utf7_encode(mailbox.name), readonly=True)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/src/core/utils/fetchers/SafeIMAPMixin.py", line 161, in safe_action
    response = imap_action(self, *args, **kwargs)
  File "/opt/src/core/utils/fetchers/SafeIMAPMixin.py", line 208, in safe_select
    return self._mail_client.select(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.14/imaplib.py", line 861, in select
    typ, dat = self._simple_command(name, mailbox)
               ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.14/imaplib.py", line 1372, in _simple_command
    return self._command_complete(name, self._command(name, *args))
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.14/imaplib.py", line 1174, in _command_complete
    typ, data = self._get_tagged_response(tag, expect_bye=logout)
                ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.14/imaplib.py", line 1315, in _get_tagged_response
    self._get_response()
    ~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.14/imaplib.py", line 1217, in _get_response
    resp = self._get_line()
  File "/usr/local/lib/python3.14/imaplib.py", line 1325, in _get_line
    line = self.readline()
  File "/usr/local/lib/python3.14/imaplib.py", line 393, in readline
    buf = self.sock.recv(DEFAULT_BUFFER_SIZE)
  File "/usr/local/lib/python3.14/ssl.py", line 1285, in recv
    return self.read(buflen)
           ~~~~~~~~~^^^^^^^^
  File "/usr/local/lib/python3.14/ssl.py", line 1140, in read
    return self._sslobj.read(len)
           ~~~~~~~~~~~~~~~~~^^^^^
  File "/usr/local/lib/python3.14/site-packages/gunicorn/workers/base.py", line 204, in handle_abort
    sys.exit(1)
    ~~~~~~~~^^^
SystemExit: 1

2026-02-01 15:20:21,167 ERROR - core.utils.fetchers.IMAP4_SSL_Fetcher.__exit__: An error <class 'core.utils.fetchers.exceptions.MailboxError'> occurred exiting Fetcher!
Traceback (most recent call last):
  File "/opt/src/core/utils/fetchers/SafeIMAPMixin.py", line 161, in safe_action
    response = imap_action(self, *args, **kwargs)
  File "/opt/src/core/utils/fetchers/SafeIMAPMixin.py", line 208, in safe_select
    return self._mail_client.select(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.14/imaplib.py", line 861, in select
    typ, dat = self._simple_command(name, mailbox)
               ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.14/imaplib.py", line 1372, in _simple_command
    return self._command_complete(name, self._command(name, *args))
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.14/imaplib.py", line 1174, in _command_complete
    typ, data = self._get_tagged_response(tag, expect_bye=logout)
                ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.14/imaplib.py", line 1315, in _get_tagged_response
    self._get_response()
    ~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.14/imaplib.py", line 1217, in _get_response
    resp = self._get_line()
  File "/usr/local/lib/python3.14/imaplib.py", line 1325, in _get_line
    line = self.readline()
  File "/usr/local/lib/python3.14/imaplib.py", line 393, in readline
    buf = self.sock.recv(DEFAULT_BUFFER_SIZE)
  File "/usr/local/lib/python3.14/ssl.py", line 1285, in recv
    return self.read(buflen)
           ~~~~~~~~~^^^^^^^^
  File "/usr/local/lib/python3.14/ssl.py", line 1140, in read
    return self._sslobj.read(len)
           ~~~~~~~~~~~~~~~~~^^^^^
TimeoutError: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/src/core/models/Mailbox.py", line 177, in test
    fetcher.test(self)
    ~~~~~~~~~~~~^^^^^^
  File "/opt/src/core/utils/fetchers/IMAP4Fetcher.py", line 182, in test
    self.safe_select(utf7_encode(mailbox.name), readonly=True)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/src/core/utils/fetchers/SafeIMAPMixin.py", line 168, in safe_action
    raise exception_class(error, imap_action.__name__) from error
core.utils.fetchers.exceptions.MailboxError: Ein TimeoutError: The read operation timed out trat während safe_select im Postfach auf!

Screenshots

Files/Emails

Possible diagnosis

Fix it yourself?

EDIT: Sry for that edge case ^^'

[Dacid99]

Yup I am already aware of this issue and the fix you propose (background worker, async fetching, etc) is in my todo list for a while now. I'm currently waiting until I can update to django version 6, where background tasks like this will be easier to implement.

In your case the timeout seems to occur on select (IMAP command for opening) of the mailbox. The imapserver tries to compile a list of all email ids in that mailbox in that process, and taking too much time with that. Funny enough, you haven't even reached the email download yet.
You can increase the timeout value of the account, so the server is granted more time for each operation and see whether this helps.

To transfer all emails from that humongous mailbox to Eonvelope, I recommend exporting them from your server in a format like mbox (thunderbird has an addon for this) and uploading that file to the mailbox in Eonvelope. That's still gonna take a lot of time to finish but there will be less overhead.

This is a good example why fetching all emails at once can be a bad idea. Even if the thing was finishing in the background, it would take a while.

[DunklerPhoenix]

Good good :)
Jup I did that often in the past while switching the mail providers ^^'
[EDIT: even thunderbird burns down with so many emails xD]

Ah a little bit of off-topic. Do you plan to implement also an auto deletion after some time or something like a duplicate check? My normal workflow is: Email comes into inbox and goes to archive if done.
How would this work with Eonvelope? I think the best thing here is to just backup the archive, but what if someone add routines for inbox and archive? Does the mail then get saved multiple times?

[Dacid99]

The idea of Eonvelope is that you can change your workflow to

• Email comes into inbox
• Email is picked up by Eonvelope routine
• You delete the email when done
• If you need the mail again you can search it on eonvelope and restore it if necessary or download it or its attachments.

If you stick with your current flow you are right that a routine for the archive (and for Sent, to get outgoing mails) is the best solution.

Emails that have been found in two different mailboxes are stored twice, but only once for every mailbox.
I currently don't plan to have auto-deletion for ancient emails, but depending on user interest that may change.

[Dacid99]

I just committed emails from IMAP being fetched in batches of 100 (19753cf). Previously they were fetched one by one, with a lot of overhead from the network. For large mailboxes, there should be a significant speedup now

[Kitround]

Hi everyone. I'm confronted to the same problem with a mailbox of 22k mails right now, getting the container to reboot. How do you find the parameter to got the fetch "all mails" by batch ? Or it's not yet implemented ? Thank you.

[Dacid99]

Hey, the batch size is currently hard coded to 100.

It's not a wise idea to fetch a lot more emails at once, as you will run into an imap request timeout, essentially archiving nothing.

Why does the container restart when you try to archive the mailbox? I would expect the fetch process to run for a (very) long time but nothing that would crash the container.
Are there any logs for this crash event?

[Kitround]

Seems like the memory is getting full quickly and making the container out of it and then crash + restart. The main reason on the log is : "[178] [ERROR] Worker (pid:180) was sent SIGKILL! Perhaps out of memory?"

Even with 6go of RAM and more proc it doesn't change anything.

[Dacid99]

Ok that makes sense. I know how to fix it and I'll get right on it

[Dacid99]

I have changed a small thing about the code that fetches the emails to make this a lot more memory-efficient.

Until now, all emails were fetched and stored in memory, only after that they were saved to the database and archived.

Starting with commit faf0d0c, after an email is fetched, it is saved and archived right away.
This should resolve the issue you see where the container runs out of memory.

All unittests pass, I'm still gonna take this for a manual test run tomorrow.
After that I'll do some routine updates of dependencies and release a new version that contains this fix.