#!/bin/cat
$Id: FAQ.Big_Mailbox.txt,v 1.13 2024/10/24 23:07:23 gilles Exp gilles $

This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Big_Mailbox.txt


=====================================================================
            Imapsync tips to deal with huge mailboxes
=====================================================================

Questions answered in this FAQ are:

Q. How to deal with huge mailboxes, whose size is over dozens of GB?
Q. How to know if imapsync is stalling or working?
Q. How to deal with huge mailboxes when imapsync fails to handle them?

Now the questions again with their answers.

=====================================================================
Q. How to deal with huge mailboxes, whose size is over dozens of GB?

A. It should be ok with imapsync. Keep reading if it doesn't.

=====================================================================
Q. How to know if imapsync is stalling or working?

Sometimes with big folders, imapsync seems to stall when sizing the
folders or maybe before syncing a huge folder. So you wonder whether
imapsync is doing something or just being frozen. 

A1. A solution to increase the verbosity is to add the option
--debugimap. Option --debugimap shows what is currently done, it's
quite a big output but it helps to wait, saying to ourselves "ok it's
long but it's working, let's wait a little more".  The real purpose of
--debugimap is to show genuine IMAP commands used and their
responses. The problem with --debugimap is that it is very verbose and
so with a big mailbox it generates a very BIG logfile.

A2. Another solution is to monitor the imapsync process with standard
Linux tools like strace, iftop, nethogs etc.:

  strace -f -p 12345 # print all the system calls of process PID 12345

  iftop -f 'port imap or port imaps' -B # then press t p > 

  nethogs # see all processes
  
  nethogs -P 12345 # Only the process with PID 12345

You can have the imapsync PID with the command:

  head -1 $HOME/tmp/imapsync.pid 

You can also set the pidfile with the imapsync option --pidfile

  imapsync ... --pidfile whatever.txt

and use it in the previous commands:

  strace -f -p `head -1 whatever.txt`
  nethogs   -P `head -1 whatever.txt`
  

=====================================================================
Q. How to deal with huge mailboxes when imapsync fails to handle them?

A1. If it is not ok with imapsync, the sync finally fails, then 
consider spliting the whole sync into smaller chunks using 
the option --search

For example, select messages only for a given year:

...
..imapsync ... --search "SENTSINCE 1-Jan-2017 SENTBEFORE 31-Dec-2017"
  imapsync ... --search "SENTSINCE 1-Jan-2018 SENTBEFORE 31-Dec-2018"
  imapsync ... --search "SENTSINCE 1-Jan-2019 SENTBEFORE 31-Dec-2019"

A2. Use option --useuid
Option --useuid speeds up the process but --useuid implies --usecache
and --usecache may trigger some issues sometimes. Read:
https://imapsync.lamiral.info/FAQ.d/FAQ.Use_cache.txt

Don't use --useuid in Docker context, the cache will be destroyed 
each time, or put it in a stable place, I don't know how to do it,
I just know it's possible and not very difficult.

A3. Another solution.
We installed Thunderbird, entered the mailbox from which we need to 
copy emails. We used the "Archive" function for Incoming messages.
The emails were divided into folders by year. 
After that, all folders were synchronized.
Thanks to vasilieva_v for this tip.

=====================================================================
=====================================================================