News about next imapsync, currently distributed 2.229, next and previous releases (back to menu)

Roadmap

If you are very interrested in those projects, don't hesitate to solicit me!

• Roadmap: Build a docker image including everything, to have a complete online /X in a single command.
• Roadmap: Or better: Include a webserver in imapsync to have an online /X integrated. (NB: proof of concept already coded and working)
• Roadmap: Add the sync of multiple accounts inside imapsync itself instead of via an external shell script doing the loop from a csv file.
• Roadmap: Build an online tool, like /X, for multiple accounts at once. (NB: proof of concept already coded and working)

New features or bugfixes since previous releases:

imapsync 2.229 was written on Wednesday, 14-Sep-2022 18:09:17 UTC

See ChangeLog to read what has been done in details since 2001.

• 2.229
• General:
• Dependencies added: Proc::ProcessTable
• Dependencies removed: None
• Enhancement: Added a stat value: memory footprint times time spent during the sync. The unity is MiB * hour. The stat line is "Memory consumption at the end".
• Enhancement: Filtering buggy flags is now case insensitive. \Junk or \JUNK or \jUnK etc. (sub tests_filterbuggyflags)
• Enhancement: Added --memorystress to check memory crunching in normal run.
• Usability: Do not email final reports by default. Use --emailreport1 and --emailreport2 to add final emails reports in each INBOX.
• CGI context:
  • Refuse to serve when the cpu load is greater than 1 per cpu. A load of 3.9 with 4 cores is ok, 4.1 is not. (sub heavy_load_reached_by_cpu)
  • Refuse to serve when the RAM memory available for imapsync processes is less than 1 GB, presented by percentage on the first line like "64.80% used by processes". (sub heavy_load_percent_threshold)
  • When refuse to serve, exits with value 69 EX_UNAVAILABLE
• Testing: 1992 unit tests

• 2.200
• Dependencies removed: None
• Dependencies added: None
• Enhancement: Append a final email report on each account at the end of the synchronization. Use --noemailreport1 and --noemailreport2 to avoid final emails reports in each INBOX.
• Enhancement: Documented why total sizes can differ even when the sync is perfect.
• Enhancement: Added SERVER_NAME SERVER_ADDR SERVER_ADMIN variables and values to the output.
• Enhancement: Added the local ip address for the imap connexions. It can help configuring firewalls to allow the imap source IP.
• Enhancement: Added "Posta inviata" for \Sent with --automap
• CGI context: Upped ERRORS_MAX_CGI from 20 to 500.
• CGI context: Added --var to get values from proximapsync for variables REMOTE_ADDR REMOTE_HOST HTTP_REFERER HTTP_USER_AGENT SERVER_SOFTWARE SERVER_PORT HTTP_COOKIE
• Usability: Added warning "parsing headers of folder ... It can take time for huge folders. Be patient."
• Bug fix: Do not turn on --delete2duplicates when --syncduplicates is on, unless --delete2duplicates is given.
• Bug fix: Redirect STDERR to STDOUT in all cases, --log or --nolog
• Bug fix: Do not allow --skipcrossduplicates and --usecache. Exit EX_USAGE 64
• Bug fix: Dates were not displayed under Windows because POSIX::strftime %e is not portable. Fixed several other places where dates were wrong on Windows.
• Bug fix: --tests --testslive was listing 0 folders.
• Refactoring: Deglobalized $debuglist $debugflags
• Refactoring: Added sync->{ permanentflags2 }

• 2.178
• Dependencies removed: None
• Dependencies added: Perl module Compress::Zlib
• Enhancement: Added final statistic line: "CPU time and %cpu : 13.27 sec 8.3 %cpu 2.1 %allcpus
• Enhancement: Removed the hard limit --maxsize 35_651_584 set by --gmail2 since CAPABILITY is used and good.
• Enhancement: Does not checknoabletosearch if --justfolders
• Enhancement: When creating a folder, check if it is selectable and consider ok if it is selectable.
• Enhancement: Options --gmail1 --gmail2 together sets --useheader X-Gmail-Received --useheader Message-Id
• Enhancement: Added compression in the imap connexions. Compression is off by default no matter the capability announced. Use --compress1 and --compress2 to turn compression on.
• Enhancement: Added --keepalive1 and --keepalive2 options. Some firewalls and network gears like to timeout connections prematurely if the connection sits idle. This option enables SO_KEEPALIVE on the host1 socket. --keepalive1 is on by default since imapsync release 2.169 Use --nokeepalive1 to disable it. Same thing for --keepalive2 but for --host2. See https://metacpan.org/pod/Mail::IMAPClient#Keepalive
• Usability: Removed --fast option. It didn't do anything anyway, it was a fake option.
• Usability: Added advice inline advice on errors
  • ERR_OVERQUOTA. "The destination mailbox is 100% full, get free space on it and then resume the sync."
  • ERR_TRANSFER_EXCEEDED. "The maximum transfer size for a single sync is reached ( over 3 GB ). Relaunch the sync to sync more."
  • ERR_APPEND_SIZE. "The destination server refuses too big messages. Use --truncmess option. Read https://imapsync.lamiral.info/FAQ.d/FAQ.Messages_Too_Big.txt"
  • ERR_CONNECTION_FAILURE_HOST1. "Check that host1 on port1 is the right IMAP server to be contacted for your mailbox."
  • ERR_CONNECTION_FAILURE_HOST2. "Check that host2 on port2 is the right IMAP server to be contacted for your mailbox."
  • ERR_AUTHENTICATION_FAILURE_USER1. "Check the credentials for user1."
  • ERR_AUTHENTICATION_FAILURE_USER2. "Check the credentials for user2."
• Usability: Bytes sizes human readable are now presented in KB MB GB TB PB (1000 base) instead of KiB MiB GiB TiB PiB (1024 base)
• Usability: Added README help for --truncmess
• CGI context: Experimental loaddelay forced to 0 (no delay at all). So far a huge load is rarely a problem.
• CGI context: Bugfix. Abort didn't set charset=UTF-8; same thing for heavy load 503 Service Unavailable
• Bug fix: Made reconnect mechanism works with --oauthdirect1 --oauthdirect2 --oauthaccesstoken1 --oauthaccesstoken2
• Bug fix: Upped the maximum number a pid can be. From 99999 to 2^22. See what-is-the-maximum-value-of-the-process-id
• Bug fix: Load average on Mac was sometimes buggy. locale float 3,14 instead of 3.14
• Bug fix: errors_incr() exited with CATCH_ALL
• Bug fix: Memory consumption on Mac was not relevant. Use RSS instead of VSZ, on Mac only. Linux stays with VSZ.
• Docker context: Current working directory changed from /var/tmp to /var/tmp/uid_$EFFECTIVE_USER_ID

• 2.140
• General: New release numbering from 1.999 directly to 2.100; I want to avoid 1.1000 and sorting issues in directory listing.
• Dependencies removed: None
• Dependencies added: None
• Enhancement: Check if SEARCH ALL works on both accounts and then add --noabletosearch if one can not. Use --nochecknoabletosearch to avoid that behavior.
• Enhancement: Added an analyse and classification of the errors encountered. Most common error encountered is printed at the end. Exit value 112 (EXIT_WITH_ERRORS_MAX) is now replaced by the most common error encountered exit value:

       EX_OK          => 0  ; #/* successful termination */
       EX_USAGE       => 64 ; #/* command line usage error */
       EX_NOINPUT     => 66 ; #/* cannot open input */
       EX_UNAVAILABLE => 69 ; #/* service unavailable */
       EX_SOFTWARE    => 70 ; #/* internal software error */
       EXIT_CATCH_ALL              =>   1 ; # Any other error
       EXIT_BY_SIGNAL              =>   6 ; # Should be 128+n where n is the sig_num
       EXIT_BY_FILE                =>   7 ;
       EXIT_PID_FILE_ERROR         =>   8 ;
       EXIT_CONNECTION_FAILURE     =>  10 ;
       EXIT_TLS_FAILURE            =>  12 ;
       EXIT_AUTHENTICATION_FAILURE =>  16 ;
       EXIT_SUBFOLDER1_NO_EXISTS   =>  21 ;
       EXIT_WITH_ERRORS            => 111 ;
       EXIT_WITH_ERRORS_MAX        => 112 ;
       EXIT_OVERQUOTA              => 113 ;
       EXIT_ERR_APPEND             => 114 ;
       EXIT_ERR_FETCH              => 115 ;
       EXIT_ERR_CREATE             => 116 ;
       EXIT_ERR_SELECT             => 117 ;
       EXIT_TRANSFER_EXCEEDED      => 118 ;
       EXIT_ERR_APPEND_VIRUS       => 119 ;
       EXIT_TESTS_FAILED           => 254 ; # Like Test::More API
       EXIT_CONNECTION_FAILURE_HOST1     =>  101 ;
       EXIT_CONNECTION_FAILURE_HOST2     =>  102 ;
       EXIT_AUTHENTICATION_FAILURE_USER1 =>  161 ;
       EXIT_AUTHENTICATION_FAILURE_USER2 =>  162 ;
  

• Enhancement: Added option --dry1 it avoids to fetch the message on host1 in --dry mode. Option --dry1 is on when --dry is on. It speeds up --dry very much. To get the old --dry behavior, use --dry --nodry1
• Enhancement: Do both sides login and authentication before exiting on failure.
• Enhancement: --timeout can now be a float (was only integer). Same for --timeout1 --timeout2
• Enhancement: Added option --syncduplicates to sync duplicates. On by default in CGI context. Off by default on command line.
• Enhancement: Added option --trylogin. After a login failure of any kind, imapsync tries to login with the LOGIN command. Use --notrylogin to avoid that. --trylogin is on by default.
• Enhancement: Added options --oauthaccesstoken1 and --oauthaccesstoken2 to support OAUTH2 authentication.
• Enhancement: Added options --oauthdirect1 and --oauthdirect2 to support OAUTH2 authentication.
• Enhancement: Added statistic line "CPU time and %cpu". Example: "CPU time and %cpu : 9.2 sec 39.8 %cpu 19.9 %allcpus".
• Enhancement: Added option --filterbuggyflags to filter flags known to be buggy and generators of errors "BAD Invalid system flag" or "NO APPEND Invalid flag list".
• CGI context: Back from Message-Id to Message-Id Received in cgisetcontext
• CGI context: In CGI context, the log directory is no longer ./LOG_imapsync/ but simply ./ It should fasten the searches and grep
• CGI context: Added --loglogfile option. Append ../list_all_logs_auto.txt with the logfile name. On by default in CGI context.
• Usability: Changed time presentation on ETA and in final stats. Example: Wednesday 16 June 2021-06-16 00:32:05 +0200 CEST
• Usability: Do not check selectable folders for huge accounts with more than 152 folders, 98 percentile of /X; It could be very long and ended up with some users transformed in stone just by staring at the fixed output.
• Bug fix: The number of detected cpus (cores) was accurate only on Linux. Added Freebsd and fixed Darwin and Windows parts.
• Refactoring: Continuing to deglobalize the global variables.
• Refactoring: 1734 unit tests.

• 1.977
• Dependencies removed: None.
• Dependencies added: Perl modules Encode::IMAPUTF7
• Windows: imapsync.exe is now 64 bits (it was a 32bit binary before). If you use an old 32bit PC, use imapsync_32bit.exe instead.
• Enhancement: Folders sizes are now presented per folder for host1 and host2, as well as the differrences between them. Comparing folders is now easier
• Enhancement: cgidir is . when used by Net::Server::HTTP
• Enhancement: Added context in --releasecheck user_agent. CGI or Docker or Standard.
• Usability: Added ETA at the beginning of a folder process (it is also to improve the /X progress bar accuracy).
• Usability: Added the number of errors and the max number of errors allowed at the final line. Example: "Exiting with return value 0 (EX_OK: successful termination) 0/50 nb_errors/max_errors"
• Bug fix: Folders names with + characters must keep their + as they are when prensenting them in uft8 (in folders listings [] = [], right part)
• Bug fix: PIDs can be negative under Windows.
• Bug fix: --resynclabels was not on with --gmail1 --gmail2
• Bug fix: Count a fatal error as a error before presenting the count errors when exiting.
• Bug fix: Docker context. Killing myself in docker context doesn't seem to work well, as well as sending signals several times. So just install catch_exit with INT (ctr-c on terminals) and exits if still alive after killing myself.
• Bug fix: Option --justconnect did not show the line "IP address:"
• Bug fix: Reviewed all exit_clean to include a final CR on the error message.
• Bug fix: Added SSL_cipher_list => 'DEFAULT:!DH' to tests with ssl.
• Bug fix: When using --subfolder2 SUB and --delete2folders, do not delete the folder SUB (even it is not on host1).
• Bug fix: CGI context. charset was charset=ISO-8859-1 in header (CGI.pm default). Now "text/plain; charset=UTF-8"
• Bug fix:
• Refactoring: Moved main instructions into sub single_sync(). The goal is to prepare multiple different syncs in imapsync.
• Refactoring: Cgi context is now set on with env variables SERVER_SOFTWARE and SERVER_PORT (was only SERVER_SOFTWARE). It is because Net::Server::HTTP does not set SERVER_SOFTWARE but SERVER_PORT. It is a preparation to a homemade web server to serve on imapsync as a cgi and get free from Apache or a complicated web server stuff configuration.

• 1.945 (2019/06/25)
• Dependency added: Perl modules Text::ParseWords and File::Tail
• Enhancement: A new /X including:
  • Options like --dry --justlogin --justfolders --justfoldersizes,
  • An extra mode with --delete1 --delete2 --subfolder1 --subfolder2 in order to make backup/restore syncs online,
  • A button to reverse source and destination parameters.
• Usability: Added the error message text after "Exiting with return value xx". Example: Exiting with return value 16 (EXIT_AUTHENTICATION_FAILURE)
• Usability: Now "imapsync --host1 imap.lamiral.info" or "imapsync --host2 127.0.0.1" or both do like a --justconnect on servers. Useful to check rapidely an imap server. Caveat: no longer a command line error.
• Usability: Added some error message texts explaining the exit numbers. Example: "Exiting with return value 16 (EXIT_AUTHENTICATION_FAILURE)"
• Usability: Added what function is called when installing a signal. Example: "kill -QUIT 19839 # special behavior: call to sub catch_exit"
• Usability: Now --justfolders activates --foldersizesatend
• Usability: Moved the warning "no header by parse_headers so taking whole header with BODY.PEEK[HEADER]" to --debug mode.
• Usability: Moved the warning "no header found so adding our own" to --debug mode.
• Bug fix: USR1 signal for toggle_sleep call was never installed.
• Bug fix: The number of messages deleted reported at the end was not accurate at all.
• Bug fix: Change the longest line in --exchange2 and --office from 10500 to 10239. (\n inserted for those long lines)
• Enhancement: Added PROCESS_ID after "Tail -f finished."
• Enhancement: --maxage and --minage can take float as value in order to filter part of a day. (Need --noabletosearch also).
• Enhancement: Gmail context. Added --resynclabels to resync the Gmail labels X-GM-LABELS.
• Enhancement: Backup/Restore context. Added --subfolder1 "Foo/Bar" option in order to restore a backup made with --subfolder2 "Foo/Bar". The option --folderrec "Foo/Bar" is implied by --subfolder1 "Foo/Bar" because it's the most common purpose. Also, options --subfolder1 and --subfolder2 turn off --automap if it is set because no one would have it set in this context.
• Enhancement: All exits are now done with a specific return code related to the error causing the exit. Here is the list of the exit code values (an integer between 0 and 255), the names reflects their meaning:

       EX_OK          => 0  ; #/* successful termination */
       EX_USAGE       => 64 ; #/* command line usage error */
       EX_NOINPUT     => 66 ; #/* cannot open input */
       EX_UNAVAILABLE => 69 ; #/* service unavailable */
       EX_SOFTWARE    => 70 ; #/* internal software error */
       EXIT_CATCH_ALL              =>   1 ; # Any other error
       EXIT_BY_SIGNAL              =>   6 ; # Should be 128+n where n is the sig_num
       EXIT_PID_FILE_ERROR         =>   8 ;
       EXIT_CONNECTION_FAILURE     =>  10 ;
       EXIT_TLS_FAILURE            =>  12 ;
       EXIT_AUTHENTICATION_FAILURE =>  16 ;
       EXIT_SUBFOLDER1_NO_EXISTS   =>  21 ;
       EXIT_WITH_ERRORS            => 111 ;
       EXIT_WITH_ERRORS_MAX        => 112 ;
       EXIT_TESTS_FAILED           => 254 ; # Like Test::More API
  

• Usability: Gmail context. Activate --resynclabels by default with --gmail1 --gmail2
• Usability: The maximum size of a message is the minimum of --maxsize and APPENDLIMIT=xxx (given the imap servers) or each one if the other no exists.
• Usability: CGI context. Added default like --maxsize 1_000_000_000 (Out of memory prevention?).
• Usability: CGI context. Only use "Message-Id" to identify messages, instead of "Message-Id" + "Received". We'll see.
• Usability: CGI context. --addheader on by default. In order to remove it from the /X call. Still allow it as an option.
• Enhancement: Added option --appendlimit xxxx in order to override the value got by APPENDLIMIT in CAPABILITY. The goal is to permit --truncmess of large messages.
• Enhancement: Added option --truncmess to allow transfer of too big messages. Use it if you think it's better to have a truncated message than no message at all.
• Usability: CGI context. Instead of quit with "another imapsync may be curently running", do like a tail -f on the other running sync, if the standart (crendentials) parameters are the same.
• Usability: CGI context. A heavy load is now logged.
• Usability: CGI context. Tolerate more load. Double the threshold. Now accept load at 2 per core instead of 1 per core
• Usability: CGI context. Remove edging blanks of the 6 options --hostX --userX --passwordX (where X = 1 or 2). It will solve some login failures on /X just because blanks are not visible but sometimes there, copy/paste my friend you're not always my friend... use --nosanitize to avoid it.
• Bug fix: With --gmail1 or --gmail2 do not sync [Gmail]/All Mail if it is not requested.
• Bug fix: When asked to terminate by a signal, kill myself by the signal instead of an exit call. Applied Martin Cracauer's advice on Proper handling of SIGINT/SIGQUIT. I hope it fixes the not always working double-ctrl-c to end the sync.
• Bug fix: Fixed tests to pass imapsync --tests under the docker image.
• Bug fix: Gmail context. --synclabels --subfolder2 "Foo" create only sublabels under "Foo" and not a mixture with root labels.
• Bug fix: --subfolder2 "Pf2/Sub2" and --subfolder2 "Sub2" perform the same when "Pf2" is the prefix of host2
• Bug fix: imapsync now syncs messages whatever their proclaimed size are. (ProtonMail Bridge 1.1.3 does RFC822.SIZE null for all messages).
• Refactoring: Fixed some perl critics.
• Refactoring: Deglobalized some variables
• Security: Sanitized --subfolder1 and --subfolder2 values.
• Portability: Upgraded Mac binary to handle TLSv1.2
• Roadmap: Will add --subfolder1 and --subfolder2 parameters to /X in order to make backup/restore syncs.

• 1.921 (2019/02/18)
• Enhancement: Script sync_parallel_unix.sh to parallelize massive syncs. It uses the powerful GNU parallel command.
• Enhancement: Added option --synclabels to sync Gmail labels. It should speed up Gmail to Gmail syncs. The option --synclabels applies only during a copy of a message for now. Resyncing Gmail labels will be coded later.
• Enhancement: Option --synclabels is activated with --gmail1 --gmail2 used together.
• Portability: The imapsync script Shebang (first line of imapsync) uses env now, instead of direct perl path. I'm not sure it's a good change that #!/usr/bin/perl becomes #!/usr/bin/env perl
• Enhancement: Final stats:
  • Added a counter of crossduplicates on host2 ( a crossduplicate is a message found accross different folders )
  • Added counter of "Messages found in host1 not in host2"
  • Added counter of "Messages found in host2 not in host1"
  • Added stat "Load end is"
  • Added comment on final stat about messages in host1 not in host2
  • Added comment on final stat about messages in host2 not in host1
  • Added comment on final stat about unidentified messages and an --addheader suggestion
  • Removed counters of "Total bytes duplicate host1" and "Total bytes duplicate host2"
  • Removed counter of "Total bytes error"
• Enhancement: Added authmech X-MASTERAUTH. To be used for Kerio with --authmech1 X-MASTERAUTH or --authmech2 X-MASTERAUTH ( Patch from https://github.com/imapsync/imapsync/pull/53/files )
• Enhancement: Added --sigprint HUP (signal HUP by default) in order to allow some asynchronous prints (with, for example, a kill -HUP IMAPSYNC_PID) in CGI mode and avoid timeouts from browser
• Enhancement: --skipcrossduplicates activated with --gmail1 alone
• Enhancement: --skipcrossduplicates is NOT activated with --gmail2 alone or with --gmail1 --gmail2 together used.
• Enhancement: Speed. Deletions on host1 are done by batch during a resync.
• Enhancement: Added automatic --maxsize from "CAPABILITY APPENDLIMIT=xxxx" (Gmail)
• Enhancement: Added --folderlast '[Gmail]/All Mail' with --gmail1 --gmail2
• Enhancement: With --office1 option, folder "Files" is excluded because it's a "special" folder. Like --exclude "^Files$". Unless --noexclude is used.
• Enhancement: With --office2 option, folder "Files" on host1 is renamed like --f1f2 "Files=Files_renamed_by_imapsync". Unless --nof1f2 is used.
• Enhancement: Added --exclude '^Files$' with --office2 option.
• Enhancement: Added inline explanations for the options --office1 --office2 --exchange1 --exchange2 (what they set and how to avoid what they set).
• Enhancement: Added --noregexmess to avoid the regexmess setting with --office2 and --exchange2 (wrap long lines >= 10500 char).
• Enhancement: Added --noregexflag to avoid the regexflag set by --exchange2 (remove \Flagged flag).
• Enhancement: Added IO::Socket::IP and CGI in the list of the modules versions.
• Usability: Added PPID info where PID is printed. Unix only. Can be useful in some contextes, CGI, embeded, debugging etc.
• Bug fix: Now split the deletion of messages in several chunks (using --split1)
• Bug fix: --debugssl 0 was impossible and equivalent to --debugssl 1
• Bug fix: Removed --nosubscribed usage since it is not coded and useless anyway.
• Bug fix: Option --maxsleep override variable $MAX_SLEEP in gmail*() subroutines.
• Bug fix: Do not create host2 folder when no messages are found on this folder on host1 (See https://github.com/imapsync/imapsync/issues/95)
• Bug fix: The regex to remove the flag \Flagged was wrong. It worked because of another filter run after.
• Bug fix: The options --exchange1 and --exchange2 were not options! No one complained...
• Bug fix: A pid can be up to 99999 on FreeBSD
• Refactoring: Deglobarised some global variables
• Refactoring: Refactored sub appendlimit() to use Mail::IMAPClient::capability like the API. No hack from internal $imap->{ APPENDLIMIT }
• Refactoring: Fixed some perl critics.
• Portability: Added freebsd support for loadavg and cpu_cores.
• Portability: Calls memory_consumption_of_pids_win32() within Cygwin environment (https://github.com/imapsync/imapsync/pull/22)
• Dependency removed: None.
• Dependency added: None.
• Roadmap: Proxy mode abandoned. Will be another lighter memory tool (since less memory consumption is the goal).

• 1.882 (2018/05/06) On the road to a proxy mode! (not done yet)
• Enhancement: Added option --nocheckfoldersexist ( Default is like --checkfoldersexist )
• Enhancement: Added option --noresyncflags : Do not resync flags for already transfered messages. May be useful when a user has already started to play with its host2 account.
• Enhancement: Option --abort now creates a log too, in order to see if imapsync ended by signal comes from option --abort or something else.
• Enhancement: Added option --sigexit; For example --sigexit QUIT --sigexit TERM to exit when receiving QUIT or TERM signals.
  Added option --sigreconnect. For example --sigreconnect INT to reconnect to both imap servers when receiving INT signal.
  Added option --sigignore. For example --sigignore TERM to "ignore" the TERM signal. In fact print stats during the sync.
  Default is like: --sigexit QUIT --sigexit TERM --sigreconnect INT
• Enhancement: Moved the hostname printing at first line. The goal is for clarity when in proxy mode.
  Added free RAM info in the banner.
  
• Enhancement: When using sleeping behavior, a signal USR1 toggles sleep from 2s to 0s (back to 2s with on next USR1 signal). Unix only.
• Usability: Detect extra arguments. Meaning usually a quoting issue in the command line or that options are not well parsed.
• Usability: Added inline documentation about --releasecheck and --noreleasecheck Default upstream is still --releasecheck. Just a line to change to have --noreleasecheck by default (as before)
• Usability: Added inline advice to resync after an end commanded by a TERM or QUIT signal.
• Bug fix: Fixed sub epoch() to allow that kind of silly INTERNALDATE 00-Jan-0000 00:00:00 +0000 (Thanks to IBM Domino...)
• Bug fix: --releasecheck activated was hardcoded in cgi context. Now it depends on the default behavior chosen. Bug pending. Still have to fix --releasecheck on with --help
• Bug fix: Added \t\r\n characters to be replaced by character _ in sub filter_forbidden_characters()
• Bug fix: fixes to pass --tests under root user.
• CGI context: Added SERVER_SOFTWARE and SERVER_PORT info.
• CGI context: Added free RAM, total RAM and hostname in the status.
• CGI context: Reduced the loadavg info in the header status.
• Refactoring: Fixed some perlcritics level 4.
• Refactoring: Deglobalized several global variables.
• Dependency added: Perl module Regex::Common

• 1.836 (2017/09/05) More secure by default, ssl or tls activation!
• Enhancement: An Imapsync Docker image available!
• Usability: Now goes to SSL by default if port 993 is open. Use --nosslcheck to avoid that.
• Usability: Now goes to TLS by default if possible, ie, only if STARTTLS is in CAPABILITY. If you want only TLS and nothing else, use --tls1 --nossl1
• Usability: Now if you want a basic imap connection on port 143 with no default encryption behavior, ie, no ssl nor tls, then use --nossl1 --notls1 for host1 and --nossl2 --notls2 for host2.
• Enhancement: Added --gmail1 and --gmail2 to simplify Gmail options setting. It sets parameters suggested in the Gmail FAQ --ssl, --host, etc.
• Enhancement: Added --office1 and --office2 to simplify Office 365 options setting. It sets parameters suggested in the Exchange/Office365 FAQ.
• Enhancement: Added --exchange1 and --exchange2 to simplify Exchange options setting. It sets parameters suggested in the Exchange FAQ.
• Enhancement: Added --domino1 and --domino2 to simplify Domino options setting. It sets parameters suggested in the Domino FAQ.
• Enhancement: Added --maxsleep in order to avoid timeouts with --maxbytespersecond and --maxmessagespersecond options. By default imapsync will sleep 2 seconds maximum, like if the command line contained --maxsleep 2
• Enhancement: Added --maxbytesafter in order to start --maxbytespersecond limitation only after --maxbytesafter amount of data transferred. Usefull for Gmail limits, for example, in order to active a 50K/s limit rate only after 500 MB of data transfer, use --maxbytesafter 500_000_000 --maxbytespersecond 50_000
• Enhancement: Added --testsunit in order to run any unit test individualy from the command line. Several --testsunit are allowed. Example: imapsync --testsunit tests_true --testsunit tests_always_fail
• Enhancement: Added password setting via environment variables IMAPSYNC_PASSWORD1 and IMAPSYNC_PASSWORD2
• Usability: No more useless and false warning "says it has NO CAPABILITY for AUTHENTICATE LOGIN"
• Usability: Options --delete1 and --delete are now aliases. Option --delete1 is preferable over --delete (--delete is still supported).
• Usability: Now prints always permanentflags info. It helps to understand most flag issues at first run, without --debugflags
• Usability: Now prints "could not append ( Subject:[$subject], Date:[$h1_date], Size:[$h1_size] )" when append fails.
• Usability: Option --showpasswords now shows also passwords with --debugimap. Useful to debug quoting issues.
• Usability: --ipv4 is now synonym of --inet4 and --ipv6 is now synonim of --inet6
• Usability: Added --testslive6 to check pure ipv6 connectivity.
• Enhancement: Added --noabletosearch1 and --noabletosearch2; Still support --noabletosearch, which turn on both --noabletosearch1 and --noabletosearch2
• Enhancement: Added --abort option to terminate a previous call still running. In command line context --abort uses the pidfile to know what to abort. In cgi context, ie online, exact same credentials are needed in order to really abort the other sync.
• Enhancement: Added milliseconds in the default logfile name since several runs is possible within one second in cgi context or on a powerful machine.
• Docker context: Added docker context in order to be run under the nobody user without permission issues.
• Docker context: Can run imapsync --tests under nobody user on Unix (or at least Linux).
• Bugfix: Fixed issue "SSL routines:ssl3_check_cert_and_algorithm:dh key too small" with openssl-dh-key-too-small-error SSL_cipher_list => 'DEFAULT:!DH'
• CGI context: Allow parameters passed by POST.
• CGI context: Abort, before doing anything, if the server load is already too heavy, and invite to come later depending on the current load (1, 5, or 15 minutes later).
• Bug fix: Guessed prefix is the empty string even when there is no folders (which is a bad sign anyway, since INBOX should be listed).
• Bug fix: Option --skipmess could not work most of the time. I guess it was a mistake arrived by badly converting an "unless" to an "if". Perl critics with no tests added => caveat emptor!
• Refactoring: Removed Mail::IMAPClient overload definitions
• Dependency added: IO::Socket::SSL
• Dependency added: Sys::MemInfo
• Dependency added: Pod::Usage

• 1.727 (2016/08/19) https website and CGI on the way!
• Security: Thanks to free and easy letsencrypt, Imapsync website is available in https: https://imapsync.lamiral.info/
• Enhancement: Added option --delete1emptyfolders. It deletes empty folders on host1, INBOX excepted. Useful with --delete since what remains on host1 is only what failed to be synced.
• Enhancement: Imapsync can be used as a CGI program. Still experimental. The web UI interface has to be written.
• Enhancement: XOAUTH2 authentication works with json files without extra tool openssl. Old pk12 files are still supported (still with openssl).
• Enhancement: Memory usage passed from 5x to 2x of the biggest message size. Some users do send 700MB messages, imapsync needed 3.5 GB of RAM, now it crunches only 1.4 GB, a more common RAM value available on small hosts.
• Enhancement: Added option --inet4 to force ipv4 in ssl and tls modes. Useful with Gmail and Office365 when ipv6 is disable on the imapsync host but not ipv6 resolution.
• Enhancement: Reconnect to imap servers with a Ctrl-c. Exit with two Ctrl-c within one second or one Ctrl-\
• Enhancement: Copy is not done when one --pipemess command ends with an empty message generated.
• Enhancement: STDERR of --pipemess commands are reported on STDOUT and in the logfile (they were ignored before).
• Enhancement:
• Dependency: Added Perl module dependency of Readonly to transform hard coded magic numbers into names. Mandatory module.
• Dependency: Added Perl module dependency of Test::MockObject for some tests. Only a "require" for now.
• Security: In CGI mode, remove all content in the unsafe "eval"uated options (--regextrans2 --regexflag --regexmess --skipmess --pipemess --delete2foldersonly --delete2foldersbutnot). Consider these options not available in CGI mode.
• Bug fix: Fixed -1 wrong output in statistics for big accounts (%d => %s) in printf.
• Bug fix: Uncommented call to $test_builder->reset( ) Don't know why it was commented but it was bad for knowing tests go wrong.
• Bug fix: Check connected state in sub create_folder(). Avoid multiple useless failures.
• Refactoring: Wrote Perl package Imapsync::Getopt::Long to handle CGI requests like command line options. Started as a copy of Luke Ross Getopt::Long::CGI
• Refactoring: Perlcritic cleanup 627 violations of CodeLayout::ProhibitTrailingWhitespace.
• Refactoring: Perlcritic cleanup 60 Double-sigil dereferences.
• Refactoring: Perlcritic cleanup 458 ValuesAndExpressions::ProhibitInterpolationOfLiterals. "Useless interpolation of literal string"
• Refactoring: Perlcritic cleanup 420 CodeLayout::ProhibitParensWithBuiltins. "Builtin function called with parentheses". (Severity: 1).
• Refactoring: Perlcritic cleanup, various small advices.
• Refactoring: Replaced print by myprint, printf by myprintf, sprinf by mysprintf. Will ease --quiet mode and general output handling.

• 1.684 (2016/03/17) Small changes, bugfixes
• Bug fix: Imapsync.exe updated to Mail-IMAPClient-3.38 (3.37 had a bug with credentials containing special characters)
• Usability: Added inline help about "use --noexpungeaftereach to speed up" if --delete is used.
• Usability: Added --sslargs1 in documentation, usage option list.
• Bug fix: No more warning about "Use of uninitialized value $sockargs[7] in join or string at" in ssl or tls mode. Was due to "'SSL_version' => undef".
• Bug fix: No more "Can't use an undefined value as a HASH reference at /usr/bin/imapsync line 1247."
• Bug fix: Added --expungeaftereach check in sub delete_message_on_host1().

• 1.678 (2016/01/21) Improved website! (I hope...)
• Enhancement: Added --sslargs1 and --sslargs2 to pass any ssl parameter for host1 or host2 connection. Example: --sslargs1 SSL_verify_mode=1 --sslargs1 SSL_version=SSLv3
• Enhancement: Added --timeout1 int and --timeout2 int in seconds (--timeout int still available to set both with the same value)
• Enhancement: Added --debugssl int. Default is like --debugssl 1 (Only print out errors).
• Enhancement: Added several polls to know where I shall focus improvements.
• Usability: Added env_proxy call in sub xoauth2() to read proxy settings from environment variable http_proxy without using PERL_LWP_ENV_PROXY=1
• Usability: Wrote TUTORIAL_Unix.
• Usability: Check f1 folder exists when specified by --f1f2 and warns when it doesn't
• Usability: Added --automap --justfolders --dry in imapsync_example.sh and imapsync_example.bat
• Bug fix: Added require Encode::Byte to solve "The locale codeset (cp1252) isn't one that perl can decode" on Win32.
• Refactoring: Removed --allow3xx option.
• Refactoring: Continue to move to one global $sync-> in order to reduce number of parameters in routines.

• 1.670 (2015/12/03) Folders mapping made easy with --automap
• Enhancement: Added option --automap that guesses folders mapping, for folders like "Sent", "Junk", "Drafts", "All", "Archive", "Flagged".
  • IMAP Special-Use folders described in rfc6154 are supported and also well known names for Exchange and common clients.
  • Automap is turned off by default, to try it a safe method is to use --automap --justautomap, it will exit early
• Enhancement: Added --f1f2 str1=str2 : Force folder str1 to be synced to str2.
  • Option --f1f2 overrides any --automap mapping and any --regextrans2 transformation.
  • Example --f1f2 Spam=Junk to map Spam folder to Junk.
• Enhancement: Added --justautomap to exit after seeing what will happen with --automap and --f1f2 options.
• Enhancement: Added IMAP4 QUOTA values (RFC 2087) when they are available. A warning goes up when it's time to find space.
• Enhancement: Added IMAP4 ID extension.
  • ID feature is on by default if supported by the IMAP servers, use --noid to turn it off.
  • ID Infos returned by the imap servers are shown.
  • Infos sent to the servers are name=imapsync, version=`imapsync --version`, os=$OSNAME`, vendor='Gilles LAMIRAL' and date=`rcs date of release`.
  • For github or other distributors you can hack this part by searching $imapsync_id_github
• Enhancement: Added --logdir path to change the default log directory. Default is LOG_imapsync/
• Usability: Added folders counting in outputs.
• Usability: Better output in the login part
• Usability: Guess prefixes and separators instead of forcing the user to find them.
• Usability: Output, added Host1: or Host2: at the beginning of lines.
• Bug fix: SSL_VERIFY_NONE in --ssl and --tls modes
• Bug fix: Fixed xoauth2 calls. Now XOAUTH2 Works on Windows --authmech1 XOAUTH2 with openssl installed. See FAQ.d/FAQ.XOAUTH2.txt.
• Bug fix: Fixed IO::Socket::SSL constants issue with latest releases
• Bug fix: Changed "Host1: checking all wanted folders exist" not efficient algorithm in order to allow efficiently a 2.4 million folders account. Yes, some accounts have this.
• Refactoring: Started to use global $sync-> in order to reduce number of parameters in routines.
• Refactoring: Some perlcritic fixes.

• 1.644 (2015/07/17) Mac OS X standalone binary provided!
• Enhancement: Mac OS X standalone binary called imapsync_bin_Darwin
• Enhancement: Added option --subfolder2 SUB Move whole host1 folders hierarchy under folder SUB.
• Enhancement: Added --fetch_hash_set "1:*" to permit Mail2World success. Need a patched Mail::IMAPClient 3.35 in sub fetch_hash()
• Usability: No folders sizes if --justfolders, unless really wanted.
• Usability: No warning about messages when --dry --justfolders are used together.
• Bug fix: Added NOOP in --dry mode during fake APPEND in order to avoid timeouts

• 1.636 (2015/03/31)
• Enhancement: Added errors dump listing at the end, before last folder sizes or the statistics. Turned on by default. Use --noerrorsdump to avoid it.
• Enhancement: Added --maxlinelengthcmd that will be called upon when a line over --maxlinelength value is detected.
• Enhancement: Exchange maxlinelength issue fixed with --maxlinelengthcmd 'reformime -r7' on Linux. Install maildrop package to get reformime command
• Enhancement: Added --testslive. Running imapsync --testslive performs a live test on real accounts (I own).
• Enhancement: Added --pipemess in order to pass all message to an external filter tool like reformime -r7.
• Enhancement: Added xoauth2 support, available on Unix with underlying openssl. Thanks to Joaquin Lopez.
• Enhancement: Added --skipmess regex to skip messages matching a regex. Example --skipmess 'm/[\x80-ff]/' to slip messages with non-7bit characters ( ie a byte somewhere begins with 1 )
• Usability: Better output of folders excluded by --exclude and folders included by --include
• Usability: Added inline help in many places. What you can do with switches to change imapsync behaviour, contextually. Learn imapsync the easy way.
• Bug fix: imapsync --tests should work under any circumstance now, any Unix, Windows, exe or bin or script. 561 non-regression tests
• Bug fix: Options --folderfirst and --folderlast generated an error when their value folder does not exist. Existence is checked.
• Bug fix: Option --disarmreadreceipts used to change Disposition-Notification-To in the body when not available in the header. Now never changes the body in all cases.
• Bug fix: Change default useheader values. Now it is really like --useheader "Message-Id" --useheader "Received". HMailServer replies two lines with --useheader "Message-Id" --useheader "Message-ID" in older imapsync releases.
• Bug fix: in imap_utf7_decode() + character must not be escaped. Was a bug with Cyrillic characters
• Refactoring: Started to split the FAQ file in several parts in FAQ.d/*
• Refactoring: Split the INSTALL file in several parts in INSTALL.d/*

• 1.607 (2014/11/14) Folders shown also in utf8.
• Caveat to upgrade on Unix!: New Unicode::String Perl module dependency, for utf8 output. See the INSTALL file.
• Enhancement: Added --skipemptyfolders to skip syncing empty host1 folders. They are not created on host2.
• Enhancement: Windows exe now uses IO::SSL 2.002 instead of 1.98 (is it really an enhancement?).
• Usability: Remove /x from regexes applied with --include and --exclude. Blanks no longer have to be explicit with \ or [ ].
• Usability: Added utf8 output for folder names, protocol native utf7imap special characters are not user friendly readable
• Usability: Moved foldersizes output after folders lists output.
• Usability: Added total size transferred after each message copied.
• Usability: After ETA, added number of total to be synced, as a denominator dddd after each message copied, like nnn/dddd where nnn is the number of messages copied. Before there was only nnn without /dddd.
• Bug fix: Cache fix on Windows. File paths with brackets [] are special, they have to be escaped with glob. For example, [Gmail] must be written [[]Gmail[]].
• Bug fix: Added several checks if IsUnconnected. Goal avoid imap commands while disconnected. Reconnexion would be better anyway.

• 1.597 (2014/10/06) Small things
• Enhancement: Added --nomixfolders to avoid merging folders that are considered different on host1 but the same on destination host2 because of case sensitivities and insensitivities.
• Bug fix: Fixed "imapsync doesn't see created folders in the listing of folder sizes at the end". (second time this bug shows up)

• 1.592 (2014/05/22) Logging by default! (an internal feature now)
• Caveat to upgrade: New IO::Tee Perl module dependency, for logging. See the INSTALL file.
• Windows upgrade: Win32 binary is now distributed in a zip file with two batches scripts ready to use it.
• Enhancement: Added logging by default in subdirectory ./LOG_imapsync/
• Enhancement: Added option --log turned on by default. Use --nolog to disable logging.
• Enhancement: Added option --logfile. Default is like ./LOG_imapsync/YYYY_MM_DD_HH_MM_SS_user2.txt where YYYY_MM_DD_HH_MM_SS is the launching date and time like 2014_05_20_23_45_10
• Enhancement: Added option --disarmreadreceipts to fix resending read receipts on host2. A must with Exchange as host2.
• Enhancement: Added FTGate support (no EXAMINE imap command used anymore).
• Usability: Updated the code to easily switch on --noreleasecheck by default but allowing --releasecheck (some users want this feature).
• Usability: Warn syncing messges will not be simulated when --dry mode is on and folder on host2 does not exist yet.
• Bug fix: Make TLS works with last IO::Socket::SSL default value (Added SSL_verify_mode => 'SSL_VERIFY_NONE') in --tls mode.
• Bug fix: Fixed bug when creating folders on host2 with --sep2=\\ (Lotus Notes).
• Bug fix: Try to get folder size when a folder on host2 has the same name but with a different case like OLD_MAIL -> old_mail. This avoids "does not exist" output.

• 1.584 (2014/02/05)
• Enhancement: Added --minmaxlinelength to select messages with long lines only. It helps to diagnostic Echange error on messages with lines longer than 9000 characters
• Enhancement: Added --debugmaxlinelength
• Bug fix: --ssl1 --tls2 was buggy because of default SSL_VERIFY_PEER. "Can not go to tls encryption on [localhost]:Unable to start TLS: Cannot determine peer hostname for verification..."
• Bug fix: Check if going to tls is ok, exit otherwise with explicit error message. Thanks to Dennis Schridde for reporting this ugly bug that deserves this CVE-2014-2014.
• Debug: Print separator given by NAMESPACE even when --sep1 oe --sep2 is used.
• Debug: Prints prefix given by NAMESPACE even when --prefix1 or --prefix2 is used. It is a preparation for advising not to use --prefix unless suggested.

Good changes made before, listed because they can help:

• Enhancement: Added --skipcrossduplicates to avoid copying messages that are already copied in another folder, good from Gmail to X.
• Enhancement: Added --debugcrossduplicates to print which messages (UIDs) are skipped with --skipcrossduplicates (and in what other folders they are).
• Enhancement: Added --folderfirst string: sync this folder first. --folderfirst "Work" --folderfirst "Friends"
• Enhancement: Added --folderlast string: sync this folder last. --folderlast "[Gmail]/All Mail" etc.
• Enhancement: Added --maxmessagespersecond to limit messages tranfer rate and fix Office365 throttle limitation (use --maxmessagespersecond 4).
• Enhancement: Added --maxbytespersecond to limit byte transfer rate.
• Enhancement: Added --debugmemory option that prints memory consumption after each message is copied.
• Enhancement: Added --authmech EXTERNAL.
• Enhancement: Adapted behavior for allowing --maxdate --mindate with --noabletosearch. Use internat date instead of Date: header.
• Enhancement: QQMail IMAP4Server success (with --noabletosearch option)
• Usability: Apply same treatment to --delete2duplicates as --delete2. If --uidexpunge2 can be done, do it, else do --expunge2 (unless --nouidexpunge2 or --noexpunge2).
• Usability: Added --timeout in the --help output. Default timeout is 120 seconds now (was nothing by imapsync, 600 with underlying modules).
• Enhancement/Bug fix: Added --ssl1_SSL_version and --ssl2_SSL_version to force the SSL_version in case the default auto-negociation does not work. Example: --ssl1 --ssl1_SSL_version SSLv3
• Enhancement: Added option --messageidnodomain to fix Mailenable bug changing the domain part of Message-Id header, and avoid duplicates
• Enhancement: Added option --syncflagsaftercopy to fix Mailenable bug not getting the flags with the APPEND
• Enhancement: Added option --maxlinelength to skip messages whose max line length is over a number of bytes. Exchange 2013 and Office365 need --maxlinelength 1000 (which is a RFC2822 must) to avoid disconnections.
• Usability: Added --fixInboxINBOX, turned on by default, to map automatically Inbox INBOX folder names.
• Usability: IMAP server removing consecutive spaces on the header part doesn't generate duplicates (MailEnable does that)
• Usability: SSL_verify_mode 0 to avoid warning about Man-In-The-Middle.
• Enhancement: Added XOAUTH authentication. Thanks to Eduardo Bortoluzzi Junior.
• Enhancement: Added --search1 and --search2 to allow different searches on each host.
• Usability: Added --delete1 as an alias for --delete
• Speed: Option --nocheckmessageexists is activated by default since --checkmessageexists often slow down transfers too much.
• Usability: Option --foldersizesatend is on if --foldersizes is on. Off if --nofoldersizesatend
• Usability: Added host2 minus host1 statistic: number of messages and bytes. One difference at the start and one at the end. Useful to detect quickly a difference between host2 and host1. Need both --foldersizes and --foldersizesatend options (which are on by default).
• Usability: Added --foldersizesatend. It gets the folders sizes at the end of the whole transfer. Turned on by default (use --nofoldersizesatend to turn it off).
• Enhancement: Added option --delete2duplicates; it deletes messages in host2 that are duplicates. --delete2duplicates is on when --delete2 is set unless --nodelete2duplicates is set too.
• Enhancement: Added option --pidfilelocking; it aborts imapsync, when just launched, in case another imapsync may be running. Turned off by default
• Usability: Added option --showpasswords. it shows passwords on output instead of "MASKED". Turned off by default. Useful to restart a run by reading the log or to see what happen with special characters.
• Bug fix: Added option --fixslash2 to avoid the 'Invalid mailbox name' error. It acts when --sep1 is character / and --sep2 is not character / and host1 folder name contains --sep2 characters. Turned on by default (--nofixslash2 to unable it).
• Enhancement: Added option --noabletosearch to allow the listing of messages without the imap "SEARCH ALL" command. It's useful for playing with poor imap servers like Softalk 7.6.4 (8.6 is fine with SEARCH ALL).
• Enhancement: Added --noexpungeaftereach to speedup --delete --expunge from Gmail.
• Enhancement: Added option --nocheckmessageexists to speed up with Tobit imap server as host1.
• Enhancement: Added option --exitwhenover option to avoid locking when transfers exceed maximum limit. See for example this Gmail constraint
  • imapsync ... --exitwhenover 2500000000 # 2.5GB if host1 is Gmail
  • imapsync ... --exitwhenover 500000000 # 500MB if host2 is Gmail
• Usability: Default headers used to identify a message are like --useheader "Message-Id" --useheader "Received" (Exchange compatibility bu default).
• Enhancement: Added option --addheader to fix the issue "ignored (no header so we ignore this message)" (on Sent folder usually). Thanks to Ameir Abdeldayem
• Enhancement: Added --nofilterflags to check any STORE flags by ignoring PERMANENTFLAGS (Exchange tests)

This document last modified on Thursday September 15, 2022 ($Id: news.shtml,v 1.87 2022/09/15 08:42:40 gilles Exp gilles $)
Top of the page