News about next imapsync, currently distributed 2.290,
next and previous releases (back to menu)
Roadmap
If you are very interrested in those projects, don't hesitate to solicit me!
- Roadmap: Include a webserver in imapsync to have an online /X integrated.
- Roadmap: Integrate oauth2 token generation inside imapsync, ie, put oauth_imap in imapsync.
- Roadmap: Add the sync of multiple accounts inside imapsync itself instead of via an external shell script doing the loop from a csv file.
- Roadmap: Build an online tool, like /X, for multiple accounts at once. (NB: proof of concept already coded and working)
- Done: Build a docker image including everything, to have a complete online /X in a single command.
New features or bugfixes since previous releases:
imapsync 2.290
was written on Tuesday, 20-Aug-2024 10:39:22 UTC
See ChangeLog to read what has been done in details since 2001.
- 2.295
- Enhancement: In --extra can be passed --delete2folders --delete1emptyfolders --delete2duplicates --delete2 --delete1 --expunge1 --expunge2 --syncduplicates --skipcrossduplicates --debugcrossduplicates --useheader --errorsmax --folder --oauthaccesstoken1 --oautha
ccesstoken2 --ssl1 --ssl2 --tls1 --tls2 --compress1 --compress2 --debugimap --debugflags. I am not sure of this goodness...
- Usability: Added --idatefromheader for --gmail2 and --gmail1 --gmail2, unless --noidatefromheader
- Refactoring: Deglobalized @folderrec, now $mysync->{ folderrec }
- Docker context: Logging is on by default.
- 2.290
- Dependencies added: None
- Dependencies removed: None
- Usability: new values for Gmail, hoping they're ok.
- --gmail1 implies --maxbytespersecond 300_000 --maxbytesafter 3_000_000_000
- --gmail2 implies --maxbytespersecond 300_000 --maxbytesafter 3_000_000_000
- --gmail1 --gmail2 implies also --maxbytespersecond 300_000 --maxbytesafter 3_000_000_000
- Enhancement: Added the syntax --host1 imap.foo:1234/bar/truc equivalent to --host1 imap.foo --port1 1234 --folderrec bar/truc
- Enhancement: Added the syntax --host2 imap.foo:1234/bar/truc equivalent to --host2 imap.foo --port2 1234 --subfolder2 bar/truc
- Enhancement: Added tom@mklab patch to have --oauthrefreshcmd1 str : An optional script to call, to refresh the host1 access token before each authentication. Added also --oauthrefreshcmd2 str
- Enhancement: Allow --exitonload in normal context, ie, command line.
- Enhancement: Now the ssl check is done systematically on the port given by --port1 and --port2 and allow an automatic connection in ssl. This way, options --ssl1 and --ssl2 are no longer mandatory when using ports different than the usual ones, 143 and 993.
- Enhancement: Added the option --extra that allows to add options in the UI like in the command line.
- Enhancement: Now logging by default in Docker+CGI context. This allows the "tail -f" mechanism with /servimapsync when two same syncs are requested in parallel.
- Enhancement: Added an early debugsleep() call to debug timeout with /servimapsync
- Enhancement: Added in final stats the number of folders deleted on host2, if any.
- Enhancement: Added exit value 121 when most errors are SEARCH errors.
- Usability: Now --subfolder2 FOO implies --exclude FOO to avoid subfoldering recursion.
- Usability: Options --gmail1 --gmail2 now implies --folderlast for '[Gmail]/Sent Mail' '[Gmail]/Important' '[Gmail]/Starred' '[Gmail]/Drafts' '[Gmail]/Trash' '[Gmail]/Spam' '[Gmail]/Chats' '[Gmail]/All Mail', in this order.
- Usability: Logging is on with --log, off with --nolog, on in Docker+CGI context, off in Docker only context, on otherwise.
- Bug fix: Button "Abort!" now working in Docker+CGI context. Added a pid file in that context.
- Bug fix: Now detect errors in SEARCH command given by --search.
- Refactoring: 2151 unit tests.
- 2.264
- Dependencies added: Perl module Digest::SHA
- Dependencies removed: Perl module IO::Socket::INET6
- Enhancement: It is now possible to check login with one side. Example: imapsync --host1 test1.lamiral.info --user1 test1 --password1 secret1
- Enhancement: Office365 OAUTH2 modern authentication supported with an external script.
- Bug fix: Reread oauth2 files upon disconnections.
- Bug fix: Options --password1 and --password2 are no longer required when --oauthaccesstoken1 and --oauthaccesstoken2 are used, respectively. (release 2.261 was buggy on this, not 2.264)
- Bug fix: Options --user1 and --user2 are no longer required when --oauthdirect1 and --oauthdirect2 are used, respectively. (release 2.261 was buggy on this, not 2.264)
- Refactoring: 2065 unit tests.
- Refactoring: 200 functionnal tests.
- Security: Changed the default temporary directory. Now, the temporary directory taken from --tmpdir or $HOME/tmp or current directory, in order if defined (instead of /tmp on Linux) CVE-2023-34204
See https://nvd.nist.gov/vuln/detail/CVE-2023-34204
and also https://github.com/imapsync/imapsync/issues/399
I did not change the default tmpdir in the CGI context because /var/tmp/imapsync_hash is written only if it doesn't exist
and /var/tmp/imapsync_cgi/ is created under the supervision of an admin. And I am lazy to change all my hosts for now.
- Portability: removed "use IO::Socket::INET6 ;"
- CGI context: CGI context. Exit with a "503 Service Unavailable" when no disk space is available.
- CGI context: Now the logfile name can only be composed with characters a-zA-Z0-9_-
- 2.229
- General:
- Dependencies added: Proc::ProcessTable
- Dependencies removed: None
- Enhancement: Added a stat value: memory footprint times time spent during the sync. The unity is MiB * hour. The stat line is "Memory consumption at the end".
- Enhancement: Filtering buggy flags is now case insensitive. \Junk or \JUNK or \jUnK etc. (sub tests_filterbuggyflags)
- Enhancement: Added --memorystress to check memory crunching in normal run.
- Usability: Do not email final reports by default. Use --emailreport1 and --emailreport2 to add final emails reports in each INBOX.
- CGI context:
- Refuse to serve when the cpu load is greater than 1 per cpu. A load of 3.9 with 4 cores is ok, 4.1 is not. (sub heavy_load_reached_by_cpu)
- Refuse to serve when the RAM memory available for imapsync processes is less than 1 GB, presented by percentage on the first line like "64.80% used by processes". (sub heavy_load_percent_threshold)
- When refuse to serve, exits with value 69 EX_UNAVAILABLE
- Testing: 1992 unit tests
- 2.200
- Dependencies removed: None
- Dependencies added: None
- Enhancement: Append a final email report on each account at the end of the synchronization. Use --noemailreport1 and --noemailreport2 to avoid final emails reports in each INBOX.
- Enhancement: Documented why total sizes can differ even when the sync is perfect.
- Enhancement: Added SERVER_NAME SERVER_ADDR SERVER_ADMIN variables and values to the output.
- Enhancement: Added the local ip address for the imap connexions. It can help configuring firewalls to allow the imap source IP.
- Enhancement: Added "Posta inviata" for \Sent with --automap
- CGI context: Upped ERRORS_MAX_CGI from 20 to 500.
- CGI context: Added --var to get values from proximapsync for variables REMOTE_ADDR REMOTE_HOST HTTP_REFERER HTTP_USER_AGENT SERVER_SOFTWARE SERVER_PORT HTTP_COOKIE
- Usability: Added warning "parsing headers of folder ... It can take time for huge folders. Be patient."
- Bug fix: Do not turn on --delete2duplicates when --syncduplicates is on, unless --delete2duplicates is given.
- Bug fix: Redirect STDERR to STDOUT in all cases, --log or --nolog
- Bug fix: Do not allow --skipcrossduplicates and --usecache. Exit EX_USAGE 64
- Bug fix: Dates were not displayed under Windows because POSIX::strftime %e is not portable. Fixed several other places where dates were wrong on Windows.
- Bug fix: --tests --testslive was listing 0 folders.
- Refactoring: Deglobalized $debuglist $debugflags
- Refactoring: Added sync->{ permanentflags2 }
- 2.178
- Dependencies removed: None
- Dependencies added: Perl module Compress::Zlib
- Enhancement: Added final statistic line: "CPU time and %cpu : 13.27 sec 8.3 %cpu 2.1 %allcpus
- Enhancement: Removed the hard limit --maxsize 35_651_584 set by --gmail2 since CAPABILITY is used and good.
- Enhancement: Does not checknoabletosearch if --justfolders
- Enhancement: When creating a folder, check if it is selectable and consider ok if it is selectable.
- Enhancement: Options --gmail1 --gmail2 together sets --useheader X-Gmail-Received --useheader Message-Id
- Enhancement: Added compression in the imap connexions.
Compression is off by default no matter the capability announced.
Use --compress1 and --compress2 to turn compression on.
- Enhancement: Added --keepalive1 and --keepalive2 options.
Some firewalls and network gears like to timeout connections prematurely if the connection sits idle.
This option enables SO_KEEPALIVE on the host1 socket.
--keepalive1 is on by default since imapsync release 2.169
Use --nokeepalive1 to disable it. Same thing for --keepalive2 but for --host2. See https://metacpan.org/pod/Mail::IMAPClient#Keepalive
- Usability: Removed --fast option. It didn't do anything anyway, it was a fake option.
- Usability: Added advice inline advice on errors
- ERR_OVERQUOTA. "The destination mailbox is 100% full, get free space on it and then resume the sync."
- ERR_TRANSFER_EXCEEDED. "The maximum transfer size for a single sync is reached ( over 3 GB ). Relaunch the sync to sync more."
- ERR_APPEND_SIZE. "The destination server refuses too big messages. Use --truncmess option. Read https://imapsync.lamiral.info/FAQ.d/FAQ.Messages_Too_Big.txt"
- ERR_CONNECTION_FAILURE_HOST1. "Check that host1 on port1 is the right IMAP server to be contacted for your mailbox."
- ERR_CONNECTION_FAILURE_HOST2. "Check that host2 on port2 is the right IMAP server to be contacted for your mailbox."
- ERR_AUTHENTICATION_FAILURE_USER1. "Check the credentials for user1."
- ERR_AUTHENTICATION_FAILURE_USER2. "Check the credentials for user2."
- Usability: Bytes sizes human readable are now presented in KB MB GB TB PB (1000 base) instead of KiB MiB GiB TiB PiB (1024 base)
- Usability: Added README help for --truncmess
- CGI context: Experimental loaddelay forced to 0 (no delay at all). So far a huge load is rarely a problem.
- CGI context: Bugfix. Abort didn't set charset=UTF-8; same thing for heavy load 503 Service Unavailable
- Bug fix: Made reconnect mechanism works with --oauthdirect1 --oauthdirect2 --oauthaccesstoken1 --oauthaccesstoken2
- Bug fix: Upped the maximum number a pid can be. From 99999 to 2^22.
See what-is-the-maximum-value-of-the-process-id
- Bug fix: Load average on Mac was sometimes buggy. locale float 3,14 instead of 3.14
- Bug fix: errors_incr() exited with CATCH_ALL
- Bug fix: Memory consumption on Mac was not relevant. Use RSS instead of VSZ, on Mac only. Linux stays with VSZ.
- Docker context: Current working directory changed from /var/tmp to /var/tmp/uid_$EFFECTIVE_USER_ID
- 1.977
- Dependencies removed: None.
- Dependencies added: Perl modules Encode::IMAPUTF7
- Windows: imapsync.exe is now 64 bits (it was a 32bit binary before).
If you use an old 32bit PC, use imapsync_32bit.exe instead.
- Enhancement: Folders sizes are now presented per folder for host1 and host2,
as well as the differrences between them.
Comparing folders is now easier
- Enhancement: cgidir is . when used by Net::Server::HTTP
- Enhancement: Added context in --releasecheck user_agent. CGI or Docker or Standard.
- Usability: Added ETA at the beginning of a folder process (it is also to improve the /X progress bar accuracy).
- Usability: Added the number of errors and the max number of errors allowed at the final line.
Example: "Exiting with return value 0 (EX_OK: successful termination) 0/50 nb_errors/max_errors"
- Bug fix: Folders names with + characters must keep their + as they are when prensenting them in uft8 (in folders listings [] = [], right part)
- Bug fix: PIDs can be negative under Windows.
- Bug fix: --resynclabels was not on with --gmail1 --gmail2
- Bug fix: Count a fatal error as a error before presenting the count errors when exiting.
- Bug fix: Docker context. Killing myself in docker context doesn't seem to work well,
as well as sending signals several times.
So just install catch_exit with INT (ctr-c on terminals)
and exits if still alive after killing myself.
- Bug fix: Option --justconnect did not show the line "IP address:"
- Bug fix: Reviewed all exit_clean to include a final CR on the error message.
- Bug fix: Added SSL_cipher_list => 'DEFAULT:!DH' to tests with ssl.
- Bug fix: When using --subfolder2 SUB and --delete2folders, do not delete the folder SUB (even it is not on host1).
- Bug fix: CGI context. charset was charset=ISO-8859-1 in header (CGI.pm default). Now "text/plain; charset=UTF-8"
- Bug fix:
- Refactoring: Moved main instructions into sub single_sync(). The goal is to prepare multiple different syncs in imapsync.
- Refactoring: Cgi context is now set on with env variables SERVER_SOFTWARE and SERVER_PORT (was only SERVER_SOFTWARE).
It is because Net::Server::HTTP does not set SERVER_SOFTWARE but SERVER_PORT.
It is a preparation to a homemade web server to serve on imapsync as a cgi and
get free from Apache or a complicated web server stuff configuration.
- 1.921 (2019/02/18)
- Enhancement: Script sync_parallel_unix.sh to parallelize massive syncs.
It uses the powerful GNU parallel command.
- Enhancement: Added option --synclabels to sync Gmail labels. It should speed up Gmail to Gmail syncs.
The option --synclabels applies only during a copy of a message for now. Resyncing Gmail labels will be coded later.
- Enhancement: Option --synclabels is activated with --gmail1 --gmail2 used together.
- Portability: The imapsync script Shebang
(first line of imapsync) uses env now, instead of direct perl path.
I'm not sure it's a good change that
#!/usr/bin/perl becomes #!/usr/bin/env perl
- Enhancement: Final stats:
- Added a counter of crossduplicates on host2 ( a crossduplicate is a message found accross different folders )
- Added counter of "Messages found in host1 not in host2"
- Added counter of "Messages found in host2 not in host1"
- Added stat "Load end is"
- Added comment on final stat about messages in host1 not in host2
- Added comment on final stat about messages in host2 not in host1
- Added comment on final stat about unidentified messages and an --addheader suggestion
- Removed counters of "Total bytes duplicate host1" and "Total bytes duplicate host2"
- Removed counter of "Total bytes error"
- Enhancement: Added authmech X-MASTERAUTH. To be used for Kerio with --authmech1 X-MASTERAUTH or --authmech2 X-MASTERAUTH ( Patch from https://github.com/imapsync/imapsync/pull/53/files )
- Enhancement: Added --sigprint HUP (signal HUP by default) in order to allow some asynchronous prints (with, for example, a kill -HUP IMAPSYNC_PID) in CGI mode and avoid timeouts from browser
- Enhancement: --skipcrossduplicates activated with --gmail1 alone
- Enhancement: --skipcrossduplicates is NOT activated with --gmail2 alone or with --gmail1 --gmail2 together used.
- Enhancement: Speed. Deletions on host1 are done by batch during a resync.
- Enhancement: Added automatic --maxsize from "CAPABILITY APPENDLIMIT=xxxx" (Gmail)
- Enhancement: Added --folderlast '[Gmail]/All Mail' with --gmail1 --gmail2
- Enhancement: With --office1 option, folder "Files" is excluded because it's a "special" folder. Like --exclude "^Files$".
Unless --noexclude is used.
- Enhancement: With --office2 option, folder "Files" on host1 is renamed like --f1f2 "Files=Files_renamed_by_imapsync".
Unless --nof1f2 is used.
- Enhancement: Added --exclude '^Files$' with --office2 option.
- Enhancement: Added inline explanations for the options --office1 --office2 --exchange1 --exchange2 (what they set and how to avoid what they set).
- Enhancement: Added --noregexmess to avoid the regexmess setting with --office2 and --exchange2 (wrap long lines >= 10500 char).
- Enhancement: Added --noregexflag to avoid the regexflag set by --exchange2 (remove \Flagged flag).
- Enhancement: Added IO::Socket::IP and CGI in the list of the modules versions.
- Usability: Added PPID info where PID is printed. Unix only. Can be useful in some contextes, CGI, embeded, debugging etc.
- Bug fix: Now split the deletion of messages in several chunks (using --split1)
- Bug fix: --debugssl 0 was impossible and equivalent to --debugssl 1
- Bug fix: Removed --nosubscribed usage since it is not coded and useless anyway.
- Bug fix: Option --maxsleep override variable $MAX_SLEEP in gmail*() subroutines.
- Bug fix: Do not create host2 folder when no messages are found on this folder on host1
(See https://github.com/imapsync/imapsync/issues/95)
- Bug fix: The regex to remove the flag \Flagged was wrong. It worked because of another filter run after.
- Bug fix: The options --exchange1 and --exchange2 were not options! No one complained...
- Bug fix: A pid can be up to 99999 on FreeBSD
- Refactoring: Deglobarised some global variables
- Refactoring: Refactored sub appendlimit() to use Mail::IMAPClient::capability like the API.
No hack from internal $imap->{ APPENDLIMIT }
- Refactoring: Fixed some perl critics.
- Portability: Added freebsd support for loadavg and cpu_cores.
- Portability: Calls memory_consumption_of_pids_win32() within Cygwin environment (https://github.com/imapsync/imapsync/pull/22)
- Dependency removed: None.
- Dependency added: None.
- Roadmap: Proxy mode abandoned. Will be another lighter memory tool (since less memory consumption is the goal).
- 1.882 (2018/05/06) On the road to a proxy mode! (not done yet)
- Enhancement: Added option --nocheckfoldersexist ( Default is like --checkfoldersexist )
- Enhancement: Added option --noresyncflags : Do not resync flags for already transfered messages.
May be useful when a user has already started to play with its host2 account.
- Enhancement: Option --abort now creates a log too, in order to see if imapsync ended by signal comes from option --abort or something else.
- Enhancement: Added option --sigexit; For example --sigexit QUIT --sigexit TERM to exit when receiving QUIT or TERM signals.
Added option --sigreconnect. For example --sigreconnect INT to reconnect to both imap servers when receiving INT signal.
Added option --sigignore. For example --sigignore TERM to "ignore" the TERM signal. In fact print stats during the sync.
Default is like: --sigexit QUIT --sigexit TERM --sigreconnect INT
- Enhancement: Moved the hostname printing at first line. The goal is for clarity when in proxy mode.
Added free RAM info in the banner.
- Enhancement: When using sleeping behavior, a signal USR1 toggles sleep from 2s to 0s (back to 2s with on next USR1 signal). Unix only.
- Usability: Detect extra arguments. Meaning usually a quoting issue in the command line or that options are not well parsed.
- Usability: Added inline documentation about --releasecheck and --noreleasecheck
Default upstream is still --releasecheck. Just a line to change to have --noreleasecheck by default (as before)
- Usability: Added inline advice to resync after an end commanded by a TERM or QUIT signal.
- Bug fix: Fixed sub epoch() to allow that kind of silly INTERNALDATE 00-Jan-0000 00:00:00 +0000
(Thanks to IBM Domino...)
- Bug fix: --releasecheck activated was hardcoded in cgi context. Now it depends on the default behavior chosen.
Bug pending. Still have to fix --releasecheck on with --help
- Bug fix: Added \t\r\n characters to be replaced by character _ in sub filter_forbidden_characters()
- Bug fix: fixes to pass --tests under root user.
- CGI context: Added SERVER_SOFTWARE and SERVER_PORT info.
- CGI context: Added free RAM, total RAM and hostname in the status.
- CGI context: Reduced the loadavg info in the header status.
- Refactoring: Fixed some perlcritics level 4.
- Refactoring: Deglobalized several global variables.
- Dependency added: Perl module Regex::Common
- 1.836 (2017/09/05) More secure by default, ssl or tls activation!
- Enhancement: An Imapsync Docker image available!
- Usability: Now goes to SSL by default if port 993 is open. Use --nosslcheck to avoid that.
- Usability: Now goes to TLS by default if possible, ie, only if STARTTLS is in CAPABILITY. If you want only TLS and nothing else, use --tls1 --nossl1
- Usability: Now if you want a basic imap connection on port 143 with no default encryption behavior, ie, no ssl nor tls,
then use --nossl1 --notls1 for host1 and --nossl2 --notls2 for host2.
- Enhancement: Added --gmail1 and --gmail2 to simplify Gmail options setting.
It sets parameters suggested in the Gmail FAQ --ssl, --host, etc.
- Enhancement: Added --office1 and --office2 to simplify Office 365 options setting.
It sets parameters suggested in the Exchange/Office365 FAQ.
- Enhancement: Added --exchange1 and --exchange2 to simplify Exchange options setting.
It sets parameters suggested in the Exchange FAQ.
- Enhancement: Added --domino1 and --domino2 to simplify Domino options setting.
It sets parameters suggested in the Domino FAQ.
- Enhancement: Added --maxsleep in order to avoid timeouts with --maxbytespersecond and --maxmessagespersecond options.
By default imapsync will sleep 2 seconds maximum, like if the command line contained --maxsleep 2
- Enhancement: Added --maxbytesafter in order to start --maxbytespersecond limitation only after
--maxbytesafter amount of data transferred. Usefull for Gmail limits, for example,
in order to active a 50K/s limit rate only after 500 MB of data transfer, use
--maxbytesafter 500_000_000 --maxbytespersecond 50_000
- Enhancement: Added --testsunit in order to run any unit test individualy from the command line.
Several --testsunit are allowed. Example:
imapsync --testsunit tests_true --testsunit tests_always_fail
- Enhancement: Added password setting via environment variables IMAPSYNC_PASSWORD1 and IMAPSYNC_PASSWORD2
- Usability: No more useless and false warning "says it has NO CAPABILITY for AUTHENTICATE LOGIN"
- Usability: Options --delete1 and --delete are now aliases.
Option --delete1 is preferable over --delete (--delete is still supported).
- Usability: Now prints always permanentflags info.
It helps to understand most flag issues at first run, without --debugflags
- Usability: Now prints "could not append ( Subject:[$subject], Date:[$h1_date], Size:[$h1_size] )"
when append fails.
- Usability: Option --showpasswords now shows also passwords with --debugimap. Useful to debug quoting issues.
- Usability: --ipv4 is now synonym of --inet4 and --ipv6 is now synonim of --inet6
- Usability: Added --testslive6 to check pure ipv6 connectivity.
- Enhancement: Added --noabletosearch1 and --noabletosearch2;
Still support --noabletosearch, which turn on both --noabletosearch1 and --noabletosearch2
- Enhancement: Added --abort option to terminate a previous call still running.
In command line context --abort uses the pidfile to know what to abort.
In cgi context, ie online, exact same credentials are needed in order to really abort the other sync.
- Enhancement: Added milliseconds in the default logfile name since several runs is possible within one second in cgi context or on a powerful machine.
- Docker context: Added docker context in order to be run under the nobody user without permission issues.
- Docker context: Can run imapsync --tests under nobody user on Unix (or at least Linux).
- Bugfix: Fixed issue "SSL routines:ssl3_check_cert_and_algorithm:dh key too small" with
openssl-dh-key-too-small-error
SSL_cipher_list => 'DEFAULT:!DH'
- CGI context: Allow parameters passed by POST.
- CGI context: Abort, before doing anything, if the server load is already too heavy,
and invite to come later depending on the current load (1, 5, or 15 minutes later).
- Bug fix: Guessed prefix is the empty string even when there is no folders (which is a bad sign anyway, since INBOX should be listed).
- Bug fix: Option --skipmess could not work most of the time. I guess it was a mistake arrived by badly converting an "unless" to an "if". Perl critics with no tests added => caveat emptor!
- Refactoring: Removed Mail::IMAPClient overload definitions
- Dependency added: IO::Socket::SSL
- Dependency added: Sys::MemInfo
- Dependency added: Pod::Usage
- 1.727 (2016/08/19) https website and CGI on the way!
- Security: Thanks to free and easy letsencrypt,
Imapsync website is available in https:
https://imapsync.lamiral.info/
- Enhancement: Added option --delete1emptyfolders. It deletes empty folders on host1, INBOX excepted.
Useful with --delete since what remains on host1 is only what failed to be synced.
- Enhancement: Imapsync can be used as a CGI program. Still experimental. The web UI interface has to be written.
- Enhancement: XOAUTH2 authentication works with json files without extra tool openssl. Old pk12 files are still supported (still with openssl).
- Enhancement: Memory usage passed from 5x to 2x of the biggest message size.
Some users do send 700MB messages, imapsync needed 3.5 GB of RAM, now it crunches only 1.4 GB,
a more common RAM value available on small hosts.
- Enhancement: Added option --inet4 to force ipv4 in ssl and tls modes.
Useful with Gmail and Office365 when ipv6 is disable on the imapsync host but not ipv6 resolution.
- Enhancement: Reconnect to imap servers with a Ctrl-c. Exit with two Ctrl-c within one second or one Ctrl-\
- Enhancement: Copy is not done when one --pipemess command ends with an empty message generated.
- Enhancement: STDERR of --pipemess commands are reported on STDOUT and in the logfile (they were ignored before).
- Enhancement:
- Dependency: Added Perl module dependency of Readonly to transform hard coded magic numbers into names. Mandatory module.
- Dependency: Added Perl module dependency of Test::MockObject for some tests. Only a "require" for now.
- Security: In CGI mode, remove all content in the unsafe "eval"uated options
(--regextrans2 --regexflag --regexmess --skipmess --pipemess --delete2foldersonly --delete2foldersbutnot).
Consider these options not available in CGI mode.
- Bug fix: Fixed -1 wrong output in statistics for big accounts (%d => %s) in printf.
- Bug fix: Uncommented call to $test_builder->reset( ) Don't know why it was commented but it was bad for knowing tests go wrong.
- Bug fix: Check connected state in sub create_folder(). Avoid multiple useless failures.
- Refactoring: Wrote Perl package Imapsync::Getopt::Long to handle CGI requests like command line options.
Started as a copy of Luke Ross Getopt::Long::CGI
- Refactoring: Perlcritic cleanup 627 violations of CodeLayout::ProhibitTrailingWhitespace.
- Refactoring: Perlcritic cleanup 60 Double-sigil dereferences.
- Refactoring: Perlcritic cleanup 458 ValuesAndExpressions::ProhibitInterpolationOfLiterals. "Useless interpolation of literal string"
- Refactoring: Perlcritic cleanup 420 CodeLayout::ProhibitParensWithBuiltins. "Builtin function called with parentheses". (Severity: 1).
- Refactoring: Perlcritic cleanup, various small advices.
- Refactoring: Replaced print by myprint, printf by myprintf, sprinf by mysprintf.
Will ease --quiet mode and general output handling.
- 1.684 (2016/03/17) Small changes, bugfixes
- Bug fix: Imapsync.exe updated to Mail-IMAPClient-3.38 (3.37 had a bug with credentials containing special characters)
- Usability: Added inline help about "use --noexpungeaftereach to speed up" if --delete is used.
- Usability: Added --sslargs1 in documentation, usage option list.
- Bug fix: No more warning about "Use of uninitialized value $sockargs[7] in join or string at"
in ssl or tls mode. Was due to "'SSL_version' => undef".
- Bug fix: No more "Can't use an undefined value as a HASH reference at /usr/bin/imapsync line 1247."
- Bug fix: Added --expungeaftereach check in sub delete_message_on_host1().
- 1.678 (2016/01/21) Improved website! (I hope...)
- Enhancement: Added --sslargs1 and --sslargs2 to pass any ssl parameter for host1 or host2 connection.
Example: --sslargs1 SSL_verify_mode=1 --sslargs1 SSL_version=SSLv3
- Enhancement: Added --timeout1 int and --timeout2 int in seconds (--timeout int still available to set both with the same value)
- Enhancement: Added --debugssl int. Default is like --debugssl 1 (Only print out errors).
- Enhancement: Added several polls to know where I shall focus improvements.
- Usability: Added env_proxy call in sub xoauth2()
to read proxy settings from environment variable http_proxy
without using PERL_LWP_ENV_PROXY=1
- Usability: Wrote TUTORIAL_Unix.
- Usability: Check f1 folder exists when specified by --f1f2 and warns when it doesn't
- Usability: Added --automap --justfolders --dry in imapsync_example.sh and imapsync_example.bat
- Bug fix: Added require Encode::Byte to solve "The locale codeset (cp1252) isn't one that perl can decode" on Win32.
- Refactoring: Removed --allow3xx option.
- Refactoring: Continue to move to one global $sync-> in order to reduce number of parameters in routines.
- 1.670 (2015/12/03) Folders mapping made easy with --automap
- Enhancement: Added option --automap that guesses folders mapping,
for folders like "Sent", "Junk", "Drafts", "All", "Archive", "Flagged".
- IMAP Special-Use folders described in rfc6154
are supported and also well known names for Exchange and common clients.
- Automap is turned off by default,
to try it a safe method is to use --automap --justautomap, it will exit early
- Enhancement: Added --f1f2 str1=str2 : Force folder str1 to be synced to str2.
- Option --f1f2 overrides any --automap mapping and any --regextrans2 transformation.
- Example --f1f2 Spam=Junk to map Spam folder to Junk.
- Enhancement: Added --justautomap to exit after seeing what will happen with
--automap and --f1f2 options.
- Enhancement: Added IMAP4 QUOTA values (RFC 2087)
when they are available. A warning goes up when it's time to find space.
- Enhancement: Added IMAP4 ID extension.
- ID feature is on by default if supported by the IMAP servers,
use --noid to turn it off.
- ID Infos returned by the imap servers are shown.
- Infos sent to the servers are name=imapsync, version=`imapsync --version`, os=$OSNAME`,
vendor='Gilles LAMIRAL' and date=`rcs date of release`.
- For github or other distributors you can hack this part
by searching $imapsync_id_github
- Enhancement: Added --logdir path to change the default log directory.
Default is LOG_imapsync/
- Usability: Added folders counting in outputs.
- Usability: Better output in the login part
- Usability: Guess prefixes and separators instead of forcing the user to find them.
- Usability: Output, added Host1: or Host2: at the beginning of lines.
- Bug fix: SSL_VERIFY_NONE in --ssl and --tls modes
- Bug fix: Fixed xoauth2 calls. Now XOAUTH2 Works on Windows --authmech1 XOAUTH2
with openssl installed.
See FAQ.d/FAQ.XOAUTH2.txt.
- Bug fix: Fixed IO::Socket::SSL constants issue with latest releases
- Bug fix: Changed "Host1: checking all wanted folders exist" not efficient algorithm in order
to allow efficiently a 2.4 million folders account. Yes, some accounts have this.
- Refactoring: Started to use global $sync-> in order to reduce number of parameters in routines.
- Refactoring: Some perlcritic fixes.
- 1.644 (2015/07/17) Mac OS X standalone binary provided!
- Enhancement: Mac OS X standalone binary called imapsync_bin_Darwin
- Enhancement: Added option --subfolder2 SUB Move whole host1 folders hierarchy under folder SUB.
- Enhancement: Added --fetch_hash_set "1:*" to permit Mail2World success.
Need a patched Mail::IMAPClient 3.35 in sub fetch_hash()
- Usability: No folders sizes if --justfolders, unless really wanted.
- Usability: No warning about messages when --dry --justfolders are used together.
- Bug fix: Added NOOP in --dry mode during fake APPEND in order to avoid timeouts
- 1.636 (2015/03/31)
- Enhancement: Added errors dump listing at the end, before last folder sizes or the statistics. Turned on by default. Use --noerrorsdump to avoid it.
- Enhancement: Added --maxlinelengthcmd that will be called upon when a line over --maxlinelength value is detected.
- Enhancement: Exchange maxlinelength issue fixed with --maxlinelengthcmd 'reformime -r7' on Linux.
Install maildrop package to get reformime command
- Enhancement: Added --testslive. Running imapsync --testslive performs a live test on real accounts (I own).
- Enhancement: Added --pipemess in order to pass all message to an external filter tool like reformime -r7.
- Enhancement: Added xoauth2 support, available on Unix with underlying openssl. Thanks to Joaquin Lopez.
- Enhancement: Added --skipmess regex to skip messages matching a regex.
Example --skipmess 'm/[\x80-ff]/' to slip messages with non-7bit characters ( ie a byte somewhere begins with 1 )
- Usability: Better output of folders excluded by --exclude and folders included by --include
- Usability: Added inline help in many places. What you can do with switches to change imapsync behaviour, contextually. Learn imapsync the easy way.
- Bug fix: imapsync --tests should work under any circumstance now, any Unix, Windows, exe or bin or script. 561 non-regression tests
- Bug fix: Options --folderfirst and --folderlast generated an error when their value folder does not exist. Existence is checked.
- Bug fix: Option --disarmreadreceipts used to change Disposition-Notification-To in the body when not available in the header.
Now never changes the body in all cases.
- Bug fix: Change default useheader values. Now it is really like --useheader "Message-Id" --useheader "Received".
HMailServer replies two lines with --useheader "Message-Id" --useheader "Message-ID" in older imapsync releases.
- Bug fix: in imap_utf7_decode() + character must not be escaped. Was a bug with Cyrillic characters
- Refactoring: Started to split the FAQ file in several parts in FAQ.d/*
- Refactoring: Split the INSTALL file in several parts in INSTALL.d/*
- 1.607 (2014/11/14) Folders shown also in utf8.
- Caveat to upgrade on Unix!: New Unicode::String Perl module dependency, for utf8 output. See the INSTALL file.
- Enhancement: Added --skipemptyfolders to skip syncing empty host1 folders. They are not created on host2.
- Enhancement: Windows exe now uses IO::SSL 2.002 instead of 1.98 (is it really an enhancement?).
- Usability: Remove /x from regexes applied with --include and --exclude. Blanks no longer have to be explicit with \ or [ ].
- Usability: Added utf8 output for folder names, protocol native utf7imap special characters are not user friendly readable
- Usability: Moved foldersizes output after folders lists output.
- Usability: Added total size transferred after each message copied.
- Usability: After ETA, added number of total to be synced, as a denominator dddd after each message copied, like nnn/dddd
where nnn is the number of messages copied. Before there was only nnn without /dddd.
- Bug fix: Cache fix on Windows. File paths with brackets [] are special, they have to be escaped with glob.
For example, [Gmail] must be written [[]Gmail[]].
- Bug fix: Added several checks if IsUnconnected. Goal avoid imap commands while disconnected.
Reconnexion would be better anyway.
- 1.597 (2014/10/06) Small things
- Enhancement: Added --nomixfolders to avoid merging folders that are considered different on host1 but the same on destination host2
because of case sensitivities and insensitivities.
- Bug fix: Fixed "imapsync doesn't see created folders in the listing of folder sizes at the end". (second time this bug shows up)
- 1.592 (2014/05/22) Logging by default! (an internal feature now)
- Caveat to upgrade: New IO::Tee Perl module dependency, for logging. See the INSTALL file.
- Windows upgrade: Win32 binary is now distributed in a zip file with two batches scripts ready to use it.
- Enhancement: Added logging by default in subdirectory ./LOG_imapsync/
- Enhancement: Added option --log turned on by default. Use --nolog to disable logging.
- Enhancement: Added option --logfile. Default is like ./LOG_imapsync/YYYY_MM_DD_HH_MM_SS_user2.txt
where YYYY_MM_DD_HH_MM_SS is the launching date and time like 2014_05_20_23_45_10
- Enhancement: Added option --disarmreadreceipts to fix resending read receipts on host2. A must with Exchange as host2.
- Enhancement: Added FTGate support (no EXAMINE imap command used anymore).
- Usability: Updated the code to easily switch on --noreleasecheck by default but allowing --releasecheck (some users want this feature).
- Usability: Warn syncing messges will not be simulated when --dry mode is on and folder on host2 does not exist yet.
- Bug fix: Make TLS works with last IO::Socket::SSL default value (Added SSL_verify_mode => 'SSL_VERIFY_NONE') in --tls mode.
- Bug fix: Fixed bug when creating folders on host2 with --sep2=\\ (Lotus Notes).
- Bug fix: Try to get folder size when a folder on host2 has the same name but with a different case
like OLD_MAIL -> old_mail. This avoids "does not exist" output.
- 1.584 (2014/02/05)
- Enhancement: Added --minmaxlinelength to select messages with long lines only.
It helps to diagnostic Echange error on messages with lines longer than 9000 characters
- Enhancement: Added --debugmaxlinelength
- Bug fix: --ssl1 --tls2 was buggy because of default SSL_VERIFY_PEER.
"Can not go to tls encryption on [localhost]:Unable to start TLS: Cannot determine peer hostname for verification..."
- Bug fix: Check if going to tls is ok, exit otherwise with explicit error message.
Thanks to Dennis Schridde for reporting this ugly bug that deserves this CVE-2014-2014.
- Debug: Print separator given by NAMESPACE even when --sep1 oe --sep2 is used.
- Debug: Prints prefix given by NAMESPACE even when --prefix1 or --prefix2 is used.
It is a preparation for advising not to use --prefix unless suggested.
Good changes made before, listed because they can help:
- Enhancement: Added --skipcrossduplicates to avoid copying messages that are already copied in another folder, good from Gmail to X.
- Enhancement: Added --debugcrossduplicates to print which messages (UIDs) are skipped with --skipcrossduplicates (and in what other folders they are).
- Enhancement: Added --folderfirst string: sync this folder first. --folderfirst "Work" --folderfirst "Friends"
- Enhancement: Added --folderlast string: sync this folder last. --folderlast "[Gmail]/All Mail" etc.
- Enhancement: Added --maxmessagespersecond to limit messages tranfer rate and fix Office365 throttle limitation (use --maxmessagespersecond 4).
- Enhancement: Added --maxbytespersecond to limit byte transfer rate.
- Enhancement: Added --debugmemory option that prints memory consumption after each message is copied.
- Enhancement: Added --authmech EXTERNAL.
- Enhancement: Adapted behavior for allowing --maxdate --mindate with --noabletosearch.
Use internat date instead of Date: header.
- Enhancement: QQMail IMAP4Server success (with --noabletosearch option)
- Usability: Apply same treatment to --delete2duplicates as --delete2.
If --uidexpunge2 can be done, do it, else do --expunge2 (unless --nouidexpunge2 or --noexpunge2).
- Usability: Added --timeout in the --help output. Default timeout is 120 seconds now
(was nothing by imapsync, 600 with underlying modules).
- Enhancement/Bug fix: Added --ssl1_SSL_version and --ssl2_SSL_version to force the SSL_version
in case the default auto-negociation does not work. Example: --ssl1 --ssl1_SSL_version SSLv3
- Enhancement: Added option --messageidnodomain to fix Mailenable bug changing the domain part of Message-Id header, and avoid duplicates
- Enhancement: Added option --syncflagsaftercopy to fix Mailenable bug not getting the flags with the APPEND
- Enhancement: Added option --maxlinelength to skip messages whose max line length is over a number of bytes.
Exchange 2013 and Office365 need --maxlinelength 1000 (which is a RFC2822 must) to avoid disconnections.
- Usability: Added --fixInboxINBOX, turned on by default, to map automatically Inbox INBOX folder names.
- Usability: IMAP server removing consecutive spaces on the header part doesn't generate duplicates (MailEnable does that)
- Usability: SSL_verify_mode 0 to avoid warning about Man-In-The-Middle.
- Enhancement: Added XOAUTH authentication. Thanks to Eduardo Bortoluzzi Junior.
- Enhancement: Added --search1 and --search2 to allow different searches on each host.
- Usability: Added --delete1 as an alias for --delete
- Speed: Option --nocheckmessageexists is activated by default since --checkmessageexists often slow down transfers too much.
- Usability: Option --foldersizesatend is on if --foldersizes is on. Off if --nofoldersizesatend
- Usability: Added host2 minus host1 statistic: number of messages and bytes.
One difference at the start and one at the end.
Useful to detect quickly a difference between host2 and host1.
Need both --foldersizes and --foldersizesatend options (which are on by default).
- Usability: Added --foldersizesatend.
It gets the folders sizes at the end of the whole transfer.
Turned on by default (use --nofoldersizesatend to turn it off).
- Enhancement: Added option --delete2duplicates;
it deletes messages in host2 that are duplicates.
--delete2duplicates is on when --delete2 is set unless --nodelete2duplicates is set too.
- Enhancement: Added option --pidfilelocking;
it aborts imapsync, when just launched,
in case another imapsync may be running.
Turned off by default
- Usability: Added option --showpasswords.
it shows passwords on output instead of "MASKED".
Turned off by default.
Useful to restart a run by reading the log
or to see what happen with special characters.
- Bug fix: Added option --fixslash2
to avoid the 'Invalid mailbox name' error.
It acts when --sep1 is character / and --sep2 is not character /
and host1 folder name contains --sep2 characters.
Turned on by default (--nofixslash2 to unable it).
- Enhancement: Added option --noabletosearch
to allow the listing of messages without the imap "SEARCH ALL" command.
It's useful for playing with poor imap servers like Softalk 7.6.4
(8.6 is fine with SEARCH ALL).
- Enhancement: Added --noexpungeaftereach
to speedup --delete --expunge from Gmail.
- Enhancement: Added option --nocheckmessageexists to speed up with Tobit imap server as host1.
- Enhancement: Added option --exitwhenover option to avoid locking
when transfers exceed maximum limit.
See for example this Gmail constraint
- imapsync ... --exitwhenover 2500000000 # 2.5GB if host1 is Gmail
- imapsync ... --exitwhenover 500000000 # 500MB if host2 is Gmail
- Usability: Default headers used to identify a message are like --useheader "Message-Id" --useheader "Received"
(Exchange compatibility bu default).
- Enhancement: Added option --addheader to fix the issue
"ignored (no header so we ignore this message)" (on Sent folder usually).
Thanks to Ameir Abdeldayem
- Enhancement: Added --nofilterflags to check any STORE flags
by ignoring PERMANENTFLAGS (Exchange tests)
This document last modified on Thursday October 17, 2024
($Id: news.shtml,v 1.95 2024/10/17 12:25:33 gilles Exp gilles $)
Top of the page