#!/bin/cat $Id: FAQ.Migration_Plan.txt,v 1.14 2024/11/17 22:31:58 gilles Exp gilles $ This document is also available online at https://imapsync.lamiral.info/FAQ.d/ https://imapsync.lamiral.info/FAQ.d/FAQ.Migration_Plan.txt ===================================================================== Imapsync. Suggestions for a good, low impact on users, well-executed email migration plan. ===================================================================== ===================================================================== In a hurry? Key to failure. Preparation: Key to success. ===================================================================== Preparation. What is preparation? Know your world, know your context. You don't know them? Go get them, or delegate. Anyway, you or the person that will perform the migration will need several things to get the work done. What is the stuff to be known? Some is mandatory, some is not. =================== Mandatory knowledge =================== What is the system where you operate the migration? Is it a Browser, or Windows, or Linux, or MacOS, or an embedded system in any of those, or Docker, or whatever? The way you install and launch imapsync on your system depends on your system, like any software tool. To find how to install imapsync on your system, see https://imapsync.lamiral.info/#install Other mandatory knowledge, what are the triplet credentials of each account, on both sides? Okay. Subsidiary question, what is a triplet credentials composed of? The triplet credentials is composed of * The IP address or the internet name of the IMAP server. * The user login name. * The password. In some cases, depending on the IMAP server software tool, an admin authentication is possible, allowing to use only one password for all accounts. To go further on this way, see https://imapsync.lamiral.info/FAQ.d/FAQ.Admin_Authentication.txt In other cases, with OAUTH2 authentication, aka the "Modern" authentication, the password is a token string, an access token string or even a refresh token string which allows to get a new access token. =================== Optional knowledge ================== What are the names of the IMAP servers software tools on both sides. Are they Dovecot? Gmail? Office365? Yahoo? Exchange? etc. Why is it good to know which IMAP servers software tools you deal with? Knowing the IMAP servers software tools used on both sides allows you to access and read the FAQ documentation about them, to glance over the well known issues and their solutions for this particular IMAP server. Imapsync usually doesn't care about any specific IMAP server. As long as they talk IMAP, imapsync handles them; but sometimes, it needs a specific option to perform well. ===================================================================== Two scenarios, one magic but rare, one most common but still ok. ===================================================================== There are two main different scenarios to migrate IMAP mailboxes. Choosing which one fits your context depends on the response to the following question: Will the imap software tools used by the users use the same credentials triplet for both imap servers, the old server host1 and the new server host2? The credentials triplet is hostname/username/password. If the answer is yes, that clients' email tools use the same triplet credentials, then it is possible to perform a migration without changing anything on the users side. This may be a very time-saving or energy-saving option since you won't have to touch the email client tools, they will handle the change by themselves and re-synchronize the entire mailbox at the time the user open his old/new mailbox, after the migration, or even during the migration. But it's a rare condition so I'll describe this scenario later in this document. ===================================================================== Classical scenario, credentials triplets are different on both sides ===================================================================== * Do not change the MX name first. Wait that the new mailboxes exist, and that they are fully synchronized from the old/current ones. If you change the MX name first then either the users won't have the new messages in their old/current mailboxes, either they won't have their old messages in their new/current mailboxes. Surprise and frustration on the menu for them, and reputation of incompetence for you. * If you can, decrease the TTL of the MX to 5 minutes (or even less). See the document https://imapsync.lamiral.info/FAQ.d/FAQ.TTL.txt to understand why it is an advantage. If you can't decrease the TTL, the migration will span a little more, more than the current TTL, maybe 24 hours or 4 days but that's ok, the situation is not that bad with imapsync, which can handle it when it is played carefully. * Create the new mailboxes on the destination server host2. If the users are already playing with the new mailboxes on host2, don't follow this scenario, but continue reading anyway, it will help you to mitigate your case. * Pre-synchronize all the mailboxes from the old server host1 to the new server host2. If an imap server name is going to change its IP address, from the old one to the new one, then don't use this name, use a name that will always match the same imap servers, or use their fixed IP addresses. Why? A account synced from itself won't do much transfer. Pre-synchronizations can usefully be done with the --delete2 option to get an exact synchronization. But never use the option --delete2 once the users have started to play with their new account on host2, otherwise their play will be lost on the next synchronization. Don't use --delete2 either once the MX is changed since INBOX will start to receive new messages that are not on host1 and then removing them with --delete2 is not a good idea either. * Decide a migration day/hour. * Repeat the pre-synchronizations (with the --delete2 options) daily until the migration hour. This repeated process will show how long should take the last synchronization. * At the migration hour, cut access to the users to the old server host1, if you can. Or tell them to not use it anymore. * Do the last pre-synchronization exactly like the previous ones. * Change the MX, the new messages should start to arrive in the new imap server host2 after the TTL time, 5 minutes, (or days...). * Wait for the TTL value, aka 5 minutes (or days...). Now, new messages should not arrive at the old server host1. * Tell the users that the old imap server host1 is down and no longer available. * Do a post-synchronization. A post-synchronization is a run with the following options: --folder INBOX --delete1 --maxage 1 This post-synchronization will copy the messages arrived in the last day (--maxage 1) in the folder INBOX (--folder INBOX) on the source account, to the destination account. It will also delete them on host1 (--delete1). Be careful, it's --delete1, it's not --delete2. Remember, do not use the option --delete2 in a post synchronization, as users won't appreciate seeing their newly arrived messages disappear because of you. * Give access to new accounts to the users with their new credential triplet hostname/username/password. If the way to contact users is by email then you should give them the new credentials and the timeline of the process long before shutting down the old server. Otherwise, you'll be in the "cut the branch users are sitting on" case. * Migration done. Congratulations! Champagne! * In case there are still messages arriving at the old imap server host1, you can perform more post-synchronizations, ie, runs every day with the options: --maxage 1 --delete1 --folder INBOX * Increase the TTL of the MX back to its previous value, usually 24 hours, 86400 seconds. You don't want all your email system to break down completely when your DNS are not available temporarily, keeping DNS values in cache for a 24h is a savvy practice, unless you're in a email migration. ===================================================================== Lucky scenario, credentials triplets are the same on both sides ===================================================================== * Decrease the TTL of the MX, as well as the imap hostname resolution, to 5 minutes (or even less). The document FAQ.TTL.txt explains why. * Create the new mailboxes on the destination server host2. * Pre-synchronize all the mailboxes from the old host1 to the new server host2, using different names than the ones used by the imap software clients (use their IP for example). Presyncs have to be done with --delete2 but never use --delete2 once users have started playing with their new account on host2. * Decide a migration day/hour. * Repeat the pre-synchronizations (the runs with the --delete2 options) daily until the migration hour. This repeated process will show how long should take the last sync. * At the migration hour, cut access to the users to the old server. You can do this by changing the imap host1 hostname to a non-imap server for example, or by changing their password on host1. * Do the last run exactly like the pre-synchronizations. * Change also the MX resolution, the new messages should start to arrive in the new imap server very soon. * Wait for the TTL value, aka 5 minutes. Now, new messages should not arrive at the old server host1. * Do a post-synchronization. A post-synchronization is a run with the following options: --folder INBOX --delete1 --maxage 1 This post-synchronization will copy the messages arrived in the last day (--maxage 1) in the folder INBOX (--folder INBOX) on the source account, to the destination account. It will also delete them on host1 (--delete1). It's --delete1, it's not --delete2. Remember, do not use the option --delete2 in a post synchronization, as users won't appreciate seeing their newly arrived messages disappear because of you. * Shut down the old imap server. * Change the user imap hostname resolution from the old IP of host1 to the IP of the new imap server host2. * Migration done. * Increase the TTL of the MX back to its previous value, usually 24 hours, 86400 seconds. You don't want all your email system to break down completely when your DNS are not available temporarily, keeping dns values in cache for a 24h is a savvy practice. * A comment from Justus on this process: No change of the MUA configs worked like a charm. However, I still (needlessly) had some moments of fear and suspense when incoming test mails were invisible to the IMAP client connecting to the target server for more than an hour, then all 9000+ mails disappeared in the MUA, ... and then they reappeared and all was well. Turns out some mail clients just take loooooong to resync a handful of GB of mails (with no visible indication that they are doing anything) and show some weird behavior along the way. I should just have waited patiently but had no idea what was going on. Previously, I tried to alleviate Justus's fear by those words: Normally, no havoc should occur. The local state of any MUA agent is a slave state. As says the RFC about this: https://datatracker.ietf.org/doc/html/rfc4549 Synchronization Operations for Disconnected IMAP4 Clients ... "All mailbox state or content information stored on the disconnected client should be viewed strictly as a cache of the state of the server. The "master" state remains on the server, just as it would with an interactive IMAP4 client." Me: So any MUA should handle gracefully a server crash, replaced by a new one, or any imap server presenting a new UIDVALIDITY for a folder. The MUA is a client. It syncs at each connection. If something changed, it takes it. If everything changed, from its point of view, UIDs, it retakes everything. That's its job. Exceptions are the flags, read the rfc4549 document. ======================================================================= =======================================================================