#!/bin/cat
$Id: FAQ.Migration_Plan.txt,v 1.14 2024/11/17 22:31:58 gilles Exp gilles $

This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Migration_Plan.txt


=====================================================================
   Imapsync. Suggestions for a good, low impact on users, 
   well-executed email migration plan.
=====================================================================


=====================================================================
                  In a hurry?  Key to failure.
                  Preparation: Key to success.
=====================================================================

Preparation. 
What is preparation? 
Know your world, know your context. 

You don't know them? Go get them, or delegate. Anyway, you or the
person that will perform the migration will need several things to get
the work done.  What is the stuff to be known? Some is mandatory, some
is not.

===================
Mandatory knowledge
===================

What is the system where you operate the migration?

Is it a Browser, or Windows, or Linux, or MacOS, or an embedded system
in any of those, or Docker, or whatever?

The way you install and launch imapsync on your system depends on your
system, like any software tool.  To find how to install imapsync on
your system, see https://imapsync.lamiral.info/#install

Other mandatory knowledge, what are the triplet credentials of each
account, on both sides?

Okay. Subsidiary question, what is a triplet credentials composed of?

The triplet credentials is composed of
*  The IP address or the internet name of the IMAP server.
*  The user login name.
*  The password.

In some cases, depending on the IMAP server software tool, an admin
authentication is possible, allowing to use only one password for all
accounts. To go further on this way, see
https://imapsync.lamiral.info/FAQ.d/FAQ.Admin_Authentication.txt

In other cases, with OAUTH2 authentication, aka the "Modern"
authentication, the password is a token string, an access token string
or even a refresh token string which allows to get a new access token.

===================
Optional knowledge
==================


What are the names of the IMAP servers software tools on both sides.
Are they Dovecot?  Gmail?  Office365? Yahoo? Exchange? etc.

Why is it good to know which IMAP servers software tools you deal
with?

Knowing the IMAP servers software tools used on both sides allows you
to access and read the FAQ documentation about them, to glance over
the well known issues and their solutions for this particular IMAP
server. Imapsync usually doesn't care about any specific IMAP server.
As long as they talk IMAP, imapsync handles them; but sometimes, it
needs a specific option to perform well.


=====================================================================
 Two scenarios, one magic but rare, one most common but still ok.
=====================================================================

There are two main different scenarios to migrate IMAP mailboxes.
Choosing which one fits your context depends on the response to the
following question:

Will the imap software tools used by the users use the same
credentials triplet for both imap servers, the old server host1 and
the new server host2?

The credentials triplet is hostname/username/password.

If the answer is yes, that clients' email tools use the same triplet
credentials, then it is possible to perform a migration without
changing anything on the users side. This may be a very time-saving or
energy-saving option since you won't have to touch the email client
tools, they will handle the change by themselves and re-synchronize
the entire mailbox at the time the user open his old/new mailbox,
after the migration, or even during the migration.  But it's a rare
condition so I'll describe this scenario later in this document.


=====================================================================
Classical scenario, credentials triplets are different on both sides
=====================================================================

 * Do not change the MX name first. Wait that the new mailboxes exist,
   and that they are fully synchronized from the old/current ones.  If
   you change the MX name first then either the users won't have the
   new messages in their old/current mailboxes, either they won't have
   their old messages in their new/current mailboxes.  Surprise and
   frustration on the menu for them, and reputation of incompetence
   for you.

 * If you can, decrease the TTL of the MX to 5 minutes (or even less).
   See the document https://imapsync.lamiral.info/FAQ.d/FAQ.TTL.txt to
   understand why it is an advantage.  If you can't decrease the TTL,
   the migration will span a little more, more than the current TTL,
   maybe 24 hours or 4 days but that's ok, the situation is not that
   bad with imapsync, which can handle it when it is played carefully.

 * Create the new mailboxes on the destination server host2.  If the
   users are already playing with the new mailboxes on host2, don't
   follow this scenario, but continue reading anyway, it will help you
   to mitigate your case.

 * Pre-synchronize all the mailboxes from the old server host1 to the
   new server host2. If an imap server name is going to change its IP
   address, from the old one to the new one, then don't use this name,
   use a name that will always match the same imap servers, or use
   their fixed IP addresses. Why? A account synced from itself won't
   do much transfer.
   
   Pre-synchronizations can usefully be done with the --delete2 option
   to get an exact synchronization. But never use the option --delete2
   once the users have started to play with their new account on
   host2, otherwise their play will be lost on the next
   synchronization. Don't use --delete2 either once the MX is changed
   since INBOX will start to receive new messages that are not on
   host1 and then removing them with --delete2 is not a good idea
   either.

 * Decide a migration day/hour.

 * Repeat the pre-synchronizations (with the --delete2 options) daily
   until the migration hour. This repeated process will show how long
   should take the last synchronization.

 * At the migration hour, cut access to the users to the old server
   host1, if you can. Or tell them to not use it anymore.

 * Do the last pre-synchronization exactly like the previous ones.

 * Change the MX, the new messages should start to arrive in the new
   imap server host2 after the TTL time, 5 minutes, (or days...).

 * Wait for the TTL value, aka 5 minutes (or days...). Now, new
   messages should not arrive at the old server host1.

 * Tell the users that the old imap server host1 is down and no longer
   available.

 * Do a post-synchronization. A post-synchronization is a run with the
   following options: --folder INBOX --delete1 --maxage 1

   This post-synchronization will copy the messages arrived in the
   last day (--maxage 1) in the folder INBOX (--folder INBOX) on the
   source account, to the destination account. It will also delete
   them on host1 (--delete1). Be careful, it's --delete1, it's not
   --delete2.  Remember, do not use the option --delete2 in a post
   synchronization, as users won't appreciate seeing their newly
   arrived messages disappear because of you.

 * Give access to new accounts to the users with their new credential
   triplet hostname/username/password.  If the way to contact users is
   by email then you should give them the new credentials and the
   timeline of the process long before shutting down the old
   server. Otherwise, you'll be in the "cut the branch users are
   sitting on" case.

 * Migration done. Congratulations! Champagne!

 * In case there are still messages arriving at the old imap server
   host1, you can perform more post-synchronizations, ie, runs every
   day with the options: --maxage 1 --delete1 --folder INBOX

 * Increase the TTL of the MX back to its previous value, usually 24
   hours, 86400 seconds. You don't want all your email system to break
   down completely when your DNS are not available temporarily,
   keeping DNS values in cache for a 24h is a savvy practice, unless
   you're in a email migration.

=====================================================================
Lucky scenario, credentials triplets are the same on both sides
=====================================================================
 
 * Decrease the TTL of the MX, as well as the imap hostname resolution,
   to 5 minutes (or even less). The document FAQ.TTL.txt explains why.

 * Create the new mailboxes on the destination server host2.
 
 * Pre-synchronize all the mailboxes from the old host1 to the new
   server host2, using different names than the ones used by the imap
   software clients (use their IP for example).  Presyncs have to be
   done with --delete2 but never use --delete2 once users have started
   playing with their new account on host2.
 
 * Decide a migration day/hour.

 * Repeat the pre-synchronizations (the runs with the --delete2
   options) daily until the migration hour. This repeated process will
   show how long should take the last sync.

 * At the migration hour, cut access to the users to the old server.
   You can do this by changing the imap host1 hostname to a non-imap 
   server for example, or by changing their password on host1.

 * Do the last run exactly like the pre-synchronizations.

 * Change also the MX resolution, the new messages should start 
   to arrive in the new imap server very soon.
 
 * Wait for the TTL value, aka 5 minutes. Now, new messages should 
   not arrive at the old server host1.
 
 * Do a post-synchronization. A post-synchronization is a run with the
   following options: --folder INBOX --delete1 --maxage 1

   This post-synchronization will copy the messages arrived in the
   last day (--maxage 1) in the folder INBOX (--folder INBOX) on the
   source account, to the destination account. It will also delete
   them on host1 (--delete1). It's --delete1, it's not --delete2.
   Remember, do not use the option --delete2 in a post
   synchronization, as users won't appreciate seeing their newly
   arrived messages disappear because of you.

 * Shut down the old imap server.

 * Change the user imap hostname resolution from the old IP of host1
   to the IP of the new imap server host2.

 * Migration done.

 * Increase the TTL of the MX back to its previous value, usually
   24 hours, 86400 seconds. You don't want all your email system
   to break down completely when your DNS are not available 
   temporarily, keeping dns values in cache for a 24h is a savvy
   practice.

 * A comment from Justus on this process:
 
   No change of the MUA configs worked like a charm. However, I still
   (needlessly) had some moments of fear and suspense when incoming
   test mails were invisible to the IMAP client connecting to the
   target server for more than an hour, then all 9000+ mails
   disappeared in the MUA, ... and then they reappeared and all was
   well. Turns out some mail clients just take loooooong to resync a
   handful of GB of mails (with no visible indication that they are
   doing anything) and show some weird behavior along the way. I
   should just have waited patiently but had no idea what was going
   on.


   Previously, I tried to alleviate Justus's fear by those words:

   Normally, no havoc should occur. The local state of any MUA agent
   is a slave state.

   As says the RFC about this:
   https://datatracker.ietf.org/doc/html/rfc4549 

   Synchronization Operations for Disconnected IMAP4 Clients
   ...  
   "All mailbox state or content information stored on the disconnected
   client should be viewed strictly as a cache of the state of the
   server.  The "master" state remains on the server, just as it would
   with an interactive IMAP4 client."


   Me: So any MUA should handle gracefully a server crash, replaced by
   a new one, or any imap server presenting a new UIDVALIDITY for a
   folder.  The MUA is a client. It syncs at each connection. If
   something changed, it takes it. If everything changed, from its
   point of view, UIDs, it retakes everything. That's its job.
   Exceptions are the flags, read the rfc4549 document.

 =======================================================================
 =======================================================================