#!/bin/cat $Id: FAQ.Messages_Selection.txt,v 1.18 2021/07/04 23:05:24 gilles Exp gilles $ This document is also available online at https://imapsync.lamiral.info/FAQ.d/ https://imapsync.lamiral.info/FAQ.d/FAQ.Messages_Selection.txt ======================================================================= Imapsync tips to select messages. ======================================================================= Questions answered in this FAQ are: Q. What messages imapsync syncs by default? Q. Is there a way we can specify a date range to sync emails? If yes, can you please share an example? Q. Is there a way we can specify an age to sync emails? If yes, can you please share some examples? Q. I want to sync messages based on their UID. Q. Can I migrate only mails with attachments? Q. How can I move messages marked \Deleted from all folders to a dedicated folder? Q. What are the selection criteria available with --search option? ======================================================================= Q. What messages imapsync syncs by default? R1. By default, Imapsync syncs all messages, except duplicates. If you want to sync duplicates, add the option --syncduplicates R2. You can alter the default "all messages selected" by using one or more of the following options: Extract from the README https://imapsync.lamiral.info/README OPTIONS/message selection --maxsize int : Skip messages larger (or equal) than int bytes --minsize int : Skip messages smaller (or equal) than int bytes --maxage int : Skip messages older than int days. final stats (skipped) don't count older messages see also --minage --minage int : Skip messages newer than int days. final stats (skipped) don't count newer messages You can do (+ zone are the messages selected): past|----maxage+++++++++++++++>now past|+++++++++++++++minage---->now past|----maxage+++++minage---->now (intersection) past|++++minage-----maxage++++>now (union) --search str : Selects only messages returned by this IMAP SEARCH command. Applied on both sides. For a complete set of what can be search see https://imapsync.lamiral.info/FAQ.d/FAQ.Messages_Selection.txt --search1 str : Same as --search but for selecting host1 messages only. --search2 str : Same as --search but for selecting host2 messages only. So --search CRIT equals --search1 CRIT --search2 CRIT --maxlinelength int : skip messages with a line length longer than int bytes. RFC 2822 says it must be no more than 1000 bytes but real life servers and email clients do more. --useheader str : Use this header to compare messages on both sides. Ex: Message-ID or Subject or Date. --useheader str and this one, etc. --usecache : Use cache to speed up next syncs. Not set by default. --nousecache : Do not use cache. Caveat: --useuid --nousecache creates duplicates on multiple runs. --useuid : Use UIDs instead of headers as a criterion to recognize messages. Option --usecache is then implied unless --nousecache is used. ======================================================================= Q. Is there a way we can specify a date range to sync emails? If yes, can you please share an example? R. Yes, a date range is possible with the --search option. imapsync ... --search "SENTSINCE 1-Jan-2010" or imapsync ... --search "SENTBEFORE 31-Dec-2010" or imapsync ... --search "SENTSINCE 1-Jan-2010 SENTBEFORE 31-Dec-2010" Months are specified like this: Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec ======================================================================= Q. Is there a way we can specify an age to sync emails? If yes, can you please share some examples? R. Yes, with the --maxage or the --minage option. Have in mind that the options --maxage and --minage can NOT be combined successfully with the --search command. If the --search option is present on the command line then --maxage and --minage are ignored. Example.1 Sync only messages less than 2 days old: imapsync ... --maxage 2 Example.2 Sync only messages more than 2 days old: imapsync ... --minage 2 Example.3 Sync only messages more than 30 days old and less than 365 days old: imapsync ... --minage 30 --maxage 365 Example.4 Sync only messages less than 30 days old or more than 365 days old: imapsync ... --maxage 30 --minage 365 Full explanation: --maxage : Skip messages older than days. final stats (skipped) don't count older messages see also --minage --minage : Skip messages newer than days. final stats (skipped) don't count newer messages You can do (+ are the messages selected): past|----maxage+++++++++++++++>now past|+++++++++++++++minage---->now past|----maxage+++++minage---->now (intersection) past|++++minage-----maxage++++>now (union) Caveat.1 By default, Option --maxage is implemented as a --search SENTSINCE RFC 3501 says: SENTSINCE Messages whose [RFC-2822] Date: header (disregarding time and timezone) is within or later than the specified date. Option --minage is implemented as a --search SENTBEFORE RFC 3501 says: SENTBEFORE Messages whose [RFC-2822] Date: header (disregarding time and timezone) is earlier than the specified date. If --noabletosearch is on then --minage and --maxage deal with the internal dates given by a FETCH imap command but not the Date: header. Internal date is the arrival date in the mailbox. Same remark for --noabletosearch1 and --noabletosearch2 but only for one side then. ======================================================================= Q. I want to sync messages based on their UID. R. First have in mind that UIDs are unique only per folder, so work this way only with one folder at a time, with --folder option. To show UIDs, there is the --debugLIST parameter. imapsync ... --debugLIST To sync only a part of all messages, selected by UIDs from 10000 to 11000: imapsync ... --search1 "UID 10000:11000" To sync from INBOX only 3 messages UIDs 20000 20002 20004: imapsync ... --search1 'OR OR UID 20000 UID 20002 UID 20004' --folder INBOX To sync all messages from INBOX except 3 messages UIDs 20000 20002 20004: imapsync ... --search1 'NOT OR OR UID 20000 UID 20002 UID 20004' --folder INBOX If you search n UIDs then you have to put n-1 OR in the search line, that's IMAP. ======================================================================= Q. Can I migrate only mails with attachments? R. Use: imapsync ... --search "HEADER Content-Disposition attachment" or imapsync ... --search "HEADER Content-Type multipart/mixed" or more generally: imapsync ... --search "OR HEADER Content-Disposition attachment HEADER Content-Type multipart/mixed" The last one doesn't work on all imap servers, not on Gmail at least (tested 2020/06/16) ======================================================================= Q. How can I move messages marked \Deleted from all folders to a dedicated folder? R. To move \Deleted messages from all folders to a specific folder, let's call it Trash, use: imapsync ... --search DELETED --regextrans2 "s/.*/Trash/" ======================================================================= Q. What are the selection criteria available with --search option? R. The list of search criteria are listed below, an excerpt from RFC3501. https://tools.ietf.org/html/rfc3501#section-6.4.4 ... The SEARCH command searches the mailbox for messages that match the given searching criteria. Searching criteria consist of one or more search keys. The untagged SEARCH response from the server contains a listing of message sequence numbers corresponding to those messages that match the searching criteria. When multiple keys are specified, the result is the intersection (AND function) of all the messages that match those keys. For example, the criteria DELETED FROM "SMITH" SINCE 1-Feb-1994 refers to all deleted messages from Smith that were placed in the mailbox since February 1, 1994. A search key can also be a parenthesized list of one or more search keys (e.g., for use with the OR and NOT keys). In all search keys that use strings, a message matches the key if the string is a substring of the field. The matching is case-insensitive. The defined search keys are as follows. Refer to the Formal Syntax section for the precise syntactic definitions of the arguments. Messages with message sequence numbers corresponding to the specified message sequence number set. ALL All messages in the mailbox; the default initial key for ANDing. ANSWERED Messages with the \Answered flag set. BCC Messages that contain the specified string in the envelope structure's BCC field. BEFORE Messages whose internal date (disregarding time and timezone) is earlier than the specified date. BODY Messages that contain the specified string in the body of the message. CC Messages that contain the specified string in the envelope structure's CC field. DELETED Messages with the \Deleted flag set. DRAFT Messages with the \Draft flag set. FLAGGED Messages with the \Flagged flag set. FROM Messages that contain the specified string in the envelope structure's FROM field. HEADER Messages that have a header with the specified field-name (as defined in [RFC-2822]) and that contains the specified string in the text of the header (what comes after the colon). If the string to search is zero-length, this matches all messages that have a header line with the specified field-name regardless of the contents. KEYWORD Messages with the specified keyword flag set. LARGER Messages with an [RFC-2822] size larger than the specified number of octets. NEW Messages that have the \Recent flag set but not the \Seen flag. This is functionally equivalent to "(RECENT UNSEEN)". NOT Messages that do not match the specified search key. OLD Messages that do not have the \Recent flag set. This is functionally equivalent to "NOT RECENT" (as opposed to "NOT NEW"). ON Messages whose internal date (disregarding time and timezone) is within the specified date. OR Messages that match either search key. RECENT Messages that have the \Recent flag set. SEEN Messages that have the \Seen flag set. SENTBEFORE Messages whose [RFC-2822] Date: header (disregarding time and timezone) is earlier than the specified date. SENTON Messages whose [RFC-2822] Date: header (disregarding time and timezone) is within the specified date. SENTSINCE Messages whose [RFC-2822] Date: header (disregarding time and timezone) is within or later than the specified date. SINCE Messages whose internal date (disregarding time and timezone) is within or later than the specified date. SMALLER Messages with an [RFC-2822] size smaller than the specified number of octets. SUBJECT Messages that contain the specified string in the envelope structure's SUBJECT field. TEXT Messages that contain the specified string in the header or body of the message. TO Messages that contain the specified string in the envelope structure's TO field. UID Messages with unique identifiers corresponding to the specified unique identifier set. Sequence set ranges are permitted. UNANSWERED Messages that do not have the \Answered flag set. UNDELETED Messages that do not have the \Deleted flag set. UNDRAFT Messages that do not have the \Draft flag set. UNFLAGGED Messages that do not have the \Flagged flag set. UNKEYWORD Messages that do not have the specified keyword flag set. UNSEEN Messages that do not have the \Seen flag set. ... search = "SEARCH" [SP "CHARSET" SP astring] 1*(SP search-key) ; CHARSET argument to MUST be registered with IANA ... search-key = "ALL" / "ANSWERED" / "BCC" SP astring / "BEFORE" SP date / "BODY" SP astring / "CC" SP astring / "DELETED" / "FLAGGED" / "FROM" SP astring / "KEYWORD" SP flag-keyword / "NEW" / "OLD" / "ON" SP date / "RECENT" / "SEEN" / "SINCE" SP date / "SUBJECT" SP astring / "TEXT" SP astring / "TO" SP astring / "UNANSWERED" / "UNDELETED" / "UNFLAGGED" / "UNKEYWORD" SP flag-keyword / "UNSEEN" / ; Above this line were in [IMAP2] "DRAFT" / "HEADER" SP header-fld-name SP astring / "LARGER" SP number / "NOT" SP search-key / "OR" SP search-key SP search-key / "SENTBEFORE" SP date / "SENTON" SP date / "SENTSINCE" SP date / "SMALLER" SP number / "UID" SP sequence-set / "UNDRAFT" / sequence-set / "(" search-key *(SP search-key) ")" seq-number = nz-number / "*" ; message sequence number (COPY, FETCH, STORE ; commands) or unique identifier (UID COPY, ; UID FETCH, UID STORE commands). ; * represents the largest number in use. In ; the case of message sequence numbers, it is ; the number of messages in a non-empty mailbox. ; In the case of unique identifiers, it is the ; unique identifier of the last message in the ; mailbox or, if the mailbox is empty, the ; mailbox's current UIDNEXT value. ; The server should respond with a tagged BAD ; response to a command that uses a message ; sequence number greater than the number of ; messages in the selected mailbox. This ; includes "*" if the selected mailbox is empty. seq-range = seq-number ":" seq-number ; two seq-number values and all values between ; these two regardless of order. ; Example: 2:4 and 4:2 are equivalent and indicate ; values 2, 3, and 4. ; Example: a unique identifier sequence range of ; 3291:* includes the UID of the last message in ; the mailbox, even if that value is less than 3291. sequence-set = (seq-number / seq-range) *("," sequence-set) ; set of seq-number values, regardless of order. ; Servers MAY coalesce overlaps and/or execute the ; sequence in any order. ; Example: a message sequence number set of ; 2,4:7,9,12:* for a mailbox with 15 messages is ; equivalent to 2,4,5,6,7,9,12,13,14,15 ; Example: a message sequence number set of *:4,5:7 ; for a mailbox with 10 messages is equivalent to ; 10,9,8,7,6,5,4,5,6,7 and MAY be reordered and ; overlap coalesced to be 4,5,6,7,8,9,10. See also the excellent "Introduction to IMAP" by Nick Babcock https://nbsoftsolutions.com/blog/introduction-to-imap See also "WITHIN Search Extension to the IMAP Protocol" https://tools.ietf.org/html/rfc5032 ... search-key =/ ( "OLDER" / "YOUNGER" ) SP nz-number ; search-key defined in RFC 3501 4. Example C: a1 SEARCH UNSEEN YOUNGER 259200 S: a1 * SEARCH 4 8 15 16 23 42 Search for all unseen messages within the past 3 days, or 259200 seconds, according to the server's current time. ======================================================================= =======================================================================