Fix email date for Mail.app
After moving my emails from one server to an other with a simple cp command I realized later that lots of my emails where showing in Mail.app with the same date (which appears to be the date I did the migration).
The thing is Mail.app is not using the Date header in the email but the email's creation date on the IMAP server. This date should be the date the server actually received the email. This prevent from receiving emails in the past or the future because the sender's computer date is wrong.
So what Mail.app does is pretty clever.
But when moving the files, the creation date changed for now. I didn't noticed it at first because once Mail.app retrieved an email, it keeps the information and don't check the server value anymore. But when I reinstalled my computer, after synchronizing all my emails again, I saw that the dates were wrong.
Here is an article I found linking to a nice script:
http://www.gabrielserafini.com/blog/2006/12/18/fixing-mailapps-imap-date-problem-mostlygeek/
The script basically read the Date header from the email and set it as creation date on the file. I ran the original script in my Maildir (containing ~120,000 emails) on my server and it worked quite well except for some emails that were ignored and some others with invalid dates:
date: invalid date `Apr 03 07 12:46'
./fix_imap_time_for_apple_mail_app.sh: 61: [: Illegal number:
1300235227.M261107P15760V0000000000000802I000A93BE_307.zed,S=159255:2,S from 201201302339 to
touch: invalid date format `'
So I did some changes to the script to make it work on all my emails:
#!/bin/bash
#
# Date : July 4th, 2005
# Author: Benson Wong
# tummytech@gmail.com
#
# This shell script corrects email messages where the file system
# date does not match the Date: header in the email.
#
# This will fix problems with mail clients like Apple's mail.app
# which uses the file system timestamp resulting in emails with the
# wrong file system timestamp to display the wrong received date
#
# This script has to be run by a user [root] with the
# necessary privileges to read/modify files in a user's Maildir.
#
function usage() {
if [ "$1" != "" ]; then
echo "$1"
fi
echo "Usage: $0 /path/to/user/Maildir"
exit 1
}
function email_date() {
local DATELINE=`grep -e "^Date: " "$1" | head -1`
local DELIVERYDATELINE=`grep -e "^Delivery-date: " "$1" | head -1`
if [ -n "$DELIVERYDATELINE" ]; then
local DATELINE="${DELIVERYDATELINE/elivery-d/}"
fi
# Fucked up date like Mon, 03 Nov 03 11:37:04 Romance Standard Time
local regex='^Date: ([A-Za-z]{3}, [0-9]{2} [A-Za-z]{3} [0-9]{2,4} [0-9]{1,2}:[0-9]{2}:[0-9]{2}) ([A-Za-z ]*)$'
if [[ $DATELINE =~ $regex ]]; then
EDATE=`date -d "${BASH_REMATCH[1]}" "+%Y%m%d%H%M"`
return 0
fi
# Missing "+" before timezone
local regex='^Date: ([A-Za-z]*, [0-9]* [A-Za-z]* [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2}) ([0-9]{4})$'
if [[ $DATELINE =~ $regex ]]; then
EDATE=`date -d "${BASH_REMATCH[1]} +${BASH_REMATCH[2]}" "+%Y%m%d%H%M"`
return 0
fi
# Remove day of the week
local regex='^Date: ([A-Za-z]*,) (.*)$'
if [[ $DATELINE =~ $regex ]]; then
EDATE=`date -d "${BASH_REMATCH[2]}" "+%Y%m%d%H%M"`
return 0
fi
local regex='^Date: (.*)$'
if [[ $DATELINE =~ $regex ]]; then
EDATE=`date -d "${BASH_REMATCH[1]}" "+%Y%m%d%H%M"`
return 0
fi
}
MDIR_PATH="$1"
[ $# -lt 1 ] && usage
[ ! -d "$MDIR_PATH" ] && usage "Error: $MDIR_PATH does not exist"
[ ! -r "$MDIR_PATH" ] && usage "Error: $MDIR_PATH is not readable"
[ ! -w "$MDIR_PATH" ] && usage "Error: $MDIR_PATH is not writable"
# set the internal field separator to the newline character
# instead of the default "".
# This is required for handling filenames and directories with spaces
IFS="
"
set -f
echo "start"
# Find all emails
for i in `find $MDIR_PATH -type f | egrep -v "(courierimap|maildirsize|maildirfolder)"`; do
email_date "$i"
if [ -z "$EDATE" ]; then
echo ""
echo "Unparsable date for" `basename $i`
continue
fi
FDATE=`ls -l --time-style=long-iso "$i" | awk '{print $6,$7}'`
# Reformat the date for touch.
ODATE=`date -d "$FDATE" "+%Y%m%d%H%M"`
if [ "$EDATE" -eq "$ODATE" ]; then
# Skip it if the times are correct.
echo -n "."
continue
fi
echo ""
echo `basename $i` "from $ODATE to $EDATE"
touch -c -t "$EDATE" "$i"
done
echo ""
echo "done"
My modified version is available on GitHub, I contacted the original author to know about the license but I'm still waiting for an answer.
When everything is done, remove your mail account from Mail.app (check that the corresponding folder in ~/Library/Mail/ is deleted, deleting ~/Library/Mail/V2/MailData/Envelope Index* might be a good idea too) then add your account back again.
Next time I have to migrate emails from one server to an other I may consider using imapsync:
http://mactip.blogspot.fr/2008/04/fixing-mailapps-imap-date-problem.html
Comments Add one by sending me an email.
Hi Laurent, Thanks for sharing this script. I'm having date issues as well. I followed your instructions but could not get dates fixed. What can I possibly be doing wrong. When I run your .sh script it yields the following but it does't show any "done" message. Does that mean it's not executing fully? Any help would be appreciated. Thanks!:
Hi Param,
You have to run this script on your server, not on your personal computer, and give the path to your user's
Maildiron your server (typically theMaildirdirectory is in your user'shomedirectory).HTH.
Trying fix an old problem of bad dates in Mail.app. They got screwed when updating my OSX Mail server some time ago and I think it was the change from Cyrus to Dovecot and Apple's migration tool was not 100%.
Anyway, I have a lot of messages showing as received 26 Nov 2020 so they show at the top of folders, above newly received messages which is irritating and so I've finally decided I want to fix it.
I ran a simple command on the server that adjusted the modification date of the bad files to something a few years ago (doesn't really matter when as long as it's in the past). I have since deleted that account from Mail.app and deleted the envelope index files on the client Mac. Added the account again, rebuilt the mail folders and synchronised that account and even copied those messages to an iCloud account to see if the date was correctly set. But all to no avail as the Received date resolutely stays at that future date, even though the date on the actual files is correct (well, close enough).
I suspect that Dovecot has cached lists of the messages in the folder and this is what is being used, but can you offer any suggestions as to how I can make dovecot update itself so the message dates are correct?
Hi Ken,
I don't know dovecot but from what I saw in their documentation (if you are using Maildir format: http://wiki2.dovecot.org/MailboxFormat/Maildir) that dovecot reads several times and that they are cached in index files: http://wiki2.dovecot.org/IndexFiles.
Maybe you should try to delete
dovecot.index.cache, looks like it contains the time data (and the server recreate the file automaticlaly).I originally commented on your script some time ago and never got around to finally sorting the problem, but now I really am going to get this done. If I remove Dovecot's index files and 'Rebuild' the Mailboxes in Mail.app, the file date is again used for the received date and all is well. I just need a working script.
Looking again at yours, I cannot understand your use of the 'date' command. The -d option is used to set the kernel's DST value, according to the man page. So I cannot see how it is being used in your script. The script on which yours is based uses the -j and -f options to reformat the data which I understand, but not what you are doing.
Would appreciate some explanation.
The
datecommand is quite OS specific. I believe you watched themanpage on your Mac instead of themanpage on Linux (where the script is executed).On Mac:
-d dst Set the kernel's value for daylight saving time. If dst is non-zero, future calls to gettimeofday(2) will return a non-zero for tz_dsttime.On Linux:
-d, --date=STRING display time described by STRING, not 'now'I’m aware that there are platform differences (that’s the great thing about standards - so many to choose from :-), but reading your ‘blog’ it didn’t indicate that the server was running on linux - or I missed it. Thanks for that. I’m running OS-X Server, so I’ll amend it to suit.
One more thing. You seem to infer that it handles a 2 digit year, but I cannot see how it actually accomplishes that and if it’s left as 2 digits, the script fails. I have my own function for doing that, but wasn’t sure if your script already did it, but I cannot see how?
Yeah, forgot to mention that the script was tested on Linux (the original script/article was saying it, I should add it too). Hope you can find the equivalent in Mac OS
datecommand.From what I remember the year MUST be on 4 digits, but yes I encountered some dates with only 2 characters for the year, they are handled on line 34, getting the year on 2 or 4 characters and having the
datecommand handling it itself (this was working as is when I used it, I don't know if the Mac command handles it also the same way).Ah, thanks. I wondered if the linux date command could cope with that. The OS-X version date formatting seems less flexible and I found I had to manually look for and and correct 2 digit years, then use date.
But right now, I can’t seem to get the =~ operator functioning. Should do, but not for even the simplest test for me at the moment. I must be missing something.
It still irritates that an Apple migration got it so wrong (even cp can preserve dates if specified) and now it’s so hard to fix because of the non standard use of a date string in the headers. It really shouldn’t be this hard.
Thanks for your help though.
I finally finished my script that caught all the errant emails on my OS X Server. As I said before, the problem was all the different ways that have been used to write the dates. Anyway, having run it and corrected all the file dates, I then deleted all Dovecot’s index files and also the index database files on a client Mac. Started Mail.app again on the client and as expected it had to ‘import’ everything which at least doesn’t require downloading all the email files all over again.
But it doesn’t fix the problem:-(
It would appear that it somehow still gets the old received date which must come from cached information. I’ve no idea where that is, but Rebuild the Mailbox is what is required. In this case, Mail goes back to the Server and re-downloads the email file(s) with the CORRECT date as that now comes from the file date as Dovecot has to retrieve that again since its index files were hosed.
Since I have so many Mailboxes to sort out, trying to go through them all would be a nightmare because Apple thoughtfully give us no way to Rebuild everything. So I will simply delete the account and add it again and wait while it downloads over 60,000 emails - AGAIN.
Anyway, thought I’d let you know what is and what is not required to fix this, even once the file dates have been corrected.
Automatically download all attachmentsturned off in the account settings).Forget that earlier request. I got Benson's original (simpler) script, inserted some perl -e, and hacked other commands for Solaris comatibility.
Problem now is that this doesn’t seem to fix things for me.
My IMAP server has files representing folders. Mailbox, Old, Technology, et c. These have hundreds of mail messages in them. “Touch"ing them to change the date just gives a different, wrong, sort date/time.
I need a tool that edits the hundreds of mail messages in each folder’s file on the server. (or a Mail.app plugin that updates each message on the server?)