Desgrange.net

I’m very lucky those days. 2 weeks ago my laptop hard drive died. Today I lost a drive in my RAID array.

Few month ago I moved my RAID array from SATA to USB but USB components seems to be of lower quality than SATA ones. So from time to time I see some “USB reset” when reading/writing a lot of data, fortunately it only stop transfers for a few seconds before resuming.

Today my server was fsck‘ing the array when a disk failed badly. It looks like the external drive’s USB controller crashed. The disk was not visible anymore from my server (I had to manually restart the external drive).

Since it’s a RAID 5 array, everything was still working fine, in degraded mode:

$ sudo mdadm --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Sat Jan 19 17:03:46 2008
     Raid Level : raid5
     Array Size : 1465151808 (1397.28 GiB 1500.32 GB)
  Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Wed Jul 22 22:16:04 2009
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 02918692:35547ed8:ccb6a325:e4cda885 (local to host seth)
         Events : 0.3774

    Number   Major   Minor   RaidDevice State
       0       8       65        0      active sync   /dev/sde1
       1       0        0        1      removed
       2       8       81        2      active sync   /dev/sdf1
       3       8       33        3      active sync   /dev/sdc1

Then adding back the missing drive:

$ sudo mdadm --manage /dev/md0 --add /dev/sdd1
mdadm: re-added /dev/sdd1
$ sudo mdadm --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Sat Jan 19 17:03:46 2008
     Raid Level : raid5
     Array Size : 1465151808 (1397.28 GiB 1500.32 GB)
  Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Wed Jul 22 22:19:23 2009
          State : clean, degraded, recovering
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

 Rebuild Status : 0% complete

           UUID : 02918692:35547ed8:ccb6a325:e4cda885 (local to host seth)
         Events : 0.3780

    Number   Major   Minor   RaidDevice State
       0       8       65        0      active sync   /dev/sde1
       4       8       49        1      spare rebuilding   /dev/sdd1
       2       8       81        2      active sync   /dev/sdf1
       3       8       33        3      active sync   /dev/sdc1

Now I can monitor the slow process of rebuilding the array:

$ cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdd1[4] sde1[0] sdc1[3] sdf1[2]
      1465151808 blocks level 5, 64k chunk, algorithm 2 [4/3] [U_UU]
      [>....................]  recovery =  2.2% (11184896/488383936) finish=661.5min speed=12019K/sec
      
unused devices: <none>

At least 11 hours to rebuild!!! So, for people as stupid as me: don’t use USB for a RAID array if you are not obliged to, it’s not very safe and it’s a bit slow.

Email protocols history

A long time ago there was several protocols to send a message from a computer to an other one. There was a protocol per network (the internet was not really born at that time). Those protocols were not compatible between each others. So for instance, if you were using FidoNet, you were not able to send emails to people using BITNET. Hopefully some people created some gateways to transfer emails from a network to an other one (but it looks like it was quite a nightmare). At some point in time, ARPANET and its email protocol became the standard and other protocols started vanishing. People were now able to communicate with each other easily.

Instant messaging

Instant messaging (IM) appeared much more recently (email started in the late 60s, late 80s for IM). Personal IM became very well known in 1996 with ICQ (I still have my ICQ account! (but nobody to talk to on it anymore)). Then several other protocols appeared:

• Jabber (1998, free and open protocol)
• QQ (1998, closed and proprietary)
• Yahoo! Messenger (1998, closed and proprietary)
• MSN Messenger (1999, closed and proprietary)
• …

Of course you can't chat with somebody using MSN Messenger if you use Yahoo! Messenger. ICQ was bought by AOL, AOL created AIM (AOL Instant Messenger) based on ICQ protocol. At some point people using ICQ were able to speak with people using AIM.

In 2004, XMPP, the protocol used by Jabber, became the official IETF instant messaging standard (IETF is the group defining the internet standards (like HTTP for the web, SMTP for emails, FTP for file transfer…)).

Jabber/XMPP

XMPP as been the standard for 5 years now, but still, most of my contacts are using MSN Messenger. Why? Because they also use a Microsoft Windows based computer and Microsoft MSN Messenger is the default instant messaging software installed on those computers.

XMPP allows creating gateways to other protocols (like MSN, Yahoo!, ICQ…). The situation looks a bit like the email status in the old days. But the email standard imposed itself as the killer application of ARPANET. So, what will be the thing that will wipe out all proprietary protocols and impose the XMPP open standard?

The problem with standards is that it take time to be developed. History as shown that new versions of a standard are not implemented by all software vendors quickly, nor they are deployed as fast as possible. So changes to the standard must not happen everyday and having it quite right at version 1.0 take time.

In the meantime, other protocols evolve faster because vendors have a captive market and a better control on how the software is distributed/used (and they don't have to wait for others to implement changes).

While XMPP was on the way to be standardized, other protocols got voice then video functionalities. I had a lot of hope in 1995 when Google released it's own IM software Google Talk. GTalk is based on XMPP and add some voice extension and video extension later.

Then AOL started an experiment to allow XMPP connections to its network, Yahoo announced that they were interested by XMPP too. The chat in Facebook uses XMPP (but the network is closed, you can only talk to Facebook users), several other community websites do the same.

Nowadays

Last week, the specifications for voice/video in XMPP were released. The biggest missing features making people stay with their proprietary IM is going to be old story soon. But I'm not sure it will be enough to see a big migration to XMPP.

Those last few years, XMPP interest increased a lot and nearly all IM vendors are now looking at XMPP… except Microsoft. It looks like you will soon have the choice between speaking to nearly everybody except MSN Messenger users or speaking only to MSN Messenger users.

Of course there are lots of softwares allowing you to use several accounts at the same time (so being connected to MSN, Yahoo! and Jabber at the same time). I also have several emails account. The difference is that from my professional email account I can send emails to everybody, same for my personal email. I'm choosing the email account I'm using depending on what my "role" is. If I want to send a message to a colleague, I will use my professional email address.

In IM, you can't do that, except if you open several accounts on each protocol you use. Having a professional and a personal account on MSN, on Yahoo!, on AIM, on Jabber… With all those protocols I currently have 9 accounts. And you know what? I like keeping things simple. For my email addresses I have started closing several accounts, keeping only the mandatory ones (my personal email address and the ones I have to use for my job). And I would really like doing the same for IM but you know what? Here I can't do what I want. Why? Because if I close my MSN account, I will lose contact with a lot of people.

I feel a bit like in jail. Worse, I feel like my friends are in jail too but they are saying "Where do you see a jail? There's only walls and fences".

Anyway, lots of people are using GMail now, and there is a chat embedded in GMail. Of course this chat uses Google Talk so it uses XMPP. Even if I don't really like GMail, I prefer having my friends using GMail/Google Talk than Hotmail/MSN Messenger (or whatever the name of those services are this week).

ejabberd

Since XMPP is an open protocol, anybody can implement it. There are several XMPP clients (Pidgin, Adium, Kopete, Trillian, iChat…) and there are also several servers.

In XMPP servers there is well-known one: ejabberd. This server is open source and written in Erlang. Ejabberd use the power of Erlang to be fault-tolerant, redundant, scalable, <add here any cool property a server should have>.

And since XMPP is a decentralized system, I can install my own server (as I did for my email server for instance).

Installing ejabberd on debian is as easy as usual:

$ sudo apt-get install ejabberd

To configure it, you just need to change the domain name to serve in /etc/ejabberd/ejabberd.cfg. If your domain name is example.org change the following:

%% Hostname
{hosts, ["example.org"]}.

And set the admin user:

%% Admin user
{acl, admin, {user, "admin_user_name", "example.org"}}.

Add a user with the following command:

$ sudo ejabberdctl register user_name example.org password

Restart the server. Done.

Of course there are a lot more parameters to change if you want to fine tune it. You may also need to create a SRV entry in your DNS if your server is not the one serving "example.org" (but "im.example.org" for example).

In this third post about configuring your own mail server on Debian (first post, second post), I'm going to talk a bit about SPAM and show you 2 ways of reducing it.

There are a lot of ways of fighting spam with various degrees of effectiveness, simplicity, standards respectful, etc.

For instance, there are some stupid systems like SpamCop based on user submissions. If a user tell that he received a spam from a given IP, then the IP is temporarily marked as a SPAM sender and will be blocked by mail servers using SpamCop black list. So, when your email address is hosted by an ISP used by a lot of people, of course there are emails reported as junk, and your ISP is black listed. Several mails I sent were blocked by SpamCop. Junk emails are a pain, but being blocked is way much worse. So please, don't use system generating to many false positives.

Greylisting

Greylisting consist of sending a temporary error when an email is received, store information about this email (server IP, sender, recipient) and wait for a second attempt after a given amount of time. If the email is received again, it's allowed.

Usually spam senders send emails to a huge list of addresses, so when they received an error, they just try with the next email address. This is very simple and effective against that kind of spam. The bad news is that emails are often delayed (the sender server may try again 5, 10, 30... minutes later). By definition emails are not instantaneous, but usually it's quite fast, so it can be a bit slower with this solution.

SPF

SPF stands for Sender Policy Framework. It's a way to check if the sender if allowed to send mails. For instance, a domain owner can specify in its DNS a TXT record containing IPs allowed to send emails from that domain.

I strongly suggest that you add this record if you have a mail server. For example, if you dig my domain name for TXT records you will get:

$ dig desgrange.net TXT

; <<>> DiG 9.5.0-P2 <<>> desgrange.net TXT
(…)
;; ANSWER SECTION:
desgrange.net.		600	IN	TXT	"v=spf1 a mx ~all"
(…)

The value v=spf1 a mx ~all means that A and MX IPs are allowed to send emails and that all other IPs should not.

tumgreyspf

In order to add greylisting and SPF to postfix, I decided to install tumgreyspf. The name is quite explicit, it does both greylisting and SPF.

Installation:

$ sudo apt-get install tumgreyspf

Tell postfix how to start tumgreyspf in /etc/postfix/master.cf by adding:

tumgreyspf unix -       n       n       -       -       spawn
  user=tumgreyspf argv=/usr/bin/tumgreyspf

Tell postfix when to ask tumgreyspf to perform a check, set smtpd_sender_restrictions in /etc/postfix/main.cf to something looking like that:

smtpd_sender_restrictions =
    permit_sasl_authenticated,
    reject_unauth_destination,
    check_policy_service unix:private/tumgreyspf

And now restart postfix (reload should be enough):

$ sudo /etc/init.d/postfix restart

Misc

After some tests, I found that tumgreyspf does not reject emails when SPF returns softfail but only fail and permerror.

If you configure your DNS like I did, which seems to be a quite common configuration, you may want to consider softfail has a reason to reject an email.

To change that, change line 122 of /usr/bin/tumgreyspf to the following (it's written in python, don't change indentation):

if spfResult == 'Fail' or spfResult == 'Permerror' or spfResult == 'Softfail':

Mail Stories posts

• Installing a mail server
• Installing a webmail
• Spam reduction (this post)

Here is a quick tutorial to build your own mail server using a Debian GNU/Linux.

Disclaimer

Configuring its own mail server can be fun, but remember that there are a lot of bad guys on the internet. I'm no expert on mail server configuration, don't forget that you must not allow relaying mails to domains you don't own and from people you haven't authenticated in order to avoid spam relay. Of course, keep your softwares (operating system, mail server, etc.) up to date in order to keep your computer safe from being cracked.

Goal

I have a domain name (desgrange.net as you can see) and I would like to use it for my mails. It's not that I don't trust service providers, but I would like to be in charge of the mail server configuration (so I would be able to tune it as I want).

For instance, I want to be able to send mails using SMTP over SSL and retrieve emails using IMAP over SSL.

For those tasks I chose:

• Debian as operating system (tested on Debian 5.0 and Debian testing)
• Postfix (2.5.5) as SMTP server
• Courier-imap (4.4.0) as IMAP server

In this tutorial I assume that I'm installing a mail server for domain example.com on a computer named mailserver. Allowed users are the standard unix users created on the mail server.

DNS

First make sure that your DNS has your MX record pointing to your mail server. See with your domain name provider for that part. I'm using OVH so I did it on their website (which sucks a bit for that).

Postfix

Postfix is a well known SMTP server. As a lot of mail servers, it has several hundreds parameters, but here, all of them have good default values, so we just need to change some of them to get it working like we want. Let's start with postfix installation:

$ sudo apt-get install postfix
-> General type of mail configuration: internet with smarthost
-> System mail name: example.com
-> SMTP relay host: <blank>

Now we need to install the authentication system saslauthd.

$ sudo apt-get install sasl2-bin libsasl2-modules

As told by the installation process, to enable saslauthd, edit /etc/default/saslauthd and set START=yes.

To tell postfix to use saslauthd, create a file /etc/postfix/sasl/smtpd.conf containing:

pwcheck_method: saslauthd
mech_list: PLAIN LOGIN

As postfix is running chrooted, to allow it to talk to saslauthd we need to move some files (or change the directory specified in OPTIONS in /etc/default/saslauthd):

$ sudo mkdir -p /var/spool/postfix/var/run/
$ sudo mv /var/run/saslauthd /var/spool/postfix/var/run/
$ sudo ln -s /var/spool/postfix/var/run/saslauthd/ /var/run/saslauthd
$ sudo adduser postfix sasl

Postfix is configured to run on the standard port 25 so other mail servers can transfer mails to you. This postfix process can't use SSL only because other mail servers may not support it. So we need to start an other postfix process running over SSL, so we can use this one when we want to send an email. To allow this process to start, just uncomment the submission lines in /etc/postfix/master.cf.

Check that /etc/mailname contains your domain name. It's used by postfix for some parameters (seems to be debian specific).

Most of postfix configuration is done in /etc/postfix/main.cf, you can edit this file directly or use the following commands:

1 $ sudo postconf -e 'smtpd_tls_security_level = may'
2 $ sudo postconf -e 'smtpd_tls_mandatory_ciphers = high'
3 $ sudo postconf -e 'smtpd_tls_mandatory_exclude_ciphers = aNULL'
4 $ sudo postconf -e 'smtpd_tls_mandatory_protocols = TLSv1'
5 $ sudo postconf -e 'mydestination = $myhostname localhost.$mydomain localhost $mydomain'
6 $ sudo postconf -e 'smtpd_recipient_restrictions = permit_sasl_authenticated,reject_unauth_destination'
7 $ sudo postconf -e 'relay_domains ='
8 $ sudo postconf -e 'home_mailbox = Maildir/'
9 $ sudo postconf -e 'mailbox_command ='

Line 1 tells postfix to use TLS on standard SMTP port (25) if remote server support it.
Line 2 restricts TLS ciphers to only "HIGH" ciphers.
Line 3 excludes aNULL cipher (no encryption).
Line 4 tells to use TLSv1 as protocol for the postfix over SSL process.
Line 5 list domain names for which the server will accept mails.
Line 6 tells postfix to send mails only from authenticated users or to domains postfix is in charge of.
Line 7 postfix must not relay mails to other domains.
Line 8 and 9 are for Courier which use a specific type of mailboxes.

You can have a look at the postfix configuration parameters documentation.

Courier-IMAP

Courier is a full mail server but here I'm going to use only Courier's IMAP server component. Installation:

$ sudo apt-get install courier-imap-ssl
-> Create directories for web-based administration?: no

By default IMAP and IMAP over SSL processes are started but I don't want the first one. To disable the standard IMAP process, set IMAPDSTART to NO in /etc/courier/imapd.

Some parameters have to be changed for the IMAP over SSL process, edit the file /etc/courier/imapd-ssl and set the following values:

IMAP_TLS_REQUIRED=1
TLS_PROTOCOL=SSL23
TLS_STARTTLS_PROTOCOL=SSL23
TLS_CIPHER_LIST=HIGH

For TLS_PROTOCOL and TLS_STARTTLS_PROTOCOL I use SSL23 because I'm using Apple Mail and it does not handle TLS very well. If you use a mail client supporting TLS, replace SSL23 with TLS1.

Finishing the installation

First we need to restart everything:

$ sudo /etc/init.d/saslauthd restart
$ sudo /etc/init.d/postfix restart
$ sudo /etc/init.d/courier-authdaemon restart
$ sudo /etc/init.d/courier-imap restart
$ sudo /etc/init.d/courier-imap-ssl restart

If you type sudo netstat -lntp you should see something like that:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:587             0.0.0.0:*               LISTEN      12718/master    
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      12718/master    
tcp6       0      0 :::993                  :::*                    LISTEN      12440/couriertcpd

There are two processes for postfix (master on ports 25 and 587 (SSL)) and one for courier on port 993.

Create your mailbox. You may want to initialize your own mailbox directory on the server:

$ maildirmake ~/Maildir

Certificates

I didn't say anything about SSL certificates, you need certificates for all your SSL connections, here some default ones are created, but when you connect with your mail client, it may complain about the certificate not being trustworthy.

It may prevent you to read/send mails (for instance I had to tell Apple Mail to trust the certificate in order to be able to send emails).

Trouble shooting

While configuring my mail server I ran in several problems, mainly due to Apple Mail, TLS and certificates. So, if you have any trouble, don't forget to have a look at log files in /var/log/, use netstat and ps to be sure that your servers are running and listening on the right ports.

You may even try to manually connect to the IMAP server with a command like this one:

openssl s_client -tls1 -connect mailserver:993 -state -debug

Have a look to the IMAP commands to check if it works.

Some of the errors I had:

• SASL authentication failure: cannot connect to saslauthd server: No such file or directory because I didn't had moved saslauthd files to /var/spool/postfix/var/run.
• imapd-ssl: Unexpected SSL connection shutdown because Apple Mail was not trusting the certificate.
• imapd-ssl: couriertls: connect: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number because Apple Mail does not support SSL3/TLS1 on IMAP.

Links

Some useful resources:

• Postfix Howtos and FAQs
• Postfix Documentation
• A tutorial on SMTP authentication over SSL
• A tutotial including a postfix/courier-imap installation

Mail Stories posts

• Installing a mail server (this post)
• Installing a webmail
• Spam reduction