cPanel 11 Prep: Converting from MBOX to MailDir Format

**Mike** · June 11th, 2007, 1:40 PM

Note: this article was created by Aric Pedersen
Author of the cPanel User Guide and Tutorial and Web Host Manager Administration Guide

A few people have asked me about the whole mbox vs. maildir thing and what this means for dedicated server owners.

Here's the deal:

Right now, all cPanel servers store mail in mbox format and use uwimap.

See:

http://email.about.com/cs/standards/a/mbox_format.htm
and
http://www.washington.edu/imap/

mbox format has been in use forever in Internet years.

Lots of e-mail clients can import and export mbox files.

It's not hard since mbox files are basically plain text 7-bit ASCII files.

It's worked well for a long time, but that doesn't mean we can't do better.

cPanel Inc. has decided to move all cPanel servers to the newer, faster and more efficient maildir (short for mail directory) format and Courier IMAP service.

See:

http://www.courier-mta.org/imap/
and
http://www.courier-mta.org/mbox-vs-maildir/

This mandatory conversion probably won't happen for a while yet (they've said they are shooting for cPanel 11, whenever that gets released).

However, you CAN switch now if you want and yes, cPanel works just fine with the new format. And yes, it IS noticeably faster, especially for servers with lots of e-mail accounts.

However, there are a few things to keep in mind before you decide to jump into the conversion now:

1) Neomail does NOT work with maildir. Neomail is a dead project and isn't being updated for the new format. Users will loose the ability to work in Neomail after the maildir conversion. cPanel Inc. DOES plan to replace neomail with another webmail program (perhaps even the cool roundcubemail), but has not done so yet. Squirrelmail and Horde both work fine with the new format. If your users CANNOT do without Neomail DO NOT convert yet. However, you SHOULD let them know it will be going away!

2) The conversion does take time and can cause high loads on your server during the conversion. For this and all the other obvious reasons, try to do the conversion on a weekend late at night when load and mail use is low on your server. The whole process can take 10 minutes to several hours. How long depends on how powerful your server is, how many e-mail accounts are set up on your server and how much mail is being stored on your server. You DON'T have to watch the entire conversion process, as it mostly does everything in the background, but it is a good idea for you to plan to be around while the process is going on, just in case something goes wrong.

3) The maildir conversion process has become fairly foolproof, but it IS possible to loose e-mail stored on the server if things go horribly wrong.

4) You CAN revert to mbox format if the conversion doesn't go well, but only if you choose to back up mail before beginning. See below for more on that.

5) If you use MailScanner you MUST run the following commands via root shell after the conversion process finishes or no mail will be scanned by MailScanner:

Code:

/scripts/buildeximconf
/scripts/mailscannerupdate
service exim restart

Don't forget!

You can ignore this step if you don't use MailScanner.

6) If your clients are having problems with e-mail now or if your server is having any other issues, don't start the conversion. Make sure your server is working properly FIRST and then convert. This will save you a lot of hassle in the long run!

7) If you have installed a third-party webmail client on your server, you will probably need to update the configuration after the conversion to take into account the new mail format. cPanel's own webmail clients will automatically be converted, so you don't need to worry about those.

Are we all clear on that? Personally, I'd rather handle the conversion at a time convenient to me rather than wake up one morning and discover my server automatically converted everything (and messed it up) while I was sleeping. But doing it now is entirely up to you!

Here's what you'll need for the mbox to maildir conversion:

1) root shell access on your VPS or dedicated server
2) The latest version of cPanel. You should update cPanel if it isn't the latest version. /scripts/upcp will take care of you.
3) Enough disk space to accommodate the backup of the e-mail mbox files on your server. If you have a second hard drive mounted as /backup, the conversion backup process will use that, otherwise a directory will be created on your existing hard drive. The mbox files are compressed, but the backup can take several hundred MB or more.

I also strongly suggest you warn your users in advance if you can that you will be switching so they know to try to stay away from their e-mail boxes during the conversion.

Ready? If so, log into your server via SSH as root and run:

/scripts/convert2maildir

You will see something like this:

Code:

# /scripts/convert2maildir
        ____                  _
    ___|  _ \ __ _ _ __   ___| |
   / __| |_) / _` | '_ \ / _ \ |
  | (__|  __/ (_| | | | |  __/ |
   \___|_|   \__,_|_| |_|\___|_|
  
   Mail Directory (maildir) Conversion System 


maildir is not active, this system is using mbox

1) Backup all mail folders on this server
2) Restore a mail backup
3) Start maildir conversion process
4) Request Technical Support
*) Exit
Enter Selection---]

Note that this system does not yet have maildir active and is using mbox.

Type the number 1 and hit Enter/Return. When you are asked if you want to overwrite any old mbox backups, type y and hit Enter/Return.

This will start the backup of the mbox format files.

As I said earlier, this process can take a while and will place serious load on your server. Be patient! It may look like it is stalled, but it's still working, trust me!

You will see the username of every user on the server and eventually the word "done." after the backup of that user is completed. Users with little stored mail will backup quickly. Those with thousands of message or lots of accounts will take longer.

When done, you will find yourself back at the maildir convert menu. Press 1 if you need to backup the files again. This willl overwrite the existing backups with new ones, or we can go on to #3.

If you choose #3, you will see a single line displayed and then be dumped back to the command line. This is normal. It will say something like:

Watch /var/cpanel/updatelogs/maildirconversion.1152997653 if you want to see the progress of the conversion.

It is optional, but I suggest you do just that to make sure everything goes well.

Code:

tail -f /var/cpanel/updatelogs/maildirconversion.1152997653

Of course change the path and filename to match what your server has told you.

This will show you the progress in realtime. Again, this can take quite a while.

What it will do is:

1) Check your server and update and install several files needed for Courier and the maildir format.
2) Configure Exim to use the maildir format.
3) Convert every mbox file for ever user on your server to the new maildir format.
4) Do some final Exim and cPanel configuration (and disable Neomail).
5) Check for some needed updates.

When the process is done you will see a note in the log file. Do not try to do ANYTHING with Exim or webmail during this process until it is finished. If your users try to access or send mail that is OK. They may receive an error during this process, but no e-mail will be lost (if all goes well).

When the process is done and only if you use MailScanner run the following:

Code:

/scripts/buildeximconf
/scripts/mailscannerupdate
service exim restart

But do not try to do that until the conversion is finished!

If something happens or you need to reboot your server during the conversion process or if there are any serious errors, don't worry. Run /scripts/convert2maildir again:

Code:

# /scripts/convert2maildir 
       ____                  _
    ___|  _ \ __ _ _ __   ___| |
   / __| |_) / _` | '_ \ / _ \ |
  | (__|  __/ (_| | | | |  __/ |
   \___|_|   \__,_|_| |_|\___|_|
  
   Mail Directory (maildir) Conversion System 


maildir is enabled

1) Restore system to mbox (non-maildir) state.
2) Request Technical Support
3) Convert partially converted mail accounts
*) Exit
Enter Selection---]

Choose #3 to retry the conversion. This should go very fast if the process completed or nearly did so. If any mbox files were missed or not converted they will be done now.

Type * when done to exit the conversion script.

Now test sending and receiving mail both regularly and through webmail. If all goes well, you should have no problems.

If for some reason the conversion has NOT gone well, rerun /scripts/convert2maildir again and select #1 to revert back to mbox format using the mbox backups you made earlier.

Hopefully you won't have any serious problems, but if you do, contact the helpdesk for further assistance.

You and your users will notice that the exim line in the System Status in cPanel and WHM now looks different:

exim (exim-4.52-15_cpanel_maildir)

You've got maildir active on your server.

Regards,

Aric

Note: this article was created by Aric Pedersen
Author of the cPanel User Guide and Tutorial and Web Host Manager Administration Guide

**Neil** · June 11th, 2007, 1:50 PM

Thank you, nice to have an idiots guide nice and early.

**Aric** · June 19th, 2007, 9:39 PM

Pedersen, Penderson, who's keeping track. ;-)

**Mike** · June 20th, 2007, 9:35 AM

Penderson just rolls off the fingertips and reminds me of a Simpsons character.
Corrected

June 11th, 2007, 1:40 PM	#1 (permalink)
Mike Operations Super #1 Joined in Jun 2005 Lives in surpass headquarters 1,014 posts Gave thanks: 66 Thanked 125 times	cPanel 11 Prep: Converting from MBOX to MailDir Format Note: this article was created by Aric Pedersen Author of the cPanel User Guide and Tutorial and Web Host Manager Administration Guide A few people have asked me about the whole mbox vs. maildir thing and what this means for dedicated server owners. Here's the deal: Right now, all cPanel servers store mail in mbox format and use uwimap. See: http://email.about.com/cs/standards/a/mbox_format.htm and http://www.washington.edu/imap/ mbox format has been in use forever in Internet years. Lots of e-mail clients can import and export mbox files. It's not hard since mbox files are basically plain text 7-bit ASCII files. It's worked well for a long time, but that doesn't mean we can't do better. cPanel Inc. has decided to move all cPanel servers to the newer, faster and more efficient maildir (short for mail directory) format and Courier IMAP service. See: http://www.courier-mta.org/imap/ and http://www.courier-mta.org/mbox-vs-maildir/ This mandatory conversion probably won't happen for a while yet (they've said they are shooting for cPanel 11, whenever that gets released). However, you CAN switch now if you want and yes, cPanel works just fine with the new format. And yes, it IS noticeably faster, especially for servers with lots of e-mail accounts. However, there are a few things to keep in mind before you decide to jump into the conversion now: 1) Neomail does NOT work with maildir. Neomail is a dead project and isn't being updated for the new format. Users will loose the ability to work in Neomail after the maildir conversion. cPanel Inc. DOES plan to replace neomail with another webmail program (perhaps even the cool roundcubemail), but has not done so yet. Squirrelmail and Horde both work fine with the new format. If your users CANNOT do without Neomail DO NOT convert yet. However, you SHOULD let them know it will be going away! 2) The conversion does take time and can cause high loads on your server during the conversion. For this and all the other obvious reasons, try to do the conversion on a weekend late at night when load and mail use is low on your server. The whole process can take 10 minutes to several hours. How long depends on how powerful your server is, how many e-mail accounts are set up on your server and how much mail is being stored on your server. You DON'T have to watch the entire conversion process, as it mostly does everything in the background, but it is a good idea for you to plan to be around while the process is going on, just in case something goes wrong. 3) The maildir conversion process has become fairly foolproof, but it IS possible to loose e-mail stored on the server if things go horribly wrong. 4) You CAN revert to mbox format if the conversion doesn't go well, but only if you choose to back up mail before beginning. See below for more on that. 5) If you use MailScanner you MUST run the following commands via root shell after the conversion process finishes or no mail will be scanned by MailScanner: Code: /scripts/buildeximconf /scripts/mailscannerupdate service exim restart Don't forget! You can ignore this step if you don't use MailScanner. 6) If your clients are having problems with e-mail now or if your server is having any other issues, don't start the conversion. Make sure your server is working properly FIRST and then convert. This will save you a lot of hassle in the long run! 7) If you have installed a third-party webmail client on your server, you will probably need to update the configuration after the conversion to take into account the new mail format. cPanel's own webmail clients will automatically be converted, so you don't need to worry about those. Are we all clear on that? Personally, I'd rather handle the conversion at a time convenient to me rather than wake up one morning and discover my server automatically converted everything (and messed it up) while I was sleeping. But doing it now is entirely up to you! Here's what you'll need for the mbox to maildir conversion: 1) root shell access on your VPS or dedicated server 2) The latest version of cPanel. You should update cPanel if it isn't the latest version. /scripts/upcp will take care of you. 3) Enough disk space to accommodate the backup of the e-mail mbox files on your server. If you have a second hard drive mounted as /backup, the conversion backup process will use that, otherwise a directory will be created on your existing hard drive. The mbox files are compressed, but the backup can take several hundred MB or more. I also strongly suggest you warn your users in advance if you can that you will be switching so they know to try to stay away from their e-mail boxes during the conversion. Ready? If so, log into your server via SSH as root and run: /scripts/convert2maildir You will see something like this: Code: # /scripts/convert2maildir ____ _ ___\| _ \ __ _ _ __ ___\| \| / __\| \|_) / _` \| '_ \ / _ \ \| \| (__\| __/ (_\| \| \| \| \| __/ \| \___\|_\| \__,_\|_\| \|_\|\___\|_\| Mail Directory (maildir) Conversion System maildir is not active, this system is using mbox 1) Backup all mail folders on this server 2) Restore a mail backup 3) Start maildir conversion process 4) Request Technical Support ) Exit Enter Selection---] Note that this system does not yet have maildir active and is using mbox. Type the number 1 and hit Enter/Return. When you are asked if you want to overwrite any old mbox backups, type y and hit Enter/Return. This will start the backup of the mbox format files. As I said earlier, this process can take a while and will place serious load on your server. Be patient! It may look like it is stalled, but it's still working, trust me! You will see the username of every user on the server and eventually the word "done." after the backup of that user is completed. Users with little stored mail will backup quickly. Those with thousands of message or lots of accounts will take longer. When done, you will find yourself back at the maildir convert menu. Press 1 if you need to backup the files again. This willl overwrite the existing backups with new ones, or we can go on to #3. If you choose #3, you will see a single line displayed and then be dumped back to the command line. This is normal. It will say something like: Watch /var/cpanel/updatelogs/maildirconversion.1152997653 if you want to see the progress of the conversion. It is optional, but I suggest you do just that to make sure everything goes well. Code: tail -f /var/cpanel/updatelogs/maildirconversion.1152997653 Of course change the path and filename to match what your server has told you. This will show you the progress in realtime. Again, this can take quite a while. What it will do is: 1) Check your server and update and install several files needed for Courier and the maildir format. 2) Configure Exim to use the maildir format. 3) Convert every mbox file for ever user on your server to the new maildir format. 4) Do some final Exim and cPanel configuration (and disable Neomail). 5) Check for some needed updates. When the process is done you will see a note in the log file. Do not try to do ANYTHING with Exim or webmail during this process until it is finished. If your users try to access or send mail that is OK. They may receive an error during this process, but no e-mail will be lost (if all goes well). When the process is done and only if you use MailScanner run the following: Code: /scripts/buildeximconf /scripts/mailscannerupdate service exim restart But do not try to do that until the conversion is finished! If something happens or you need to reboot your server during the conversion process or if there are any serious errors, don't worry. Run /scripts/convert2maildir again: Code: # /scripts/convert2maildir ____ _ ___\| _ \ __ _ _ __ ___\| \| / __\| \|_) / _` \| '_ \ / _ \ \| \| (__\| __/ (_\| \| \| \| \| __/ \| \___\|_\| \__,_\|_\| \|_\|\___\|_\| Mail Directory (maildir) Conversion System maildir is enabled 1) Restore system to mbox (non-maildir) state. 2) Request Technical Support 3) Convert partially converted mail accounts ) Exit Enter Selection---] Choose #3 to retry the conversion. This should go very fast if the process completed or nearly did so. If any mbox files were missed or not converted they will be done now. Type * when done to exit the conversion script. Now test sending and receiving mail both regularly and through webmail. If all goes well, you should have no problems. If for some reason the conversion has NOT gone well, rerun /scripts/convert2maildir again and select #1 to revert back to mbox format using the mbox backups you made earlier. Hopefully you won't have any serious problems, but if you do, contact the helpdesk for further assistance. You and your users will notice that the exim line in the System Status in cPanel and WHM now looks different: exim (exim-4.52-15_cpanel_maildir) You've got maildir active on your server. Regards, Aric Note: this article was created by Aric Pedersen Author of the cPanel User Guide and Tutorial and Web Host Manager Administration Guide __________________ Mike Surpass Special Operations Last edited by Mike; June 19th, 2007 at 10:02 PM.. Reason: penderson

June 11th, 2007, 1:50 PM	#2 (permalink)
Neil Surpass Fan On a golden path... Joined in Oct 2004 Lives in UK 359 posts Gave thanks: 6 Thanked 13 times	Thank you, nice to have an idiots guide nice and early. __________________ D17/D21/P59/P62/VPSX - "Faith can move mountains" (Faiths a big girl....) - I'm not paranoid, I know they are out to get me!

June 19th, 2007, 9:39 PM	#3 (permalink)
Aric Just Some Guy.. Comfy Contributor Joined in Oct 2003 127 posts Gave thanks: 0 Thanked 5 times	Pedersen, Penderson, who's keeping track. ;-) __________________ Now Released: cPanel: A User's Guide Web Host Manager Administration Guide

June 20th, 2007, 9:35 AM	#4 (permalink)
Mike Operations Super #1 Joined in Jun 2005 Lives in surpass headquarters 1,014 posts Gave thanks: 66 Thanked 125 times	Penderson just rolls off the fingertips and reminds me of a Simpsons character. Corrected __________________ Mike Surpass Special Operations

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)