Fun with Fsync & Bind Mounts

Open Source
8 Comments

ShagadelicThe answer to “Why don’t you blog about your work Vlad” and the final dagger in the back of my female audiences libido: moving sendmail spools to tmpfs.

First off, why bother? Well, with the ram being as cheap as it is and finally some solid hardware and software tmpfs is really getting a lot of play. It also helps when yours truly wakes up from an one hour nap and finds a node with 68,000 messages waiting in the spool because of poor disk performance. Talk about a motivator to work on optimizing the mail stream!

 

Doin’ it the wrong way

The first step is to actually create a tmpfs disk and mount it.

/bin/mount -t tmpfs tmpfs -o size=256M,nr_inodes=1M /var/spool/mqueue.in

That works. This creates a 256MB ram disk and mounts it in /var/spool/mqueue.in. Start up sendmail and everything works fine until mail has to be moved around and processed. Then you get bit in the ass by fsync errors (fsync is enabled by default on the 2.6 kernels). Time to turn that beast off:

  # override compile time flag REQUIRES_DIR_FSYNC 
 O RequiresDirfsync=false

Now we got the ramdisk, we disabled fsync, it should work now without a problem, right? Heh. Nope. Sendmail queue’s need to be on the same filesystem and same partition. If they aren’t sendmail starts complaining and poof.

 

Doin’ it the right way

This one actually belongs to my bud Pablo who has hacked in bind mounts on pretty much every box I’ve ever had. Bind mounts function similarly (or exactly) like folder mounts with NTFS, a folder (or directory) from one file system can be seen at another point on another file system. Go Pablo: 

mkdir /var/sendmail/tmpfs
mount -t none /var/sendmail/tmpfs

mkdir /var/sendmail/tmpfs/mqueue
mkdir /var/sendmail/tmpfs/mqueue.in

mount /var/sendmail/tmpfs/mqueue /var/spool/mqueue -o bind
mount /var/sendmail/tmpfs/mqueue.in /var/spool/mqueue.in -o bind

 

So to sum it up

First, thank god I found a woman to marry me because girls don’t respond well to pickup lines referencing linux filesystem optimization. Second, moving sendmail to tmpfs really helped, remarkably. In the few hours since the tweak the load average really went down – Linux calculates the load average by the amount of cycles all running processes take up – so if a ton of children are fired up and waiting on IO for something the load goes through the roof. Memory and IO really improved as well but you’ll have to take my word for it because the output from vmstat looks hella ugly. See why I don’t write about what I do?

8 Responses to Fun with Fsync & Bind Mounts

  1. AlbertG says:

    Go.. to.. sleep.. Vlad..

    When you wake up, why not ramdisk?

  2. CScriber says:

    Why not just mount the entire /var/spool and be done with it?

  3. Pablo says:

    You probably don’t want to do /var/spool because of other things that spool there. In my case, Mailscanner uses /var/spool/MailScanner and the files it creates are not transient and can get large quickly. Mail for example used to be saved in /var/spool/mail although /var/mail seems to be gaining acceptance.

    Here’s what /var/spool looks like on my system:

    # du -hs /var/spool/*
    557M /var/spool/MailScanner
    28K /var/spool/cron
    0 /var/spool/mail
    84K /var/spool/mqueue
    4.0K /var/spool/mqueue-client
    48K /var/spool/mqueue.in
    4.0K /var/spool/mqueue.tmp
    4.0K /var/spool/slurpd

  4. Pablo says:

    Vlad asked me the same thing about ramdisk and to be honest, I didn’t know the difference off the top of my head either. Wikipedia answered my question.

    Ramdisk implies (and should behave) in such a way that it is always resident in memory. Tmpfs uses virtual memory so it is possible for it to be swapped to disk if it isn’t being accessed. In practice, there probably shouldn’t be much of a difference.

  5. amy says:

    zzzzzzzzzzz

  6. Jessica Meiners says:

    Yeah, this doesn’t do it for me Vlad 🙂

    Everything else does you sexy, sexy beast.

  7. vlad says:

    Jessica,

    We obviously haven’t met… but I appreciate the sarcastic crack nonetheless, thank you 🙂

    -Vlad

  8. batteryfast says:

    […]belive me Time Theif this has nothing to do with an “actual” horse, or climb on Mt. Everest, this is metaphore for exactly what the tags say. YOU have to read deeper to understand[…]

Comments are closed.