The answer to “Why don’t you blog about your work Vlad” and the final dagger in the back of my female audiences libido: moving sendmail spools to tmpfs.

First off, why bother? Well, with the ram being as cheap as it is and finally some solid hardware and software tmpfs is really getting a lot of play. It also helps when yours truly wakes up from an one hour nap and finds a node with 68,000 messages waiting in the spool because of poor disk performance. Talk about a motivator to work on optimizing the mail stream!

Doin’ it the wrong way

The first step is to actually create a tmpfs disk and mount it.

/bin/mount -t tmpfs tmpfs -o size=256M,nr_inodes=1M /var/spool/mqueue.in

That works. This creates a 256MB ram disk and mounts it in /var/spool/mqueue.in. Start up sendmail and everything works fine until mail has to be moved around and processed. Then you get bit in the ass by fsync errors (fsync is enabled by default on the 2.6 kernels). Time to turn that beast off:

  # override compile time flag REQUIRES_DIR_FSYNC 
 O RequiresDirfsync=false

Now we got the ramdisk, we disabled fsync, it should work now without a problem, right? Heh. Nope. Sendmail queue’s need to be on the same filesystem and same partition. If they aren’t sendmail starts complaining and poof.

Doin’ it the right way

This one actually belongs to my bud Pablo who has hacked in bind mounts on pretty much every box I’ve ever had. Bind mounts function similarly (or exactly) like folder mounts with NTFS, a folder (or directory) from one file system can be seen at another point on another file system. Go Pablo: 

mkdir /var/sendmail/tmpfs
mount -t none /var/sendmail/tmpfs

mkdir /var/sendmail/tmpfs/mqueue
mkdir /var/sendmail/tmpfs/mqueue.in

mount /var/sendmail/tmpfs/mqueue /var/spool/mqueue -o bind
mount /var/sendmail/tmpfs/mqueue.in /var/spool/mqueue.in -o bind

So to sum it up

First, thank god I found a woman to marry me because girls don’t respond well to pickup lines referencing linux filesystem optimization. Second, moving sendmail to tmpfs really helped, remarkably. In the few hours since the tweak the load average really went down – Linux calculates the load average by the amount of cycles all running processes take up – so if a ton of children are fired up and waiting on IO for something the load goes through the roof. Memory and IO really improved as well but you’ll have to take my word for it because the output from vmstat looks hella ugly. See why I don’t write about what I do?

Sometimes reading inbox spam pays off. Today I got the following interesting offer in my inbox from RSA Security, Inc:

Download the Data Protection Strategy Kit and learn how protecting data across the entire enterprise – applications, databases, storage, etc. – can solve business-critical security issues.

How to Meet Your Customers' Security Requirements
Information Kit at a Glance

• Regulatory mandates
  Comply with PCI, FFIEC, ISO 17799, CobiT, etc.
• Corporate policies
  Avoid, among other things, public reporting requirements of U.S. State Breach Notification laws
• Customer requirements
  Ensure that customers are confident in your ability to protect their sensitive information

Data Protection Strategy Kit
At a Glance

• Forrester Trends Report
  Secure the Data, Not Just the Underlying Infrastructure
• Current U.S. Regulatory Climate Report
  Review of Key Regulations and What They Mean to You
• Protecting Payment Card Information
  Solutions for meeting Payment Card Industry and U.S. State Breach Notification Law Requirements
• Technology Backgrounder
  Managing the Life Cycle of Encryption

Download Data Protection Privacy Kit today.

Whats so great about that? Well, one of the ways major corporations (RSA is a biggie) sometimes attract interest in their products by giving away tons of research. More often than not, all these reports and data are written to leave you with a feeling that YOU MUST BUY OUR PRODUCT OR DIE but if you’re smart you can get a lot of very insightful and very expensive information that you otherwise would never get to see.

Advice? Sign up for a gmail.com account and take a look at these reports, at least an executive summary. The more you know, right?

I am often accused of being an anarchist but thanks to my friends I won’t have to worry about suicidal. One of the awesome parts of keeping up this blog is that I don’t have to travel around the world to figure out whats going on, the world comes to me. So over the past two days I’m sitting here getting the news from all over the world about how Office did this and that. I have one thing to say:

thank god for my friends

So if you’re suicidal and this weekend’s football festivities do not interest you, take a stab at office. Until my friends figure out all the kinks I’ll be a spectator as my Office 2007 Beta 2 is running just fine.

I got an email asking me if I’ve gone gay today or if I’ve underpaid anybody on my writing staff. Writing staff? Apparently being polite just doesn’t win with my demographic. Here is a little secret: I’m not as big of an ass as I come across on my blog. Oh sure, I write some really controversial stuff here but for one purpose only: to make you think. One of the negative side effects of the community is that if we’re all in agreement that makes us right! Right? No. Sometimes you need to step out of your comfort zone, look at the ugly picture you’ve painted and choose whether you can change it or make the most out of what you got.

I am relentless in my defense of what it means to be an IT Professional. I coined the term SPF on the message board to spotlight the people we don’t like to admit are among us. Unfortunately some legitimate guys assumed that just because they were sole proprietors they were SPFs. Quite the contrary, here is how you make a difference. This message came in earlier today from a fellow partner:

I also want to say thank you for pissing me off about a month or so ago with your post about the one man band IT consultants that do everything but are masters of nothing. I was pissed for about an hour or two, then I realized you were right. I immediately started offloading the tasks that weren’t my main focus and started working to improve my key services (full service out-sourced IT for small business). I appreciate what you have done for me and forced me to really think about my business.

Notice the subtle keywords there… offloading.. improve.. think.. business.. Think of what your value is, think of were you are going with your business and take it there. Challenge yourself to put down a vision, a map to where you want to be 1 year from today. Block your calendar 1 hour a day. Just one hour. 60 minutes. Use that one hour to drive yourself to that goal. If your goal is to be a fully managed shop in one year then establish a timeline – In October, organize. Evaluate. In December, document. Evaluate. In January, seek accounting advice. Evaluate. In February, seek legal advice. Evaluate. In March, image the documents. Evaluate. In April, integrate with CRM. Evaluate. In June, go to a technical conference. Evaluate. In July, go to a business conference. Evaluate. In August, review. Evaluate. In September – evaluate the year, redraw your goals, review your vision and.. heck, take a vacation.

What are your goals? Be fully managed? Be totally efficient? Quit your current IT job? Get married? (oh, I’m going to get killed for that one) Susanne has a nice writeup of how she got her business closer to Microsoft. Don’t be afraid of the community. Don’t be afraid of asking for help. None of us are 100% satisfied with what we have and what we do – this is why we are still out there trying, working, communicating, asking, connecting, evaluating, competing, most of all: bsing. This is all a big learning experience, a big puzzle. We’re all trying to put it together one clueless piece at a time.

It’s Friday already, where does the time go? I spent last week at a conference in Redmond and to be honest have not slowed down for a minute since I landed there. As I’ve written (in the now famous suicide note) over and over, there is nothing better for your business than spending time with friends and professional associates just goofing off and getting a perspective on your business. You get to find a whole new side to your business and if you’re lucky enough, yourself. But since this is about me, todays lesson is that kindness matters.

The Ferry

Last weekend we went for a joyride on Bainbridge Island. We pretty much just goofed off, played stupid games, had an excruciatingly long conversation about a bodily movement, pulled off road at a sheep farm just to offer it to a guy from New Zealand… Check. I was feeling a bit down mentally and physically and I didn’t go along with the crowd when we got on the ferry… Few minutes later I got out of the car, got some fresh air and… a friend came down just to check on how I was doing. How nice!! As little as it may have been to them, it made my day. Someone gave enough **** about me to leave their other friends to check on me. On the way off the ferry we saw a girl that really seemed like she had enough of holding the sign. The look of utter exaustion on her face, eyes glazing somewhere in the distance with the cruise ships. Holding a sign must blow. She glanced over at our car, I smiled and waved back.. She woke up I guess, smiled, waved back. You see, sometimes it’s about the little things. 

The Security

Not a big friend of NTSB. I was flying back on Monday, late night redeye flight. I decided to drop the car off early and spend some time with a friend I rarely get to spend time with. Because of safety my airline wouldn’t check my luggage (apparently, can’t take bags more than 4 hours in advance). Because of safety they also wouldn’t hold it either. Fine, trek accross the airport, drop it off in paid storage, climb up 3 flights of stairs, off through security. Wait in line for god knows how long and… finally, through the scanner. I don’t even know why or how but I started chatting with the NTSB guys working the scanning equipment. Turns out one had been there for 5 hours and has another 8 to go. The other had been there all day. Damn. Now these guys are probably blamed for everything from the towers going down to the war in Iraq to the long lines at the airport to the tighter seats on the airplane. On that day it must have been very special too, you see, it was 9/11. I don’t know why or how but I just said:

Thank you guys for all your hard work… especially today.

I think they both stopped what they were doing and just looked at me as if they have never heard those words before…….

Take from this what you will, sometimes it is little things that matter. Think about people who do the little things for you and go out and thank them today.

Earlier today I posted a question on a mailing list trying to find out how other IT Solution Providers are dealing with the increasingly unreliabile and costly Microsoft Security patches.

Please don’t turn this into a security issue because it’s a business question:
 
I am depressed with Microsoft patching to the point that I might have to drop my SLA against all Windows-based servers at Own Web Now. Even  on a day when the patch does not cause any problems at all the reboots don’t happen as they should. Vanilla configurations just do not start all services. Make up a weirdest thing you can get a Windows server to do and we’ve seen it. Remember that this is on a good day, not on a bad day when the security patch locks out Blackberries one month, Macintosh the next, crashes Dell boxes the month after that.
 
I am considering automatically dropping all Windows servers into an automatic 8 hour maintenance cycle during the Microsoft patchday to compensate for Microsoft’s lack of QA. We can no longer minimize issues through testing because even identical boxes (Hardware and software, remember we virtualize the crap out of things) are not behaving the same. Reboots before the patch are fine, reboots after the patch.. poof.
 
How is everyone else handling this? Drop the SLA? Lower confidence in Microsoft (who does that help?) Extended maintenance cycle?
 
Second Tuesday of the month is becoming a religious holiday at Vladville…

The Process

Our process and our ingredients are pretty simple. We do a flash backup every Tuesday afternoon (EST). Those backups are generally complete by 10PM. We do a flash reboot just to make sure there are no hardware/software issues. We proceed with the patches that passed quality control / quality analysis earlier that day. We push using a collection of tools, WSUS and other bits and pieces. Other bits and pieces are used instead of WSUS when we want to apply hotfixes without a reboot to critical infrastructure systems.

Either way, pretty standard stuff. Most Windows servers run a similar configuration (actually, most are identical in both software and hardware as they are mostly Virtual Server systems) so there is little reason to expect one to work while the others fail. 

The Costs

Do not let Microsoft WSUS and “Secure by Default, Design, Description…” fool you, patching is expensive, very expensive. There is no alternative to patching, we have to do it. With critical updates, we have to do it ASAP. No complaints there though, its just a part of business.

My complaint is with the unplanned costs related to patching. Costs that I and my customers have to pay because Microsoft produces unreliable and unstable patches. Let me explain what my definition of that is: “If a patch causes unexpected downtime or adversely impacts my system performance I do not consider it to be stable or reliable.” Simple as that. A patch is supposed to close a security hole in the software without affecting the rest of the system.

This is no longer the case. Few months ago Microsoft patch knocked out Macintosh systems (Entourage) from connecting to Exchange. Month after that it stopped Blackberry from operating properly. You remember my post about it regarding Dell.

My actual complaint is that I am at the verge of losing confidence in Microsoft’s ability to reliably and predictably patch the problems in their software. It is costing me a small fortune both financially and in terms of reputation. If I cannot stand behind my SLA (Service Level Agreement) which states just how often the server will be up then what value am I providing. If I am put in the position of having to appologize for things that are not my fault to begin with, where does that put my reputation at with my customers? Forget about the cost of overtime for employees, support calls, graveyard shifts, and the near cottage industry built around the patching tools, preparation process, reporting and followup just to make sure that the software we paid for continues to behave the way it was sold to us.

Forget about me

Now this is simply a blog post that will change… nothing. But it is an opportunity to review your SLA and consider how you deal with unreliable partners whose products and services you are supporting. I am at the verge of having to rewrite my SLA to put Microsoft patches into a maintenance cycle without any assurance on the time period. Here is one of the intriguing answers I got:

Vlad, we ran into the same issues as we started to scale and eventually had to build a lab for testing where, once approved, the patches would be put on our corporate network and when approved, we would roll them out to the clients. To resolve the reboot problems we put in “lights out” cards in all our servers. I agree it is not for the faint of heart.

Anyhow, something to consider…

Now I know I said I would not post any more inappropriate stuff but I just cannot resist. Here is a good one, featuring Mr. MCNA:

Susanne wouldn’t pose with us in front of the NetWare logo. So here is one where she did grace us with her presence:

Inappropriate…. in any location.

Get the virus, baby! More pictures of SBSers talking about Shockey Monkey..

I am back in Orlando and back in the Ironman mode. This past weekend has been so fulfilling, professionally and personally, that I feel like I just got off the bike and have a marathon to run. That feeling of complete exaustion where your body cannot go anymore but your mind races at the speed of light? I spent last night in a massive braindump just taking down all the notes and as much as I’d like to write them all down here I am afraid there would not be enough time in the world to do all that.

I will say this. If there is anything about me, be it good or bad, it is my sincerity. Look at the introduction of Vladfire #2 and multiply that by a million to get to the state I’m in now. Someone once said that you are only as good as the people you surround yourself with. If thats the case then I’m much better off than I ever imagined.

So… Thank you Susanne. Thank you Susan, Paul, Wayne, Erick, Eric, Karl, Amy, Steve, Eriq, Chad, Dean, Jeff, Andy, Rick, Bob, Scott, Dave, Judd, John, The Good Amy, Dana, Beatrice, Scott B. These are the people I consider as my friends that tell me when I’m being an ass and egg me on to be even bigger. Case and point, after saying goodbye to a friend last night at the gate I went back to the airport book store and bought a book. If your friends don’t cheer you or inspire you then you need to get out more.

And that, moreso than anything, was the value of the past few days.

Andrea Russell runs the Small Business Specialist program for Microsoft…. and when I say runs I mean everything – from vision down to peeling vynil signs off the windows at the World Wide Partner Conference.

Please take a few minutes to view this video and get her take on the community and what we’re all about.

Runtime: 5:47 Minutes.

Download a WMV (Microsoft Windows Movie) | (30 Mb)

Stream Quicktime (Fast, Streaming, Requires Quicktime) | (10 Mb)