SQL table corruption can happen from time to time for a number of reasons. As we have scaled ExchangeDefender, I ran into a number of situations in which the data inserts were incomplete, or some maintenance tasks (large delete queries) crashed the table. This in turn would affect processes that produced reports and BI based on that data and really create a ton of pain and complaints.

So how does one repair crashed myisam tables?

myisamchk -r /var/lib/mysql/database/table.MYI

This process rebuilds the indexes and the entire table, but is very reactive in nature, you have to be aware of the crashed table in order to repair it.

There is a more maintenance-conscious process called mysqlcheck:

mysqlcheck –auto-repair -A -u dbuser -p database

This is a very thorough (read: slow as @#%@) process that will check all your tables and at the end of it proceed with the repair. Fantastic. Unless you have thousands and thousands of tables that you want to check quickly. This process on our dev box took a day and a half to run. Here is one that executes in less than 10 minutes in just a few lines of php:

$myquery = mysql_query(“show keys from $table”);
if(mysql_error() != “”) { backup / repair / report / log }

Stick that into a loop that goes through all your tables and you’re set.

I really don’t like talking smack about my competitors both because I know how difficult of a task we all have always being a step behind spammers and I choose to put my energy into building our own products. After all, with all due professional respect, nobody has unplugged more of their appliances than me.

But yesterday I had a particularly frustrating day of learning more about Linux load balancing than I particularly wanted to. And after several hours of piecing together the concepts through Google and outdated documents and technologies I figured – screw it, I’ll just go buy one from them and move on to one of the other billion projects I have on my desk. I look at the model breakdown and the first thing that strikes me is obvious hard locks in the appliance to limit the number of real servers so you’d have to upgrade to the higher model just because. Ok, fine, what’s $500 wasted on top of $1499? So I go to the order form and they got more fees – multiyear IDS protection subscriptions.

Ok, so now this is just getting ridiculous. So I figured let me hit up chat and see if there is “I’m not a sucker” form. So I ask politely, “Can I buy your load balancer without the extended support contract?” and instead of saying yes or no, he/she responds with “I can send you more information about that, what is your email address?” and I say “No thanks, I don’t want you spamming me, I am buying right now I just want to know if its possible to get it without the energizer updates”; They let me sit around for a minute or two and come back with the barrage of questions: “The updates come with IDS, blah, blah, blah” and I respond with “I just need a load balancer, can I please just order one without updates” and they say “No.” and close the chat faster than I can even blink.

So suffice to say they lost that order. I mean, I can understand that they are crooks and are using the same deceptive advertising that has been available from the beginning of time – low advertised price but by the time you get to the counter you end up paying almost double. And would I have paid $2K? Yeah. But after that treatment I won’t. And this is perhaps yet another reason why you don’t want to be a sales prick, you just might end up pissing off the guys that run data centers and will now spend another day trying to figure it out – and when they do, you will lose a hell of a lot more than the $399 or whatever bs markup it was.

So that’s the lesson for the day: Don’t be a prick when people are trying to give you money. You can still sell by saying “no”  but you can’t sell if you’re throwing customer out of the store.

Anyhow, if you’ve got Linux Kung Fu, this is what I’m trying to do:

ipvsadm -A -t 65.99.255.235:25 -s wrr
ipvsadm -a -t 65.99.255.235:25 -r 65.99.255.242:25 -m -w 1
ipvsadm -a -t 65.99.255.235:25 -r 65.99.255.246:25 -m -w 1

Stock CentOS 5 (RHEL 5) 2.6.18 kernel with net.ipv4.ip_forward turned on and I have a public IP that I want to distribute traffic over the two real servers with direct return (direct path return) with both real servers on the public range. The load balancer is at 65.99.255.230 and here is the tcpdump:

08:29:33.214112 IP (tos 0×0, ttl  64, id 5746, offset 0, flags [DF], proto: TCP (6), length: 60) 65.99.255.230.56232 > 65.99.255.246.smtp: S, cksum 0x4bde (correct), 2861789973:2861789973(0) win 5840 <mss 1460,sackOK,timestamp 64775527 0,nop,wscale 7>
08:29:33.214171 IP (tos 0×0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 60) 65.99.255.246.smtp > 65.99.255.230.56232: S, cksum 0x1f4c (correct), 3193828587:3193828587(0) ack 2861789974 win 5792 <mss 1460,sackOK,timestamp 377709289 64775527,nop,wscale 2>
08:29:33.214179 IP (tos 0×0, ttl  64, id 5747, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.230.56232 > 65.99.255.246.smtp: ., cksum 0×6485 (correct), ack 1 win 46 <nop,nop,timestamp 64775527 377709289>
08:29:33.214232 IP (tos 0×0, ttl  64, id 5748, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.230.56232 > 65.99.255.246.smtp: F, cksum 0×6484 (correct), 1:1(0) ack 1 win 46 <nop,nop,timestamp 64775527 377709289>
08:29:33.214753 IP (tos 0×0, ttl  64, id 38405, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.246.smtp > 65.99.255.230.56232: ., cksum 0x5f0a (correct), ack 2 win 1448 <nop,nop,timestamp 377709289 64775527>
08:29:33.218077 IP (tos 0×0, ttl  64, id 38407, offset 0, flags [DF], proto: TCP (6), length: 117) 65.99.255.246.smtp > 65.99.255.230.56232: P 1:66(65) ack 2 win 1448 <nop,nop,timestamp 377709293 64775527>
08:29:33.218091 IP (tos 0×0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) 65.99.255.230.56232 > 65.99.255.246.smtp: R, cksum 0x33d0 (correct), 2861789975:2861789975(0) win 0
08:29:33.218095 IP (tos 0×0, ttl  64, id 38409, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.246.smtp > 65.99.255.230.56232: F, cksum 0x5ec4 (correct), 66:66(0) ack 2 win 1448 <nop,nop,timestamp 377709293 64775527>
08:29:33.218102 IP (tos 0×0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) 65.99.255.230.56232 > 65.99.255.246.smtp: R, cksum 0x33d0 (correct), 2861789975:2861789975(0) win 0
08:29:33.222272 IP (tos 0×0, ttl  64, id 23742, offset 0, flags [DF], proto: TCP (6), length: 60) 65.99.255.230.45127 > 65.99.255.242.smtp: S, cksum 0xb04d (correct), 2865052113:2865052113(0) win 5840 <mss 1460,sackOK,timestamp 64775535 0,nop,wscale 7>
08:29:33.222380 IP (tos 0×0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 60) 65.99.255.242.smtp > 65.99.255.230.45127: S, cksum 0xb60a (correct), 2798131280:2798131280(0) ack 2865052114 win 5792 <mss 1460,sackOK,timestamp 408392 64775535,nop,wscale 2>
08:29:33.222396 IP (tos 0×0, ttl  64, id 23743, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.230.45127 > 65.99.255.242.smtp: ., cksum 0xfb43 (correct), ack 1 win 46 <nop,nop,timestamp 64775535 408392>
08:29:33.222476 IP (tos 0×0, ttl  64, id 23744, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.230.45127 > 65.99.255.242.smtp: F, cksum 0xfb42 (correct), 1:1(0) ack 1 win 46 <nop,nop,timestamp 64775535 408392>
08:29:33.223193 IP (tos 0×0, ttl  64, id 8985, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.242.smtp > 65.99.255.230.45127: ., cksum 0xf5c7 (correct), ack 2 win 1448 <nop,nop,timestamp 408393 64775535>
08:29:33.242440 IP (tos 0×0, ttl  64, id 8987, offset 0, flags [DF], proto: TCP (6), length: 117) 65.99.255.242.smtp > 65.99.255.230.45127: P 1:66(65) ack 2 win 1448 <nop,nop,timestamp 408412 64775535>
08:29:33.242454 IP (tos 0×0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) 65.99.255.230.45127 > 65.99.255.242.smtp: R, cksum 0×9847 (correct), 2865052115:2865052115(0) win 0
08:29:33.242458 IP (tos 0×0, ttl  64, id 8989, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.242.smtp > 65.99.255.230.45127: F, cksum 0xf572 (correct), 66:66(0) ack 2 win 1448 <nop,nop,timestamp 408412 64775535>
08:29:33.242465 IP (tos 0×0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) 65.99.255.230.45127 > 65.99.255.242.smtp: R, cksum 0×9847 (correct), 2865052115:2865052115(0) win 0

I see the connection come up, and the weights look right:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  65.99.255.235:smtp wrr
  -> i246 Route   1      0          0        
  -> i242 Route   1      0          1 

So I’m missing something here… If you can see it from there, let me know.

Do you find yourself puzzled by all the buzz surrounding the latest low cost gadgets running Linux? If you’re not that good at counting you might even think the buzz is translating into a huge opportunity. Well, no. Not really. But there is a lot of buzz. So let’s take a look at it:

First and perhaps most impressive is the EeePC from Asus, a $400 laptop, that seems to be on the neverending wishlists mostly because retailers are getting less than 10 of them at a time and it’s sold out everywhere.  Coming in at under 2lb, and with a 7” screen, 4GB flash drive (1.4GB available for storage) and a 900 MHz processor this device is as unimpressive as it’s battery life.

Second, and most interestingly completely sold out Everex TC2505 gPC at Walmart, is a $200 PC. Built around the premise that all applications will be web based and delivered through the cloud, gPC is pitched as a strict web device. Not that you can expect a lot more from this 1.5 GHz VIA C7 powered system that has sold out the entire initial shipment of 10,000.

Coming in third, for the sake of cutting down this near infinite list of Linux contenders, is Dell with competitors both unique and weird. Dell is rumored to have sold over 40,000 Dell / Ubuntu PC’s, likely one for each person that signed a petition for Dell to offer Ubuntu.

Honorable mention is also in order for OLPC XO laptop, ugly enough to make a third world child cry, which at least has a wonderful moral mission of providing children in third world countries with access to technology long before they are enslaved by Microsoft technical support and forced to say their name is Abraham or Roger.  

So what’s going on here…

Some of us that have been around for a while remember the nearly weekly announcements of NetBSD being ported from one appliance to another. Everything down right to a toaster seemed to get NetBSD support. This tradition of people porting operating systems to things that aren’t powerful enough to even control the lights in your bathroom continued into the new millenium, with Mini-ITX cases and boards, often available for as little as $50 with the motherboard and chip included and completely integrated – just supply case/power and storage.

Point is, the low end of the market is not really as low as we may imagine, at least not for the essential “computer user” operations like document management, surving and maybe some basic media.

So are we seeing a revolutionary new offering catering to the side of the market that has traditionally been ignored, or are we seeing the same old thing with some crafty demand management?

IMHO, what we see here is nothing new, at least not technically. Geeks have always been willing to part with small sums of money for the gadget power and the ability to say “My ___ runs Linux.”; What does seem to be new is that the Linux gadget makers may have mastered the intentional supply chain mismanagement to artificially paint the illusion of high demand. For Dell, Ubuntu is a failure – 40,000 machines is nothing. But if you ship 4 Eee’s and all 4 sell out, you’ve got a winner on your hands. People start tracking web sites so you can see where you can buy one, one shows up on eBay at 4x the cost… you have a winner, right? 

If you’re terribly bad at math, yes. Otherwise, you’ve got maybe 50,000 systems that cost 20–40% less than the entry level systems offered by Dell and HP, but far, far, far less functionality. The moral of the story is to always keep an eye open for the contenders but also to see it in the context of the big picture.

Perhaps two of the coolest new motherboard features in quite some time, shame it’s from Asus, but kudos on producing this.

I really cannot do it justice with my own review since I’m just a screaming fangirl when it comes to things like this so you can read the review on your own.

In a nutshell: Just another motherboard, except this once can fire up an embedded Linux environment with Firefox and Skype in 5 seconds. This is a godsent for quick computer access, troubleshooting and diagnostics, not to mention for homes that do not (or should not) have systems on 24/7. Price tag is $360, more than even a decent server mobo, but first mover always cashes in.

One of the search features I would really appreciate would be to search for the results available in the database on a particular date. For example, what if you searched for “Bush nucular evidence” in 2001 vs. 2007, you’d get some wildly different result sets.

My particular problem, and the reason I am writing this blog post at 5 am, is that for the past few hours I have been trying to locate a mailing list response from the author of dovecot from the long, long ago. I cannot figure out how to do static maildir mappings without involving uid/gid in auth_userdb database.

Google does have a search that limits how old of a result you wish to see (show only pages that were first seen over the last 6 months) but nothing to say show only results available prior to date m/d/y. Oh well, there is always hope for live.com search, lol.

(not a rant, hey there is first time for everything)

By now you’ve surely seen at least a dozen articles on how, depending on the source, Microsoft is ready to eliminate Linux from existence or collect royalties or sue “them” into oblivion. It’s a juicy topic because it’s about a lot of money, emotion, business with a great protagonist and so on. Everyone from Scoble down to the guy with a rock standing in front of a copyright billboard has chimed in. What’s it really about and why will it never happen?

It’s about Microsoft trying to put the Interop picture together for Longhorn.

Because nobody likes Microsoft already, and suing will only make it worse while admitting failure.

Point 1: It’s not about Linux.

Many bloggers will pull stories and anecdotes about how this is about a huge company that is seemingly falling apart (while making 14 billion dollars a quarter) exerting it’s power to eliminate competition. They then transition to the conspiracy theories that SCO lawsuits were funded by Microsoft. Then they support that by a dozen or so articles in which Microsoft lost the patent lawsuit and had to part with a mountain of cash. And, for the cherry on top, they sprinkle something about Microsoft stealing from Apple for 25 years, how any offending code would be rewritten/removed within hours, etc. But in a nutshell, the take of nearly every article is about an evil Microsoft trying to kill off the little Linux.

Meanwhile at the ranch Microsoft is working at perhaps the most significant software release in it’s history, Longhorn Server, which will set it miles apart from any other server suite available just like it’s done with it’s Office suite. And if you see where Microsoft actually makes money you realize that selling Microsoft’s next server into the enterprise involves having a decent interop story for the customer. Whether Microsoft likes it or not, the business reality is that more and more ISV’s are writing software and delivering it on Linux servers – so for Longhorn to be a sell it has to be able to prove it can play nicely. Thats all there is to all this: Microsoft is trying to build up it’s glossy orange flyer with Linux distribution vendors who have signed interoperability agreements so Microsoft can go to the customer and say “Look, you run Redhat, Novell, Ubuntu and Debian and we have interop agreements with those folks to make sure the application you just purchased will be compatible with the middleware you have running on Windows Server” – and how do you coerce the distribution vendors into siding with you and working on Interop? By sprinkling a little patent doubt in the air which you (Microsoft) know full well you won’t be pursuing.

Point 2: Enough people already hate Microsoft

There are 800 lawyers at Microsoft, likely 780 of them are there to deal with continuous lawsuits stemming from inappropriate behavior that happened ten years ago. But read the blogs and you’re left with the impression that those 800 lawyers are just there to do recruiting to get more lawyers to sue FOSS out of existence.

It’s not difficult to find people that hate Microsoft, be it a developer a customer or partner. When you’re big you’re bound to screw people. Heck, even a one man shop has managed to make enemies for itself. Just the nature of business.

So let me clue you in to a little thing about Microsoft that I’ve learned about their culture through working with them for the past 10 years or so. Microsoft guys, be it in sales, development, even executive levels truly believe, beyond a shadow of doubt, that they are the best. Absolutely, without question, Microsoft BOB in one hand and Windows ME in the other, 640K ought to be enough for everyone – the best. Whether they are right or not, Microsoft, by culture, believes they are the best, with the best value, best technology. How? Through innovation.

Going into extended patent lawsuits would make it pretty clear to its shareholders and employees that Microsoft is no longer a business built on innovation and expertise, but one that barely survives on the very thing they have been widely accused for years – FUD. Which is why you’ll never see the lawsuits. It would be ridiculously expensive and highly demotivating for the company culture and what Microsoft stands for.

Microsoft doing a patent suit over innovation is similar in experience to the blind test of which soda you prefer. If you don’t win do you turn around and say “Wow, I’ve been a jackass for 20 years, thank you, thank you for pointing out what I’d like drinking better” or do you walk away in resentment that you picked incorrectly?

Same for the Microsoft customers. Their customers already dislike the business practices that Microsoft has. They outright hate the licensing. Developers and partners aren’t far behind. And we’re ALL buying software and hardware from other companies. So is Microsoft going to pour gasoline on that fire by telling us, to our face, that we’re thieves as we have already purchased solutions based on non-Microsoft products? Or will they just try to build a better story about how this new thing they have coming will play nicely with that third party stuff we already bought?

Personal note:

As you’ve seen on this blog, I talk a fair amount of smack against Microsoft. Yes, they deserve it. I also run more Microsoft software than any sane person ought to be allowed to. I’ve been awarded a Microsoft MVP award for two years in a row. I send large checks to Microsoft, on monthly basis, all while bitching about how Microsoft is putting me in the early grave with lack of patch QC. I write about problems and solutions for Microsoft software, all while leading ITPRO groups focusing on Microsoft software. I develop software for the Microsoft platform. And on daily basis I am asked if I love or hate Microsoft, if I’m pro or con Microsoft, if, if if. Everyone wants a fanboy.

The key thing to understand here is that when it comes to business there are many shades of grey, many circumstances, many compromises that have to be made as you go on. Thats what makes people successful, realizing that there are tradeoffs to every position. The alternative, as you’ve seen in the blog posts on this subject, is to take black vs. white approach to it all, cite historical references and dollar figures and adopt a faith-based view on business management by which you accept a single fundamental truth while ignoring every shred of concrete evidence that smashes it to pieces. Try to see the bigger picture here or come to terms with having to move to rural Mississippi.

So the cat is out of the bag – Dell officially supporting desktop Linux. Linux desktop has been a tough sell on the desktop particularly due to the lack of IT solution providers capable of supporting it for the SMBs. However, with the Dell push things will certainly change. Have your customers considered Ubuntu? Do you know that you can install Ubuntu and sync it up to Exchange in less than an hour? As I mentioned on Vladcast #2 two weeks ago, the time to wreck your test box with Linux desktop is now.

You know who I feel sorry for most of all? Imagine that poor sucker that has to figure out how to roll all the crapware Dell packs their desktops with on Linux. All the meetings and decisions, decisions, decisions – “So we need to annoy the user with more popups but they have multiple desktops in their window manager – so how do we broadcast the popup to every single deskop available without crashing the box?”

I’m curious to see how this is taken by the SBS community that has been traditionally afraid of open source solutions. Will you try it out? Ignore it? Wait for it to show up at the customer site like Mac’s? Support it already? You know where to find me..

Because too many hobbyists discussing politics are misrepresenting themselves as IT professionals and Linux evangelists. Nearly every IT Pro I know has used Linux or currently uses Linux – whether they like it or not is not up to discussion but there is a way to influence people and then there is a way to look like a total basketcase. Case and point, courtesy of a poster on Orlando LUG:

“Remember, Microsoft IS a multiple convicted Felon pirate thief
monopolist con-artist, on several continents, and is under indictment by
governments, as is stated in the press, in several nations, at this
moment.  Japan, Australia, the EU, all have serious problems with Felon
Microsoft.

One method of dealing with convicted felons is to NOT negotiate, but, to
enforce the laws against them, especially when they are still on probation!
What is wrong with people, that they continuously choose failure?”

You want to play “software evangelist” – this makes me want to use Linux…. how? Even if it were completely true, would you not try to find a way to walk away from someone like this as fast as possible?

Congratulations to the CentOS team. They released CentOS 5.0 today, which is the free version of the Enterprise Linux OS. Basically they take the source for Redhat Enterprise Linux 5.0 and recompile and rebuild a solution so you can install and run it.. Free vs. $3K, easy choice if you know your way around the OS.

CentOS 4.x has been a rock solid solution for us at OWN and it runs pretty much everything from ExchangeDefender, to web, mail, dns and more servers. If you need a reliable solution this is the one you go with, not to mention 5 years of support. What do I mean by reliable?

[10:57 - 0.90] [root ns1] ~]$ uptime
 10:57:41 up 161 days,  2:53,  3 users,  load average: 1.63, 1.00, 0.92

(that’s 161 days of uptime, fully patched and up-to-date system that hasn’t needed a reboot in 6 months. And it ran happily ever after. Sounds like a fairly tale, doesn’t it?)

An interesting bug came up in the OWN Forums over the weekend, when downloading Office 2007 Word (docx) document from an Apache hosted site, the file either comes up garbled (application/octet-stream) or opens up a zip folder. This is usually the case when you have a mime-type and content-type mismatch and can get even uglier.. so what’s the solution? You have to define new Office 2007 mime types in your web server (or more precisely, server-side mime type database) so it knows how to communicate the content properly to the web browser. Office 2007 uses the new zipped XML OpenDocument format. After Googling for a few minutes and not finding the solution on the first four pages of searches I came to my buddy David Overton’s page – Now I remember where I read about the new Office 2007 mime types! I’m sure David will be extatic to know his hard work (which applies to IIS6 by the way) is now going to help Apache users everywhere get Office 2007 documents easilly. Joke aside, truth is Apache is the dominant web server and there is at least 70% chance you’re browsing an Apache site (you are right now, it’s what powers Vladville!) or about 98% chance if you’re not in a corporate environment.

Office 2007 mime types from David Overton.

First you’ll notice that there are many different mime types associated with Office 2007 OpenDocument format for just Word documents and templates: .docx, docm, dotm, dotx:

.docm = application/vnd.ms-word.document

.docx = application/vnd.openxmlformats-
officedocument.wordprocessingml.document
.dotm = application/vnd.ms-word.template
.dotx = application/vnd.openxmlformats-officedocument.

wordprocessingml.template

You can hardcode in one of these at a time, however, there is a shortcut. Just open your /etc/mime.types and add this to the last line:

application/vnd.openxmlformats       docx pptx xlsx

Restart the Apache service and you’re set. This takes care of the top three popular Office 2007 formats. If you don’t have access to the system mime type configuration (you likely don’t) you can always override the settings using your .htaccess file. Enjoy and thank David! Here is the end result:

Extra note: Above applies to Apache servers hosted on Linux, however, if you host your Apache server on Windows the server distribution is likely shipping a mime configuration database in a specific path, one that you may or may not be able to find easilly. No worries, you can always add the content types directly to the httpd.conf file which is very easy to locate in most popular Windows Apache distributions such as Vertrigo. To add the new mime type directly in Apache httpd.conf (for Linux fans, /etc/httpd/conf/httpd.conf):

AddType  application/vnd.openxmlformats  .docx .pptx .xlsx

Please note that httpd.conf definitions override the /etc/mime.types definitions and dependant on the directory configuration, htaccess can (and usually does) override both.