The Worst Week of My Life

Posted: 4:23 am
March 2, 2008
Vladfire
3 Comments

Last week was the worst week of my life. Much like the most depressing lines out of the movie Office Space: “Each day is worse than the previous one. So every day you see me, it is the worst day of my life.” I’ve blogged here about failure (extensively) in the past and how one should not be afraid of it – but the disheartening thing is that no bad news comes alone – it usually turns into an ugly streak. When this happens my only prerogative is to keep on working on finding a solution instead of giving up. Hard head is better than no head.

officespace_lumbergh

So what went wrong this week? How about a 40% increase in the network load combined with the 600% increase in sales? Biggest week on the books (ever) coming in during the week when you experience 40% increase in traffic is a blessing and a curse. On one hand, mo money. On the other hand, you’re stretched to the limits not just in terms of infrastructure and NOC upkeep, but you’re also spending time with service provisioning and sales management (for our purposes, service provisioning including capacity planning is a part of a sales process). So on the week in which I had planned to put in 20 hours or so to wrap up Shockey Monkey 2, I spent 40+ hours on ExchangeDefender, OWN’s infrastructure shipments to Australia and Europe, meetings and more contracts signatures than a mortgage process requires.

The Fun

This week we experienced 40% increase in the network activity. Why? Spammers got a new trick. They are bouncing mail off recipients with OOFs enabled. They also started actively abusing the loophole in rfc2821 which requires every MTA to accept null sender messages. Originally, this process was envisioned as a way to communicate transmission errors and delivery problems without putting any additional checks on the senders email address. For example, non-delivery and delivery receipts use the mail from: <> process. Try it, you can deliver any message to any server to any recipient using the mail from: <> trick.

Now, consider that you can’t block mail from: <> and also consider that you cannot build an RBL context against a server that is being attacked by a third party, and consider that there is nothing in the message to filter the junk contextually. Long story short, we implemented watermarking in ExchangeDefender that uses an encryption key inserted on the outbound server and validated on the inbound server. Basically, valid NDR and rejections will have the header value that can be validated if it passed through our servers. If it didn’t… poof.

But whatever affects ExchangeDefender, affects the Internet in general. And anything widespread enough can start taking servers down all over the place. This was the case that brought in the new customers this week. I personally spoke to several people that were referred to us by Microsoft CSS because their servers were completely DoS’ed off the network and they had no way to deal with it.

“We spent the past three days on phone with Microsoft and did everything from renaming the server, changing the MX record, changing the IP addresses and server names.. and we are still dying.”

Now, I hear that every now and then. I rarely take a dozen calls a day and have them sound the same. I also never hear the words: “Microsoft recommended you.” – I’d send you folks a gift package or something but I’m guessing your superiors wouldn’t be happy knowing you’re not selling EHS Foreskin or whatever it’s called this week. Thank you nonetheless, and in case you didn’t know, we give free ExchangeDefender to blue badges for their home servers 🙂

In Other News

You can almost taste the fear in IT hinting the imminent downturn of technology spending and hiring.

How do I know? My sales figures. They are through the roof. Why? My best guess is that everyone out there is so scared of the 2008 outlook and the looming recession in United States that they are hitting the pavement hard and trying to sell and close deals. The pace is almost unbelievable. Partners that used to add a customer a week or a month are adding multiple customers a week. They are also growing in size and in scope (though the last one is partially biased by us since we’ve stepped up the game of offering a whole bunch of services and making ourselves one-step, one-contact, one-support solution so our customers are now trusting us with more solutions instead of nickel and diming around other solutions and holding the support bag)

So I guess nothing motivates like fear. Or maybe with the SPFs and riffraff out of the way and out hunting tax certificates and mortgage foreclosures, the customer base is turning to the more reliable IT shops after getting burned. Or maybe the very same customers are turning more strategic in the face of fear and focusing on beefing up their own revenues instead of trying to DIY and patch things along.

 200px-Ayrton_Senna_Imola_1989_CroppedWhatever the reason, adversity either brings out the best or worst in us. Competition too. I was lucky enough to grow up in Europe (which has real racing, not hicks spinning around an oval) watching F1’s arguably greatest and most competitive racer Ayrton Senna. One thing that usually ended our mornings watching the RAI broadcast would be the rain on the circuit – when it started raining you knew Senna was going to take the race. He always pushed the limits, the car, and himself.. and in one of the interviews I will never forget, that hits me to this day, was the question the reporter asked Senna: “Why are you so good when it starts raining?”

Because where others hit the brakes, I floor it.

Remember, most people don’t start a business and most that do end up failing. So you’re up against the best, you’re either in this to compete and win or you should just quit now. Nobody wants to hear your troubles, though they make for great blog posts.

(Senna’s story of aggressive competitive nature and courage would be a little more impressive/relevant if he didn’t kill himself by slamming into a wall at over 200mph. Ditto with Earnheardt. But if you love what you do, and you get a thrill from being the best its hard to take #2)

Got mrtg game?

Posted: 12:40 pm
March 1, 2008
Uncategorized
3 Comments

One of my goals in 2008 is to realize when I’m being retarded and not cut out for this gig. So instead of banging my head against the desk for the next hour, I’m going to throw a little Vladville game that should be easy enough to answer for anyone with mrtg skills:

I want to use mrtg to plot two integer values on the same graph. I have a shell script that returns two integer values already, and I want mrtg to paint them on the same graph.

First person to post the answer in the blog comments gets a free copy of Microsoft Small Business Server 2003 R2 retail box with 5 CALs (retail ~$600).

Thanks for your help!

Googirl?

Posted: 5:28 pm
February 28, 2008
Misc
7 Comments

Note to journalism majors: If you’re going to be a smartass with your titles, at least run them by Google to see if they have any double meanings. If you like your job, don’t do this at work. If you don’t, here is the first hit.

googirlscan

Poor Marissa..

Virtually speaking, Windows Server 2008 STD Virtualization is my fault

Posted: 5:10 pm
February 28, 2008
Microsoft
1 Comment

Lot’s of confusion over the virtualization of Windows Server 2008 so I figured I’d explain it in some simpler terms:

Windows Server 2008 Standard can optionally be used as a VM host server only, and Microsoft allows you to use the single license you purchased for the host system in a virtual machine without paying more.

It’s really as simple as that. This came out of the penalty that many of us in the server world faced, that of having to pay for a Windows Server license just to host other systems underneath it. In effect, the host system (physical) license was being wasted. (Read between the lines folks: If we’re just going to do virtual machines why not just get vmware esx and not waste money on Windows 2008 server). Now, here is the divergence and also a bit of a clue why my colleagues aren’t getting this:

The real world of virtual machines: consolidation of old systems, uniform distribution of applications, distribution of load based on need (fire up multiple vm’s when the load spikes, shut down systems when the load is not present)

The SMB world of virtual machines: extended functionality by isolating application servers / roles into separate VMs.

Virtually (he he) nobody uses VMs to extend functionality of the core network, they use it to consolidate aging systems and standardize app distribution. So for the real world, this is a huge, huge thing. For the world of SMB, this is a puzzling choice.

So there you have it folks. If you don’t like what’s up there you can blame me, I lobbied for the licensing scheme above day and night because it makes sense for our business and fulfills the need of our clients. Sorry.

Not all servers are right for all people, not all licensing is perfect for every situation. And in Microsoft’s never-ending near-futile attempt to make the licensing fair for everyone the licensing complexity goes up and deal hunters that try to find the cheapest possible licensing scheme find confusion in the licensing solution that was never designed for them. So if you’re in the licensing headache mode, sign up for Eric’s “Licensing Deals” blog and check out the upcoming licensing tours.

Windows Server 2008: My, how the times have changed

Posted: 10:47 am
February 27, 2008
Microsoft
1 Comment

Today Microsoft launches undoubtedly the biggest product in history of the company, at least in the terms of solidifying the market it dominates – business server room. Susan mentioned the launch virtual event which I will be on because I’m anxious to see the sell job. Windows Server 2008 will certainly make a difference in our data centers, I am not sure how many people we will be able to convince to get off 2003 before its support expires, the automation and powershell at the core are absolutely mindblowing.

Microsoft seriously needs to fire its entire marketing staff, entire PR staff, and every single person involved in the launch of Vista and Windows Server 2008. Their launches are pathetic, and fail to spark even the least bit of interest from anyone outside of the server room. Even casual geeks have no idea that today is special in any way.

Just take a look at this: Microsoft headlines of the day? Microsoft fined another 1.3 billion for antitrust abuse. AOL CEO dubbing Microsoft-Yahoo a mistake, and if anyone can speak about perils of mergers of companies with radically different cultures its AOL. In other news, Apple keeps on dominating, ad clicking and page visits down (bad for Google, but far worse for Microsoft) and the rest all about the Web 2.0 world in which Microsoft for all intents and purposes simply does not exist.

I love Microsoft, and we’ve put up with some bad releases, but if they keep on mismanaging their big news like this and awareness for their products keeps on sliding…. I dunno Steve, try throwing axes and pinkslips..

Live takes a Dive

Posted: 6:27 pm
February 26, 2008
Uncategorized
3 Comments

Earlier today, bCentral / MSN / Live and whatever Microsoft’s Googlekiller is called today, decided to take a break. It woke up a little bit later, dazed and confused, and asked if Yahoo was there yet. Alas, no. 🙂 Ok, all joke aside, another major network went down. Joining Blackberry which seems to do it every other month, Twitter which does it every few days and Amazon’s S3 which went down for the first time ever.

The media freaked out. Users started burning their computers, throwing them at children that are passing by, killing kittens.. and these are IT people? Really? Reaaaaaly? Reaaaaaaaaaaaaaly? I’d love to live in the IT neverland in which people are surprised when a server (or network or networks) go down. Meanwhile in reality, it took Dell 3 days to fix a problem on a DOA (dead on arrival) system for something that was bought with a 4 hour support contract.

What are we learning here? Or rather, what should we be learning:

  1. When the service goes down, who is available to help?
  2. When the service goes down, how long does it take for the support/info request to be acknowledged?
  3. When the service goes down, do you know exactly where to go to confirm the issue?
  4. When the service goes down, do you get an ETA of the repair?
  5. When the service goes down, do you get a refund?
  6. When the service goes down, does the company offer a plausible excuse for the outage or does it just shrug its shoulders?
  7. When the service goes down, are you alerted about its recovery when it comes back up?

If you can’t easily answer those questions, you do not have a business solution. You have a best effort solution.

What’s the difference between a business solution and best effort? Well, your business is. Take a look at Sarah Perez’s account when she got locked out of her Gmail and realized there was 0 recourse for her. Woops.

Name calling for fun and profit

Posted: 11:42 pm
February 25, 2008
Gaypile
Comments Off on Name calling for fun and profit

Southpark101_2D771689Yes, I’m bringing the gaypile back. Had to be done. 

Name calling is fun. It helps people quickly identify what they are dealing with to save copy space and convey a message without getting preachy or far too elaborate. If you’ve ever had to put together a flyer you know what I mean, you try to sum up your entire universe of information to get the point across and you still want to capture attention. So you shoot for a compromise.

In professional (well, IT professional) circles, thats welcome. Coder, DBA, helpdesk can all convey a pretty clear meaning, sometimes positive and sometimes a derogative (“Oh, there goes Mr. MCSE”); And for the most part people welcome it and identify themselves as such.

In IT business, it goes the other way. We have a few dozen acronyms that we use to identify partners and opportunities. But sometimes people try their hardest to identify themselves to you. How often do you hear a pitch and just sum it up to the person to acknowledge you have received their bullshit and know what they do so they will cease the pitch? Some people can’t break out of that cycle. Some people are so sold on their own bs that they cannot escape it, even among the people that do the very same thing:

Hi, my name is Bob and I’m a trusted advisor. I do not try to make profit on products, I simply recommend whats best for you and…

Yeah, Bob, we get it, you’re full of shit. You don’t mark up the services you don’t consider material to your business. Microsoft licensing – $0 markup. Antispam – $0 markup. Offsite backups – $0 markup. You’re a great guy, thank you for recommending this to me – oh whats this $1,500 a month fee? Oh, for the managed services you provide on top of a $100 MSP software platform, $50 management agents and a $200 outsourced helpdesk? Whats that, a 500% markup, Bob?

People are trusted advisors when they are not directly trying to milk the highest possible markup from the customer. For everything else, they are trying to charge the highest possible rate that the market will handle without leaving them with less than 40 billable hours a week. That is called a business. Even if you run it by yourself in your underware 28 hours a week, you’re a business. You charge money for time, you are a vendor.

There is no shame in running a successful business. There is plenty of shame in running an unsuccessful one. Which one would you rather be? Pretty title with no substance or a success with a less glorified name?

-Vlad Mazek, MCSE, MVP, CEO. Decode that, b….

Whom do you trust?

Posted: 12:27 am
February 25, 2008
Vladville
2 Comments

So I spiked a little fire yesterday when I mentioned I don’t talk to the press. It also reveals why you rarely see quotes from the truly successful IT firm leaders and why a specific set of people seems to be quoted over and over (hint: it’s not because they matter). How does it work?

Successful people tend to be far too focused on their work to waste time on an interview for a chance of winning a surplus gadget of the week.

About two weeks ago a very polite person called me and when I informed her that I do not talk to the press she explained that it was a peer survey, that it helps companies I deal with, that blah blah blah, basically she hit me in the soft spot since I make my business on features designed through feedback. So I figured, ok, let me help, sure let’s go through the survey. Fast forward 28 minutes later, we had completed section 1of 18 – and I politely told her that I have to get back to work as that was going to be my last day at this job before I join a Christian mission excavating a vulcano in the phillipines.

Truth is, anyone even remotely successful is successful because they are on mission and have organized their day not to be interruption driven. 

So who gets quoted? The guy that was unemployed at the time the reporter went down the list and hit the first live body that had read competing publications coverage of the event. Is that really someone that ought to be providing an opinion and be quoted as an authorative source – someone who got the attention solely because they were motivated to give it because they were incentivised by a USB 1.1 switch?

So…?

So the truth is that everyone is aware of the above – which is why the request for any survey gets filled out by exactly 3 people unless it is accompanied by a free tshirt. And the sad thing is, people still buy into the hypothesis of the press coverage supported by an unemployed consultant!

Thats not how you get a clue about whats really going on out there folks. You do it by forming relationships with people, important people, over a period of time, so that when you have something you want to bounce off them they will take your call from the bleachers of their sons basketball game or from under their Harley. People that don’t talk for free, people that charge for their time.

So in case you’ve been thinking that the great way to optimize your day is in following trade press and ignoring your peers and influential business leaders…. think again.

Opinion on SBS 2008, Essential Business Server and SMB computing in 2008 and beyond..

Posted: 6:41 pm
February 23, 2008
Microsoft
7 Comments

For about two weeks I have been dodging press, peers and clients that wanted to find out about 2008 and how it is going to fit into their future plans. I was quite anxious to see just what my peers would be saying publicly, considering that under NDA we had insight to this for quite some time. So here it is, in a nutshell:

Later in 2008, Microsoft is going to begin to answer the small business technology problems that were posed in 2005. How do I do more with less, how do I consolidate my operations, how do I make my network more manageable and….

And well, EBS and SBS 2008 would have been great in 2005. In 2008, they are a throwback to the way computing used to be done in SMB and is statistically no longer the case for the majority of startups, growing companies and even the bottom tier of the midmarket that has been overwhelmed with complexity.

Sure Microsoft will try to claim that the existing install base is its biggest enemy in getting 2008 into the shop but the reality of the situation is that the world has changed a lot since 2005 and the problems we had back then have largely been solved either by third party software (thereby locking down the deployment in 2003 with no easy/cheap migration) or moving to the cloud.

Few years ago, moving to the cloud was something that was very much frowned upon by my SMB peers. Nobody could quite “get” the concept that there is no need for a local server if all the customer wanted was Exchange and SharePoint. People used to beat me up from all angles on the concepts of remote storage, offsite Intranets, offsite servers, etc.

But guess what… in 2007 and 2008 we have been retiring servers worldwide as people moved to the crowd. Which people? IT people in SMB, IT consultants, everyone that had far too much complexity, downtime, multiple offices, etc. Why do people move to the cloud? Because they are too busy managing internal applications and business processes and don’t want to waste time on the  overrun Exchange, document and file servers.

Really, the argument comes down to whether it is cheaper for you to build your own network, or get 99.999% uptime on someone elses. And much like you don’t happen to have your own cell phone tower and pay upwards of $50-60 a month for the phone plan, SMB has no beef spending half that and getting the rest of the  stuff delivered as a predictable service. It is almost impossible to defeat a service pitch in the SMB because  the internal (replacement) solution requires a nasty (expensive) migration and it is just a problem that keeps on growing so it can be done yet again in 3 years. More people coming out of schools powered by Google, more people are trading off their infrastructure budget for the specialized application powering their business and they are moving on.

So where does 2008 fit? Well, it fits in the shops of the “old school” ignorant admins who feel that if they can “see” the server, its secure. The same that turn off Windows Updates to improve reliability. It fits in the traditional high-bandwidth offices that need to move around large files. It fits in the shops that have specialized applications that require an onsite server.

Where does SBS and EBS fit? I’ll let you know when I find them. People that needed WSS 3.0 already installed it. People that needed Exchange 2007, already got it in the cloud for less than $10 a month. People that needed SQL 2005 for high bandwidth, high transaction applications – yup, the security of a data center. People that needed a better way to manage multiple servers… They got an MSP or a more competent IT guy.

While there certainly will be a market for SBS 2008 and EBS, it is no longer the most demanded option in SMB and is slated for a decline and further losses to the cloud and the MSPs (who for all intents and purposes are part of the cloud). And when something is no longer the most demanded solution, it starts a decline, it grows and empowers the competitors and the process accelerates. I don’t know about you, but I don’t plan to be around in a market of declining opportunities. There is an opportunity… for someone.

Futility of Incompetence (how Monkey is made)

Posted: 9:54 am
February 22, 2008
Shockey Monkey, Vladville
3 Comments

I have been fortunate enough to build a big enough business that calls for a lot of decisions on a daily basis – so statistically speaking, I get to deal with a lot of incompetence. I have gotten so good at it that I am no longer afraid of failing, and even have that little sixth sense when I can feel my ass getting bit as I’m making a call. Most failures are of the “Ah crap. Oh well.” nature, learn the lesson, hope not to make it again, move on..

Then there are those failures where you don’t get to shake it off and move on. The kind that grab you by the back of your head and rub your face in it. The kind that make you deal with your failure, all day long, while making you shake your head in slight disbelief at just how magnificently you suck. The kind that when you turn to your friends for some comfort they not only laugh at you, but also tell you all the other bad sides of your failure that you haven’t even considered. Such is the story of Thursday in Vladville.

This entire week I have been deploying Shockey Monkey 2.0 into a sandbox environment for the purpose of both testing and further development. It gives me a chance to write the deployment scripts and scenarios and also go through all the #VLAD code segments where I left a note for myself to spend more time on. Maybe I am alone at this, but I am far from a linear programmer, when things are going well I just keep on cranking it out and leaving breadcrumbs in the areas that I need to visit back later. Mostly complex validation, bounds checking, fixin’ the stupid user, etc.

Yesterday I decided to take the entire day to improve the UI a little. The biggest reason for the lack of management software adoption in business is the visual design. If a user has to click on 50 links in order to enter time, or if they have to dive through 6 tabs, two popup windows, three inline divs and one link to just say they took 2 minutes to call the vendor.. guess what, they aren’t going to do it! Now, some say that if you want a paycheck you’ll do it but threatening users to use the software or not get paid is not a good company morale move. Truth is, the software sucks. For the love of god, one of the two solutions that loosely compete with Shockey Monkey are either so poorly designed in both form and function that they require a friggin University to figure out. The other one follows the abandoned concepts of each era of UI design: never be afraid to pop up a window, whenever possible stick crap into a frame and back it with ActiveX so it only works in one browser (poorly) and lace the interface with more stars and input field forms than a warez site trying to get you to click on as much porn as possible before it takes you to the actual download. I thought I could do better than that.

Boy was I wrong. Yesterday I decided to improve my time entry UI. Here is a tiny screencap:

smentry

Now the first obstacle is trying to make sure that the time format is recognizable by the people that don’t follow the AM/PM system. Since the monkey is famous worldwide, this was more than important. So I ping one of my partners:

Vlad Mazek:  ping
Ian Watkins: pong
Vlad Mazek:  question, if you got a moment
Ian Watkins: Sure
Vlad Mazek:  when you’re entering time, does it take the 24 hour military time or the american am/pm time?
Ian Watkins: Both
Vlad Mazek:  you use 24hr, rght?
Ian Watkins: Can sometimes get confused
Ian Watkins: Sometimes 🙂
Ian Watkins: But we tend to use it if we need to be specific
Vlad Mazek:  well, you’ve been pretty much useless to me today, Ian 🙂

Now here is what the above actually means, in my mind: you can’t count on the user to follow the form, so bind onChange and a few other events so when they enter 23 instead of 11, deduct 12 from their total and flip the am/pm select box to the proper index. This isn’t hard, it’s just a giant pain in the butt. Fast forward a few hours, I try to seek some positive reinforcement from my peers:

Rich Walkup:  oh i’ve done all that before – trust me – i used to work on a biometric payroll system with a computer timeclock
Rich Walkup:  what really sucks is …..
Rich Walkup:  when shit starts at 11:30 and goes over the DST
Vlad Mazek:   and ends at 2:00? 🙂
Rich Walkup:  sorry to throw a wrench into your system brother but….

Visions of someone starting a time entry at 1AM on a DST day and ending it at 2 AM on a DST day fill my head with livid rage, because I know someone will do this and of course they get paid $20 million an hour and this could bankrupt them so they demand free crap. Note to self: Add an address input field to the signup form so if/when this happens I can go kill the bastard.

You see, in college I used to teach a CS course. I remember saying things like: “I understand that these fundamentals are hard for some of you to do because they are time consuming, but they help those who have never programmed before learn how the data is processed, how to do proper error checking, how to write good software.” and I would generally murmor something about how by the time they are writing any commercial code these things would be built into the IDE anyhow and will just require a drag and drop of a time control.

Yesterday I spent all day writing time validation functions for Shockey Monkey. My professional development career has effectively been reduced to keeping stupid people from trying to bill 26 hours in a 24 hour day.

FMR.