It’s been a tough few weeks at OWN as I think we have become the “it girl” at the same time as the new product releases and upgrades are hitting the market. Although we are turning the corner and things are starting to improve dramatically I figured I’d write this if for no other reason than to share a failure, let it either be a reminder or mortality or graffiti for others that will run into the same issues down the road.

Just like I beat up on Microsoft for making me look bad, I have a ton of partners that beat up on me when we start to suck. I’ve always had an open door policy and people both email and call me directly even when there are issues that should be handled by the few dozen other people in the organization whose job it is to handle those issues. Why? I feel that there comes a point where an executive needs to talk to an executive in another company as the last ditch effort to make things work. So if things look bad, regardless of if they are over $100 or $100,000, I need to know. We track the crap out of the client support and purchasing behavior, but I assume there are at least 10x to 100x problems as we get reports of, which is why they are all at least considered.

So where did I lost da ball game as they say? Well, we announced and launched four products in the space of a month. Never gonna do something that stupid again, I can tell you that much. Because here is what happens – no matter how hard you work and test and plan and prepare, something always fails.

It’s not something uber-critical, it can be something that just annoys people. The problem with small mistakes is that when they annoy the user the perception of your product and service changes from positive to a negative. The worst thing you can do as a service organization is make your users unhappy with your service. Once they are unhappy they will report every issue they face, be it your fault or not, because the last time they went through their checklist something was your fault so this time they bring it to you first.

This starts a cascade effect of a flooded helpdesk. A flooded helpdesk takes support personnel, which is the frontline of all product issues, off their game and makes them the fault monkeys. Fault monkeys spend all day and night apologizing for inconveniences and falling over one another to patch the holes as soon as they are reported but they don’t have an hour or two to take the problem back to the analyst to identify the issues, forward to the programmer or ultimately me to fix them from actually storming the helpdesk.

This creates a further stress on account managers, infrastructure managers and other people in the organization whose job it is to track metrics and the behavior of the system. How? Well, the support monkey keeps track of problems which he takes to the analyst at the end of the shift. When you bring issues one by one they can be fixed one by one, consulted, peer-reviewed, etc. When you bring 20 at 5 minutes before quit time the bottleneck just shifts.

The epic failure in the entire chain of command is when the bugfix introduces more bugs because the fix had not been tested to the full extent and the loop goes back twice the intensity.

What am I doing? Well, talking to partners, making sure the process sticks even if we’re shorthanded, apologizing, even working on support tickets and trying to reach around to the programmers (yeah, believe me, the experience is just as it sounds) so that we correct the issues.

We have been doing something monumental over the past 2-3 days to fix all of this and we’re on schedule to have everything together by the end of the weekend. But that introduces yet another problem that is a little difficult to fix. First, we need to grow rapidly, but the skills of people in the marketplace just stink. This means that training can take a little longer to bring online and we get to train people during graveyard when things are a little slower.

The more important problem, which I hope every service organization reading this is taking to heart and pounding into the paper, is that you get a chance to fail the client exactly once. Yes you can make them smile for a minute with a comp or a gift card or an apology, but once you fuck up you’re fucked. Sorry about the colorful language there, truth of the mater is that once your client has an ounce of doubt into your ability to deliver the service you have promised – even if you’ve stood by your SLA and offered refunds and made the client whole – you are now on the defense and will be thought as the first point of failure because the confidence of your client has been shaken.

It’s tough, it’s ugly and this is why IT is a business and not a lifestyle. You have to be sharp, you have to be good and you have to be on it 100% of the time or you may as well not even bother. And lord help you when you fuck up, because that 40 hour week is something you won’t even be able to dream about because you’ll be working around the clock.

Love it or leave it…

You can leave a response, or trackback from your own site.