Some of the new software we are building at Own Web Now manages it’s own password complexity, sometimes much to the chagrin of the default policies built into Windows Server 2008. You’ve heard about Security By Obscurity, so get ready for the new model: Security by presenting GPOs where you would expect to see them, just disabled and uneditable, forcing you to go modify them in a completely different place - Security By Ambiguity. Where does one modify the local security policy in Windows Server 2008?

Local Security Policy used to be managed through Administrative Tools >Local Security Policy. Things like minimum and maximum password age, minimum length, complexity and so on were tweakable under that console. In Windows Server 2008, those screens are still there but you have no way to edit them:

So, how does one disable all this stuff in Windows Server 2008 because the external application is intended to manage it (and you presumably do not want your policies to break because they override some of Microsoft’s?):

Start > Run > gpmc.msc

This is the Group Policy Management Editor, nifty tool that used to be optional with Windows Server 2003 and XP (free download) is now the way to manage your security policies. 

Computer Configuration > Policies > Windows Settings > Security Settings > Account Policies

Warning: The security policy outlined above is pretty much suicidal if you don’t enforce password policies through a different tool. Here is a brief description of the Microsoft password policy requirements.

So why did we disable it? Because we wrote our own software to manage the policies, which has the same complexity as Microsoft’s recommendations, but we found that Microsoft will at times even deem it’s own default password policy not to be strong enough, introducing inconsistencies that we were not willing to risk support expenses to narrow down.

Another bad security lesson brought to you by Vladville.

Son, you have to earn the pimp hat. I am not sure how young you have to be to start moving software but Tim is getting an early start: effective immediately Own Web Now in partnership with Scorpion Software is bringing you safer small business computing.

One person at a time.

At $20 a month.

Complex? See below.

Game changer? You bet. Where do you sign up? Here.

Passwords are the most frustrating thing for system administrators and MSPs to manage. Make them too easy and you get hacked. Make them too difficult and the staff complains, forgets their password and opens trouble tickets which eats into overall profitability and security of the organization. We can fix that!

With Scorpion Software AuthAnvil and one time password authentication you are issued a token, a keychain, with a constantly rotating short password. When you try to login into your organization you are prompted for your password but you are now also prompted for another password which you copy from the security key. This allows clients to have simple passwords and passphrases and never worry about being hacked with a password that was compromised, guessed, keylogged on a kiosk or sniffed over the air. For $20/month per employee you have one less thing to worry about!

We are partnering with Scorpion Software to make this affordable.

We are doing it one person at a time. Yes, just one. Got a small client where only one or two people travel and need additional security for SBS Remote Web Workplace? You can get it one at a time!

We are integrating this into Shockey Monkey and ExchangeDefender.  We are working with Scorpion Software on integrating this into Exchange 2007 and will have discounts for the companies that subscribe to the above services.

Most importantly, we are making this EASY. We will configure the service for you. We will install and deploy the configuration. You can use it on any server or as many servers as you want to. Going forward, you will have ability to use the same technology with other popular services including some you will see on this very blog

I told you we’re changing the game. OWN is here for you, what are you waiting for?

… and they will build a better idiot.

Honestly, there are days when I feel like people do things with my software just to dick with me. For years we didn’t enforce password complexity on ExchangeDefender, ergo 99.999% of the passwords at ExchangeDefender are “password”, “Password”, “Password1″, “P@55w0rd“, “P@ssw0rd“, “password123″, “1qa2ws” or “2ws3ed” and can be cracked by a three year old.

So I set out to write a password complexity procedure the other day, enforcing the standard MCSE complexity: at least 7 in length with the mix of three of the four options: lowercase char, uppercase char, integer, special character. So today I power on my laptop to see just how complex the passwords being generated are, thinking that people are starting to use passphrases, etc.

What I found made me scream out “Fuuuuuuuuuuuuuuuccckkk me, WHY!?$!@#” out loud. The password?

Abcd1234

Why, for the love of god, would anyone do that? Did they just look at the new password complexity alert and thought…

“Screw Vlad. What is the LEAST complex password I can come up with given the current restraints. Let’s see… how do I make this work… something everyone uses…. something sequential… start it at the beginning of that sequence. Oh, and I’ll capitalize the A for take that complexity Vlad, you Asshole.”

Thanks… whoever that was (anonymously logged)… I think my next method will check for password complexity and instead of throwing an alert or UAC or any of that annoying stuff that they can live with I will shoot back a 4096/1024bit key as their new password. “Dear Customer, The password you requested did not meet our password complexity requirements. Here is a 4096 char password to use from now on.”

Or sell them AuthAnvil….

I just don’t get it. Why spend all this money on perimeter security and protect it with a sequential password that can be guessed by a 3 year old?

Looks like there was more fun overnight than the Central Florida hurricanes, check out what OWN was up to:

For the past 10 hours or so we have been handling an 820% surge in reboot requests for hung Microsoft servers after applying the latest security patches. Our managed network of Windows 2003 servers has not been affected but a huge portion of our network apparently has, please be advised.

If your Windows Server becomes inaccessible as a result of the latest patches please open a ticket request and mark it as urgent. You will not be charged for the support request and your reboot will be handled with the highest priority. We have an additional shift on hand in all data centers to help you through this network event.

Ouch. I dig the quantified percentage surge breakdown, and they bitch about Shockey Monkey reporting

To be honest, I am quite happy with Microsoft patch quality as of late. For months now you could reliably install a patch and not think about it twice about not rebooting properly, hanging the system, bluescreening or worse. It has been pretty much as rock solid as patching gets.. Do I wish that was the case before a multimillion dollar investment to provide every server on the network with a remote reboot switch? Yeah, little bitter about that, but it has saved more money in SLA refunds that we were cutting because we stood behind the reliability of Microsoft software. I’m pretty content with Microsoft, even given the events.

I did a little bit of digging. The freezes/hangs were not associated with the systems that were on a regular patch schedule - so the systems that were managed by windows update server or regularly scheduled windows update were not the problem. The systems where people did their own patches and manually did Microsoft Update in the browser… poof. I guess that will make Susan feel a little better because her experience has been exactly opposite in the past few months - manual updates fine, managed updates causing problems.

The big picture…

I spent the better part of 2000’s bitching and moaning about Microsoft patches, cutting checks when they failed, escalating them through Microsoft and feeling like a complete douchebag when I asked my PSS friends a support question at 1 AM. Nobody likes being picked on for something that is not their problem. And to give Microsoft credit, the patch situation has greatly improved. 

Over the last few days there has been a little conversation going around the general displeasure with Microsoft, you know, things just not working up to the expectations. I shared a few details on the thread about how OWN faced the same issues and how we overcame them and why we were able to overcome them. The message was largely ignored, pushed forward by other complaints and issues until multiple people basically leveled that they don’t have the time to sit around and complain but a business to run.

Therein lies the problem. There are companies that are service focused and driven by the customer feedback. Then there are companies that are product focused and believe that the resolution to issues is an upgrade. I believe that Microsoft and I sit on the polar opposite ends of that world, one does everything it can to reassure the customer base that the fix is on its way, the other tries to fix the problem right away and hopes the customer doesn’t bash them as ignorant.

Customers see and recognize this as well. When everyone is complaining about an issue out loud and nothing is done about it, people stop complaining. For the most part, the peer system is very much a collaborative sanity check for an IT person - ok, now that we all agree its broken let’s just wait for a fix - but what happens when the problem is in licensing? Or product implementation? Or partner program benefits?

Well, you get quiet. You sit on your hands, you look around, you cough, and then you realize you got all this shit that needs to get done and you focus on your own problems and try to minimize the impact that Microsoft has on you. Whether you stop patching completely or invest millions of dollars in reboot switches, you find your solution and you move on.

MVPs want people to keep on complaining, to keep on escalating issues, to keep calling PSS, to keep flooding the newsgroups… because they care about the product and want to see it improve. They need the ammo to say that a product sucks and ask for changes. And more power to the MVPs.

Business owners have better shit to do, after multiple complaints and no response they get the message, they believe that if Microsoft was truly concerned about the problems they would work on fixing them, that it’s Microsoft’s job to fix the problems not theirs to continue to complain and that if Microsoft was truly interested they would do something about all their previous issues.

Although I’m an MVP, I am on the business owners side of the argument - If a company is truly concerned about their problems they would be more proactive in addressing, admitting and processing them, they wouldn’t be reactionary to the “filtered up” complaints of people who haven’t become completely disaffected yet.

[ __________ insert link where Susan beats me up over what I just said ]

(Note: All Windows Server post-SP2 issues will be refered to as Cousin Joel edition)

Microsoft released an out-of-band patch yesterday to cover the recent .ani/.cur exploits. These are causing confirmed issues with Windows Server 2003 SP2, SBS 2003 SP2, Windows XP, etc.

Here is something from Susan.

In short, you might need another reboot this month. Some have reported that restarting the system manually (Start, Shutdown) does not produce the same result as Reboot Now prompt after the patch installation. I have no scientific data to back that one up but if it works for you any differently please let me know.

Followup comment:

Susan Bradley says:
  btw
Susan Bradley says:
  change that to start, shutdown and ensure you select reboot
Susan Bradley says:
  otherwise on a remote machine you just turned it off 

Ok, if you’re reading this post and needed to see that disclaimer.. please.. do not come back to Vladville. It’s only a matter of time till you find something sharp laying around here and you hurt yourself. That big loud box in the “computer room” is a wine cooler, do not touch it, CEO has 1882 Merlot in it, if you break it you’ll be fired.

Much has been said, and ridiculed, about the UAC feature in Microsoft Vista. Aside from 2 security MVPs and a handful of Microsoft employees I have not met anyone else that either likes this feature or is not asking to have it removed.

Susan and Dana are on the pro side. The entire world seems to be against them.

Some background: UAC comes out of the years and years of no easy process priviledge escalation control in Microsoft Windows. While working on a Workstation in an Administrator mode (default for XP and below) any process can without permission modify the registry, install drivers, change system configuration, etc. This lack of control is the main contributor to the spread of viruses, spyware and rootkits on the Windows platform and perhaps the main reason why Windows gets the “insecure” label so easilly.

So after Microsoft tried and failed to conquer the anti-spyware market, after they tried and failed to conquer the anti-virus market, they decided to actually fix the problem. (ok, truth is these efforts were being made in parallel but lying makes it sound better) So how does one fix the problem? By lifting the technology from the people that have already solved it! So they dug up their Linux guy from the basement, chained him to the steering wheel, drove him through the mobile car wash a few times and then sat him at the table to explain su and sudo.

Long story short, they stole sudo (superuser do) technology that allows a regular user to escalate to the superuser priviledge to execute a single command as the Administrator. They wrote a wrapper so that every time a process requested a restricted object (install a driver, manage users, etc) the little window will pop up with Windows needs your permission to continue. And as they were stuffing their Linux guy back into the box of manure he said something about su but they were already well on their way to kicking him down the stairs back into the basement.

The ONLY thing thats wrong with UAC

Microsoft half-assed this big time. UAC does not, and likely will not, piss off home users during their regular computer use. However, during provisioning and system troubleshooting UAC becomes a total nightmare. 

All the Microsoft OEM partners know this – which is why they ship boxes with UAC disabled. No why, oh why, do they do this? Because they know that the first thing you’ll do with your shiny new system is to make it your own – add users, add hardware, install software, etc. Be prepared to approve the UAC half a dozen times. OEM’s know this, and they don’t want the support overhead. Thats why they ship boxes without UAC.

The troubleshooters and computer techs? They have no choice but to shut the annoying thing off? Why? Because Microsoft half-assed it. There has to be a way to permanently escalate priviledges to the superuser status while troubleshooting the system; The unix su equivalent – For example – I am installing a video card, yes, I know I am going to be escalating the priviledges over and over and over again until I tweak it completely so STFU and let me work. But no, you have to click over and over and over again. Was it so hard to allow a checkmark to not ask for permission during the next 5 minutes, or never to ask for permission when performing this kind of task or to just accept all access during the entire session? Apparently, it was.

Disabling UAC

I’ve held onto publishing this for a while but since everyone wants to point at the users as the problem here and not a Microsoft shortsightedness, let’s give this a spin shall we? If you need superuser priviledges during a session to perform a hardware maintenance, software installation or troubleshooting you should not be subject to Vista’s inability to cope with this. To disable UAC follow these steps:

Start> msconfig >  Tools > Disable UAC > Launch

Reboot and you can actually perform a maintenance task without being nagged to death. Once you’re done go back and Enable UAC the exact same way, reboot and you’re back to normal.

Microsoft: Get your head out your ass and recognize the ITPRO should not sit around the box and approve escalation more than two times to perform a hardware or system maintenance task. By the same token, don’t think that just because someone is an ITPRO they need permanent priviledge escallation. Just provide a way to put the system into the maintenance mode – more than two prompts are too many, the second permission ought to be smart enough to ask whether this escalation should be assumed for a few minutes or remainder of the session.  

Don’t worry, this isn’t The Susan Bradley ™ rant. This is just something for you to think about on a slow Friday afternoon while you’re planning your weekend and likely using the same password for your desktop as you do for Travelocity and digg.com.

I’ll cut to the chase. In over a decade of professional system administration I’ve been given so many passwords and told “don’t remember this” about a million times. Now I honestly try not to think about the password as people give it to me – it’s a skill of not caring, specific to system administrators. However, I’ve had a pleasure of dealing with some people over and over and over again and every time they call in to setup a new account they have me use their “standard” password. Then they invite me to help via RDP or LogMeIn, and yep, same password. Send a zip file over – yup, same beast. Why? Convenience. (holding back the rant… holding)

I can understand the convenience. First off, you don’t have to remember the password. Second off, you don’t have to think of a new password every time. But what if one site suddenly required a more complex password? Now you have to keep track of two. Then when you go to site A and your password doesn’t work? Hrm.. maybe its the one from site B. Thirty seconds later you’ve blown through your entire password assortment and just gave away that shiny new porn site the password to your banking account. In the day and age where all usernames happen to be email addresses, for the most part, this can be dangerous. And it is definitely not convenient any more.

I have been using a free program called Keepass for years. It is safe, completely open source and very convenient. When I go to a new site and need a password I don’t sit around thinking of a permutation or something including the site name. I have Keepass randomly generate a 16 char string. It picks the complexity. Forget about web sites supporting passphrases, their database is more likely to get stolen (or lost) than your password cracked. When I want to login there are plugins (on screen keyboard, automatic form fillers) to automatically let me in without copying and pasting. The password database is encrypted, portable and passwords are masked (*** instead of ABC) so even if your employees / bosses are walking around behind you they will not be able to see what you’re typing in on the screen. So give it a shot… It’s Friday, you ain’t got no job… you ain’t got stuff to do.

Lite it up. Did I mention it’s free? Pass it on… but if you’re really having a slow day blog about a single tool you use that saves you a lot of grief.

Sometimes reading inbox spam pays off. Today I got the following interesting offer in my inbox from RSA Security, Inc:

Download the Data Protection Strategy Kit and learn how protecting data across the entire enterprise - applications, databases, storage, etc. - can solve business-critical security issues.

How to Meet Your Customers' Security Requirements
Information Kit at a Glance

• Regulatory mandates
  Comply with PCI, FFIEC, ISO 17799, CobiT, etc.
• Corporate policies
  Avoid, among other things, public reporting requirements of U.S. State Breach Notification laws
• Customer requirements
  Ensure that customers are confident in your ability to protect their sensitive information

Data Protection Strategy Kit
At a Glance

• Forrester Trends Report
  Secure the Data, Not Just the Underlying Infrastructure
• Current U.S. Regulatory Climate Report
  Review of Key Regulations and What They Mean to You
• Protecting Payment Card Information
  Solutions for meeting Payment Card Industry and U.S. State Breach Notification Law Requirements
• Technology Backgrounder
  Managing the Life Cycle of Encryption

Download Data Protection Privacy Kit today.

Whats so great about that? Well, one of the ways major corporations (RSA is a biggie) sometimes attract interest in their products by giving away tons of research. More often than not, all these reports and data are written to leave you with a feeling that YOU MUST BUY OUR PRODUCT OR DIE but if you’re smart you can get a lot of very insightful and very expensive information that you otherwise would never get to see.

Advice? Sign up for a gmail.com account and take a look at these reports, at least an executive summary. The more you know, right?

Earlier today I posted a question on a mailing list trying to find out how other IT Solution Providers are dealing with the increasingly unreliabile and costly Microsoft Security patches.

Please don’t turn this into a security issue because it’s a business question:
 
I am depressed with Microsoft patching to the point that I might have to drop my SLA against all Windows-based servers at Own Web Now. Even  on a day when the patch does not cause any problems at all the reboots don’t happen as they should. Vanilla configurations just do not start all services. Make up a weirdest thing you can get a Windows server to do and we’ve seen it. Remember that this is on a good day, not on a bad day when the security patch locks out Blackberries one month, Macintosh the next, crashes Dell boxes the month after that.
 
I am considering automatically dropping all Windows servers into an automatic 8 hour maintenance cycle during the Microsoft patchday to compensate for Microsoft’s lack of QA. We can no longer minimize issues through testing because even identical boxes (Hardware and software, remember we virtualize the crap out of things) are not behaving the same. Reboots before the patch are fine, reboots after the patch.. poof.
 
How is everyone else handling this? Drop the SLA? Lower confidence in Microsoft (who does that help?) Extended maintenance cycle?
 
Second Tuesday of the month is becoming a religious holiday at Vladville…

The Process

Our process and our ingredients are pretty simple. We do a flash backup every Tuesday afternoon (EST). Those backups are generally complete by 10PM. We do a flash reboot just to make sure there are no hardware/software issues. We proceed with the patches that passed quality control / quality analysis earlier that day. We push using a collection of tools, WSUS and other bits and pieces. Other bits and pieces are used instead of WSUS when we want to apply hotfixes without a reboot to critical infrastructure systems.

Either way, pretty standard stuff. Most Windows servers run a similar configuration (actually, most are identical in both software and hardware as they are mostly Virtual Server systems) so there is little reason to expect one to work while the others fail. 

The Costs

Do not let Microsoft WSUS and “Secure by Default, Design, Description…” fool you, patching is expensive, very expensive. There is no alternative to patching, we have to do it. With critical updates, we have to do it ASAP. No complaints there though, its just a part of business.

My complaint is with the unplanned costs related to patching. Costs that I and my customers have to pay because Microsoft produces unreliable and unstable patches. Let me explain what my definition of that is: “If a patch causes unexpected downtime or adversely impacts my system performance I do not consider it to be stable or reliable.” Simple as that. A patch is supposed to close a security hole in the software without affecting the rest of the system.

This is no longer the case. Few months ago Microsoft patch knocked out Macintosh systems (Entourage) from connecting to Exchange. Month after that it stopped Blackberry from operating properly. You remember my post about it regarding Dell.

My actual complaint is that I am at the verge of losing confidence in Microsoft’s ability to reliably and predictably patch the problems in their software. It is costing me a small fortune both financially and in terms of reputation. If I cannot stand behind my SLA (Service Level Agreement) which states just how often the server will be up then what value am I providing. If I am put in the position of having to appologize for things that are not my fault to begin with, where does that put my reputation at with my customers? Forget about the cost of overtime for employees, support calls, graveyard shifts, and the near cottage industry built around the patching tools, preparation process, reporting and followup just to make sure that the software we paid for continues to behave the way it was sold to us.

Forget about me

Now this is simply a blog post that will change… nothing. But it is an opportunity to review your SLA and consider how you deal with unreliable partners whose products and services you are supporting. I am at the verge of having to rewrite my SLA to put Microsoft patches into a maintenance cycle without any assurance on the time period. Here is one of the intriguing answers I got:

Vlad, we ran into the same issues as we started to scale and eventually had to build a lab for testing where, once approved, the patches would be put on our corporate network and when approved, we would roll them out to the clients. To resolve the reboot problems we put in “lights out” cards in all our servers. I agree it is not for the faint of heart.

Anyhow, something to consider…

Dear partners, please stand closer together so I can get you all in the same shot. No, thats not the camera I'm holding.

Just wanted to alert you to a few things that have been going on at Casa de Vlad in our 11 data centers. We're having issues over a large portion of our Dell server base that has SCSI/PERC4 controllers installed and/or USB 2.0 drives installed. The behavior is as follows:

1. System powers on and allows OS selection. Windows 2003 Server splash screen comes up, few seconds later black screen or reboot.
2. When booted up using Safe Mode the system hangs on loading acpitabl.dat

There are dozens of solutions of which we have identified the following so far:

1. Unplug the USB 2.0 drive. This worked on just a few systems and allowed the boot to proceed. On reboot it powered on as well. Will test again later.
2. Clear SCSI controller card configuration and rescan the bus. This worked on a few as well.
3. Power system down and perform a full fs check. Windows PE is your friend.
4. At this moment we have over 200 systems that are going through a manual repair using Windows Recovery Console. Hope you set one up. The process of doing so is to replace update.sys file with the one from Windows 2003 RTM media. Generally located in %systemroot%\system32\drivers 

We are seeing the above on Dell PowerEdge 800, 1800, 1425 systems only running the Standard version of Microsoft Windows 2003 SP1. Identical systems running Windows 2003 Server Enterprise Edition, Web Edition, Small Business Server and R2 editions of Standard & Enterprise do not seem to be affected. Will update as we go along. Working closely with Microsoft and Dell to recover, sorry for the inconvenience.

Comments closed, pings allowed. Believe me, I know what you're going to say "Micro$oft security blows, switch to Linux, switch to Mac, switch to Amiga, Ballmer is evil, this wouldn't have happened on an HP…" There, I saved you 20 minutes.