BSOD - Directory Services could not start

[time]

Getting this on a Windows Server 2008 R2 box set up to be a PDC. It has a separate boot drive and the domain files are stored on a 4-drive RAID-5. The RAID-5 dropped a drive and apparently went to critical status after the drives were removed for testing.

On boot it throws BSOD C00002E2 (Directory Services could not start) or just restarts after the Windows logo loading screen.

My uninformed *guess* is that a good drive was pulled during diagnostics and that caused the array to go critical. Windows couldn't see its domain files and has had a heart attack. Sound plausible?

Any way around a complete reinstallation of Windows?

 

[ddrueding]

That sounds completely plausible. I'm sure that a MS guru could get it working without a reinstall, but I'm not one. I'd be doing a re-install.

Was there a reason the OS wasn't on the array as well?

 

[time]

It had an image backup on a removable drive, but booting from that that made no difference to Windows sulking. Is there an extra clue in there somewhere?

 

[Mercutio]

This is a "PDC". Is there another DC someplace and the version numbers are out of agreement now? What server currently has the FSMO roles?

 

[ddrueding]

FSMO roles is what caused me to treat the OS install as critical data.

 

[Mercutio]

As a note, Windows internal backup supports directory service restores in ways that some third party backups do not. If you're going to use third party disk imaging, you should use it in conjunction with regular system state backups (AD is part of system state on DCs).

Methinks time is about to enter the wide world of pain that is Directory Service Restore Mode and dealing with an Authoritative Restore.

 

[ddrueding]

Methinks time is about to enter the wide world of pain that is Directory Service Restore Mode and dealing with an Authoritative Restore.

Yup. This is why I treat the OS and the AD database as one monolithic, inseparable block. To do otherwise is to invite massive pain and suffering.

 

[mubs]

And so, the Gods have spoken...

Are these things learned only the hard way? Seems like a rather painful learning curve.

 

[Mercutio]

I've never had a mission critical DC fail, but I've demonstrated DSRM in the classroom and it's painful enough that I hope I never have to do it for real.

On a single server (the kinds of setups I manage, mostly), it's not THAT bad, because restoring system state amounts to an authoritative restore if there's nothing else tracking the AD version number.

 

[time]

This is a "PDC". Is there another DC someplace and the version numbers are out of agreement now? What server currently has the FSMO roles?

There's no other server per se, just a Windows 7 workstation doing double duty.

Yup. This is why I treat the OS and the AD database as one monolithic, inseparable block. To do otherwise is to invite massive pain and suffering.

Unfortunately, Microsoft insists that you store "FSMO roles" on a separate drive to the boot drive. So they had to be stored on the array. If anyone has a good strategy for this, I'd be pretty happy to hear it.

Methinks time is about to enter the wide world of pain that is Directory Service Restore Mode and dealing with an Authoritative Restore.

Well, not me literally - the site's very remote, remote access is obviously shot and I'm trying to help a visiting tech through the mess. He has already been using 'DSRM', but apart from being able to see the data, it hasn't helped at all.

 

[Mercutio]

It's not so much "insists" as "will put on another drive by default if there is another non-removable drive present." The theory is that there's enough disk I/O from the OS that the AD database should be located on another drive.
The thing to do, if you're uncomfortable with that arrangement - and I am, for the kinds of servers that I deal with - is to build out the system with either a single drive or single volume that will be used for the OS, and add additional storage only after the OS is installed.

But, OK, there are two problems now: The RAID controller is freaking out and AD is hosed.
So the question is, is there a backup of of the RAID anywhere?
If the backup is more than 60 days old (probably more like 45), it's essentially worthless, since the machine account passwords will all be out of sync and all connected clients will have to rejoin the domain anyway, but for this setting it might be small enough that it would be easier to rebuild the domain anyway. Of course, that would completely suck if Exchange or some other AD-dependent service were present, so it's going to be a judgment call on the part of the guy actually doing the work.

Also, and you won't see me say this very often, this crap is exactly the kind of thing that Microsoft's paid support incidents are made for.

 

[time]

The thing to do, if you're uncomfortable with that arrangement - and I am, for the kinds of servers that I deal with - is to build out the system with either a single drive or single volume that will be used for the OS, and add additional storage only after the OS is installed.

Are you certain this will work? Because it's obviously vastly preferable.

There's no additional M$ crapola on this box. So after lifting the data on it, a complete reinstall seems like the best option - particularly if we can move the AD database to the boot volume.

 

[Mercutio]

You can absolutely have a DC with only one physical drive, if that's what you're asking.
Moving the database location with ntdsutil might be an exercise in one's patience on a working install, but if you're starting from scratch it's not really an issue.


If you want to use Windows softRAID 1 for the OS volume, just create the volume with the relevant disks on another Server 2008 machine before you install the drives.

 

[ddrueding]

It's not so much "insists" as "will put on another drive by default if there is another non-removable drive present." The theory is that there's enough disk I/O from the OS that the AD database should be located on another drive.
The thing to do, if you're uncomfortable with that arrangement - and I am, for the kinds of servers that I deal with - is to build out the system with either a single drive or single volume that will be used for the OS, and add additional storage only after the OS is installed.

....

Also, and you won't see me say this very often, this crap is exactly the kind of thing that Microsoft's paid support incidents are made for.

Both of these. If you can't just do a re-install and re-enter the data easily, call MS. And watch the guy who remotes into your system do all kinds of command line magic. I've been on AD-related calls that took 16 hours+, but in the end all was well and it was still just the fixed fee ($300? $500? doesn't matter, totally worth it.)

 

[Mercutio]

My experience is that ntdstil is really poorly documented, or perhaps that the tech experience with it is so diffused that there's no wide consensus on what to do when you need to use it. So those guys dd is referring to have tons of experience dealing with it and the rest of us are just trying to keep from ever having to mess with it.

 

[Howell]

All of my FSMO data is on the system drive across 2003 and 2008 domain controllers; you have a choice of where the NTDS folder will live at promo.

 

[Howell]

Remove AD services from DSRM.

http://technet.microsoft.com/en-us/library/cc731871(v=ws.10).aspx

 

[ddrueding]

Unless your enterprise is massive, you don't need to worry about an IO bottleneck. I've run 500 person AD servers with every FSMO role from a 7200RPM SATA drive.

 

[Mercutio]

You have to remember that a lot of these practices came out of the realities of the early '90s, when Win32 was a new thing and servers were very different beasts than they are now.

 

[Howell]

Now you can load the entire DS database into memory without a thought. Plus, you are sharing the load with your additional DCs.