I thought I was getting good at my job. No significant loss of productivity company wide in the 3 years I've been in charge. No loss of data. No virus infections. And for the last 2+ years since I got most of it set up, no all-nighters.
I have 3 VMWare servers, any 2 of which have the horsepower to handle all the load. Theory goes that if a machine fails I can just shuffle the VMs and off we go.
One of the servers failed at 4:30PM yesterday.
Apparently one of the remaining servers, even though it has plenty of horsepower, goes unstable and becomes unresponsive under this higher load, even though it was fine with it's normal load. And the server that is stable regardless doesn't have the guts to handle all the excess load on it's own.
So here I am, 4:30AM, building a new ESX machine to deal with it. I need to get smarter before I get too much older, this is much harder than it used to be.
*this isn't a tech support question, just a rant. No need to try to help, I have it covered, thanks.