Exchange Database Woes (Again)

[CougTek]

On our Exchange 2010 server, one of the mailboxes is dangerously close to the limit Exchange allows. It was 979GB before I worked on it to move user accounts to other server mailboxes. I moved ~150GB worth of user accounts. Yet, after I defragmented it (dismount database, then go to CMD and type "ESEUTIL /d <database path & name> /t <temporary database path and name>"), it only shrunk by 20GB or so.

So now, after defragmenting the supposedly leaner mailbox (which took almost 12 hours BTW), I'm still stuck with a 947GB monster. I don't know what to do with this. Microsoft recommends no more than 250GB per server mailbox. Not sure, but I think 1024GB is the upper limit.

And before someone bitches about how I could let things go that far before intervening, I'll just state that it was someone else's duty up to two months ago, when it fell on my desk along with a shitload of other things. My pay hasn't increased since, which I'm pretty pissed about. The management was supposed to hire someone else to replace the guy who left at mid-January, but it became obvious two weeks ago that they wouldn't do it after all and now I don't have a choice but to clean up this mess or it will impact my own users.

 

[ddrueding]

Yup, that is a problem. The last time I was dealing with Exchange I had a mailbox corrupt itself before reaching the official limit, ended up having MS deal with it. They saw the mailbox size (smaller than yours) and admonished me. When I mentioned that I was under the limit they said it was more of a hard upper limit and that I shouldn't be anywhere near it in production. This was the big argument I was able to make to get off Exchange and onto Google. I now have individual users with 75GB+ mailboxes and no performance impact at all.

 

[Howell]

First set limits on the mailboxes so that this doesn't happen again. You can do it by user, database, or environment. Teach the users to save out large attachments to alternative storage and implement a mail archival solution. Or migrate to a cloud host and pay to make it someone else's problem. If they are not going to hire someone you really ought to bring in a consultant to get through this crisis.

I'm actually confused exactly what your numbers refer to. What version of Exchange are you running? Are you talking about mailboxes, or mailbox databases? Do you have spare mailbox databases available, or space to create more databases? It has been a long time since I've done an offline defrag because of the loss of service and I can achieve the same thing by migrating mailboxes.

Are you sure you actually have whitespace to be recovered with a defrag? If you did the offline defrag and you didn't recover much space it would indicate that you do not have whitespace. Check with Get-MailboxDatabase -Status | ft name,databasesize,availablenewmailboxspace -auto

 

[CougTek]

First set limits on the mailboxes so that this doesn't happen again. You can do it by user, database, or environment. Teach the users to save out large attachments to alternative storage and implement a mail archival solution.

In the past, my predecessor tried to enforce quota on mailbox size, but there were always lazy project manager who complained that they didn't have the time to clean up and that they really needed all of their junk to fullfil their project and blablabla. I think I'll enforce my own policies as soon as this issue is fixed. I'm the only one in the entire company who's able to manage this, so it's "obey my command, or go to Hell". Teaching them is a lost cause. They are just too lazy. I can tell my minions (the L1 and L2 tech support) to try again, but it's always been in vain for the past three years.

I'm actually confused exactly what your numbers refer to. What version of Exchange are you running?

2010

Are you talking about mailboxes, or mailbox databases?

Database

Do you have spare mailbox databases available, or space to create more databases?

I've created three additional mailboxes databases. I'll move more accounts in the other databases tonight. But I don't have enough time to retry the database defrag since it takes close to twelve hours.

It has been a long time since I've done an offline defrag because of the loss of service and I can achieve the same thing by migrating mailboxes.

That's probably what I'll end up doing too.

Are you sure you actually have whitespace to be recovered with a defrag? If you did the offline defrag and you didn't recover much space it would indicate that you do not have whitespace. Check with Get-MailboxDatabase -Status | ft name,databasesize,availablenewmailboxspace -auto

Well, the ~150 user accounts I've moved take around 150GB of space on the other three mailbox databases. I expected that it would create just as much whitespace on the original mailboxe database. I guess I was wrong.

I'll try your line in the powershell command prompt and I'll seen what it returns. Thanks.

 

[CougTek]

Hmm... the term Get-MailboxDatabase is not recognized as the name of a cmdlet, function, script file, blablabla...

But when I search it, Google tells me it should work.

I hate Microsoft.

 

[ddrueding]

Setting tiny mailbox sizes and restricting attachment sizes is so last millennium. Outsource your issues until you can get off such a crap platform.

 

[CougTek]

There isn't time to outsource anything, David. I'm days or at most a week from being truly f*cked. Once this is fixed, then I'll I'll think about longer term solutions. Right now, I'm in emergency mode.

 

[Mercutio]

Outsource your issues until you can get off such a crap platform.

To whom? Most places that offer hosted Exchange are on the order of $8 -10/user/month and would probably balk at trying to migrate Coug's mail stores in their current state, especially given the time-sensitive need for a fix. Is Coug's employer really going to be OK with a ~$15,000 annual line-item for email services given the shoestrings on which it operates when they can just shove it all on to Coug for "free?"

 

[Howell]

Hmm... the term Get-MailboxDatabase is not recognized as the name of a cmdlet, function, script file, blablabla...

But when I search it, Google tells me it should work.

I hate Microsoft.

You are probably running "powershell" not "Exchange management shell" which has the exchange powershell modules pre-loaded. You will find it on one of the exchange servers. Are you running standard or Enterprise? If you are running standard, don't create that 5th database if you can avoid it. You will need it one day to help move things around. How are you doing on drive free space whether already attached or what could be attached?

If you did an offline defrag successfully and you did not clear much space then you probably do not have much whitespace. You need to run the powershell to check.

 

[Mercutio]

Can I just say how much I love it when important functionality is buried in cmdlets but not exposed through management tools? Storage Spaces have the same issue.

 

[ddrueding]

There isn't time to outsource anything, David. I'm days or at most a week from being truly f*cked. Once this is fixed, then I'll I'll think about longer term solutions. Right now, I'm in emergency mode.

Sorry, should have been more specific. One thing Microsoft is really good at is taking your money and fixing issues with their own products. Not sure whether your company has a larger service agreement with MS, but even the pay-per-issue support is excellent. As you mentioned in the OP, this isn't supposed to be your issue. Minimize your exposure and time commitments to things that you aren't supposed to be responsible for.

 

[CougTek]

You are probably running "powershell" not "Exchange management shell" which has the exchange powershell modules pre-loaded.

Ah, I should have thought about it. The command worked, but I only have 88MB free on a 930GB database. I guess I'll have to move stuff to another database like I thought I would do.

I'm more or less ok with the disk free space. By moving accounts to another database and removing the old one once done with the transfers, I should be fine. It will just take a while.

 

[Howell]

And now move to a cloud solution as soon as you can.

 

[CougTek]

I just want to let you know that I've managed to split the database into three smaller ones. The largest database is ~370GB, while the other two are ~330GB and 130GB. I still have 28 accounts to remove which belong to former employees. Overall, I've cleaned more than 50 useless accounts (former employees that no one bothered to ask us to remove). The databases take some 830GB together and the data partition on the Exchange server is 1.45TB, so everything's fine.

Saving the data from several disabled accounts was a bit complicated, but once I've figured out how, using the Exchange Powershell, it was more time-consuming than anything else.

Counting last week-end plus this one, it took some 20 hours of work to fix. I also have at least two hours of documentation to produce related to that.

The worst part is that I'm the only one within the entire company who knows how important this was and my only congratulation will probably be the HR director telling me that I did too many hours last week.

 

[Stereodude]

Well, you could have let it all go up in flames and then gotten major kudos for putting out the proverbial fire so quickly.

 

[ddrueding]

Well, you could have let it all go up in flames and then gotten major kudos for putting out the proverbial fire so quickly.

This, but make sure you send an e-mail in advance warning of the problem and requesting the resources. When the resources are denied (notice I didn't say "if"), then you will be appreciated after fixing it while having your ass covered. Why do you think all the US intel agencies post so many warnings of every type of attack they can think of?

 

[CougTek]

Well, you could have let it all go up in flames and then gotten major kudos for putting out the proverbial fire so quickly.

The thing is, I don't see a way I could have fixed this quickly. Even when you perfectly know what to do in order to fix a morbidly obese exchange database, fixing it isn't quick. It involves transferring hundred of gigabytes of data and that takes a lot of time. That and also that it would have most likely bust during work hours, when most e-mails come in. So as soon as I would have started to free space by moving stuff, new messages would arrive to complicate the database fix.

I figured that fixing it after it would have jammed would have been much more complicated than being proactive about it. There are still plenty of "I told you so" opportunities on this screwed up network. I'll choose one that I'll be more at ease to fix than a problem with a system I'm only partly familiar with (Exchange).

On the positive side, I'm now much more competent with Exchange's administration than I was just two weeks ago. I wouldn't write a book about it, but I can safely write on my C.V. that I have administrative skills on Exchange Server.

 

[Howell]

Note: for systems without mailbox resiliency the maximum recommended mailbox database size is 200G. This helps limit loss and recovery time for a failed database. Since a recovery/maintenance DB is very helpful, this limits the maximum amount of mailbox data a single exchange standard server can hold.

 

[Howell]

There is often an inverse relationship between how stressful and complex an emergency resolution is, and the recognition that comes from fixing it.
If you value a proactive and less stressful environment, you must work for people who also value this and will work with you to prevent it becoming a crisis.

 

[CougTek]

If you value a proactive and less stressful environment, you must work for people who also value this and will work with you to prevent it becoming a crisis.

Time for a job change!

 

[Howell]

Time for a job change!

I was thinking the same thing for you.

 

[timwhit]

Coug, can you now think about moving to Office 365?

 

[Mercutio]

I'm not aware of an Office 365 plan with a single mailbox capacity limit of more than 50GB. Coug's users say they're too busy to deal with mailbox management as it is. Even if that works OK for the time being, how long would it be before one of his users hit that particular brick wall?

 

[ddrueding]

Google Apps Unlimited supports unlimited mailbox size. And (thanks to the web interface) this won't cause performance issues.

 

[CougTek]

They won't pay for anything. Now that the system is somewhat fixed for the foreseeable future (next year or so), they won't spend a dime on it.

Hopefully, it will be someone else's problem by then.

 

[CougTek]

I just want to let you know that I've managed to split the database into three smaller ones. The largest database is ~370GB, while the other two are ~330GB and 130GB. I still have 28 accounts to remove which belong to former employees. Overall, I've cleaned more than 50 useless accounts (former employees that no one bothered to ask us to remove). The databases take some 830GB together and the data partition on the Exchange server is 1.45TB, so everything's fine.

I've further trimmed down the databases and once defragmented, they now occupy 319GB, 315GB and 88GB respectively. Two are still over the 200GB-250GB recommended maximum size, but not as much as it would worry me. I now have a little over half of the data drive in free space.

In the process, I've also eliminated over 130 ghost user accounts (accounts of former employees). The management just didn't seem to care enough to tell the IT staff when they parted ways with a former employee. That company will implode when I'll leave or it will at least cost them a fortune in external IT support.

 

[ddrueding]

I've given management the choice, and they've decided to keep former employee accounts open just for easy access. The $50/user/year is nothing to maintain a full searchable record of all their communications. Protocol is that password is changed and given to former supervisor, and all new messages are forwarded to supervisor or replacement.

 

[CougTek]

In our case, an Exchange Server CAL cost more than 50$, plus the antivirus license and the space it takes... It's preferable to save the former employee's email into a .pst file and put it in a secured folder on slow storage.

 

[Howell]

On O365 the mailbox can be converted to a shared mailbox with no direct login account and cost nothing.

 

[LunarMist]

I've given management the choice, and they've decided to keep former employee accounts open just for easy access. The $50/user/year is nothing to maintain a full searchable record of all their communications. Protocol is that password is changed and given to former supervisor, and all new messages are forwarded to supervisor or replacement.

We are not allowed to use the Giggle for anything. I suppose it depends on your confidentiality/security needs and jurisdiction.

 

[ddrueding]

Nothing is entirely safe. Larger organizations that can devote enough skilled staff to maintain their own e-mail/spam/AV/calendar/etc system in a secure way can have a system more secure than trusting it with Google. Most companies running Exchange do not fall into this category.

 

[LunarMist]

Actual security isn't necessarily better. Global legal and regulatory requirements are really complicated. Apparently the Google t&c are unacceptable to some. Basically The Google has FU attitude.

 

[ddrueding]

Actual security isn't necessarily better. Global legal and regulatory requirements are really complicated. Apparently the Google t&c are unacceptable to some. Basically The Google has FU attitude.

I suspect the opposite is true. Google has enough techs, lawyers, and control over their entire environment to maintain a list of the standards that they comply with. Their exposure is also high enough that I'd bet their system is configured properly and actually complies with the standards (unlike many of the in-house e-mail systems I've migrated from). Google even runs a special product specifically for government agencies.

 

[LunarMist]

I suspect the opposite is true. Google has enough techs, lawyers, and control over their entire environment to maintain a list of the standards that they comply with. Their exposure is also high enough that I'd bet their system is configured properly and actually complies with the standards (unlike many of the in-house e-mail systems I've migrated from). Google even runs a special product specifically for government agencies.

Yeah, I don't know the deal. Maybe some Asian countries don't like the Google.
Every time I see the part about being subject to disciplinary actions, including termination, blah, blah, blah, I just try to avoid it.
Years ago there was some numb nuts that set up a SharePoint website once and got the bot.
There is about a 25-page SOP on rules for the socialism medias. Of course I just avoid the Twitters, Facebook, U-tubes, etc. entirely.

 

[Chewy509]

Yeah, I don't know the deal. Maybe some Asian countries don't like the Google.

I think it's related to Google being a US corporation, working on the basis of US laws.

While using Google infrastructure in the US, as a US citizen/corporation, things are fairly easy in regards to both federal and state laws. But once you remove one of the entities from being on US soil, things get complicated very, very quickly.

I tend to see more companies using Microsoft (in the form of Office365 with managed exchange and AD federation) here than Google, but that's partly related to MS having some data-centres maintained here and options to ensure that your data whilst in the cloud stays in Australia. I'm unsure if Google offer the same here or anyway else in the world.

I can't see a lot of companies who are happy to see their data shipped overseas to be governed by not only local laws, but also the laws of where the data is residing. (And putting on a tinfoil hat, I'm sure a lot of companies would be happy for the US government to have easy access to your data).

 

[CougTek]

We are not allowed to use the Giggle for anything.

It must be a very sad work place.

 

[Howell]

It must be a very sad work place.

It is a mirth-free workplace.