Friday, August 27, 2010

Can't move active mailbox database copy (failed content index catalog)

I've noticed the RTM version of Exchange 2010 suffers from corrupt search catalogs every now and then. You'll notice this by Event ID #123 in the Application log of the server hosting a 'passive' copy of a database. This can be further validated by running:

Get-MailboxDatabaseCopyStatus -server "SERVERNAME"

From the output of the above command you will notice the ContentIndexState shows "Failed" for the database copy. The end result here is that you can't mount the database unless you update the search catalog from another healthy copy in the DAG. If you try to activate the copy you will see an error such as:
"An Active Manager operation failed. Error: The database action failed. Error: An error occurred while trying to validate the specified database copy for possible activation. Error: Database copy 'DB01' on server 'Server01.contoso.com' has content index catalog files in the following state: 'Failed'.

To resolve this, run the following PowerShell command and be sure to specify the database name and servername hosting the failed copy:

Update-MailboxDatabaseCopy "DATABASE"\"SERVERNAME" -CatalogOnly

To confirm the catalog index has been fixed, re-run the Get-MailboxDatabaseCopyStatus command again. Notice the state is now "Healthy". Try the Move-ActiveMailboxDatabase command again and it should work.

What if I only have one copy and I need to activate it?
If this is the case, you can issue the following command to move the database copy and make it active without validating the content index:

Move-ActiveMailboxDatabase "DATABASE" -ActivateOnServer "SERVERNAME" -SkipClientExperienceChecks

I encountered the above situation when performing a DR datacenter switchover at a customer site and this resolved not being able to mount the database.

Cheers!

Monday, August 23, 2010

Event ID: 36885 and the Trusted Root Certificates (OCS2007/CS2010)

Some of you may have needed (or wanted) to install the latest Root Certificate Update from May 2010 on your OCS Edge server and noticed afterward you can't communicate with the outside world or you've seen intermittent issues with connectivity (Audio/Video/Desktop Sharing/Live Meeting/etc).

May 2010 Root Certificate Updates: http://www.microsoft.com/downloads/details.aspx?familyid=E4F9B573-66D7-4DDA-95D5-26C7D0F6C652&displaylang=en

The latest update adds quite a few new issuers to your local trusted certificate store of the OS. This causes issues with applications like OCS because there appears to be a limit with the number of certificates sent by the server. The remaining list is truncated and if your issuer is on the remainder, you get no connectivity, or in some cases, connectivity with some partners and none with others.

Am I affected?
Enable logging for 'schannel' events as follows:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders\Schannel


Value name: EventLogging
Value type: REG_DWORD
Value data: 0x3

Look for EventID: 36885 in Event Viewer. The description should read as follows:
"When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted."
How do I fix it?
There are two ways to resolve this issue:

Method 1: (recommended)
  • Click Start, click Run, type regedit, and then click OK.
  • Locate and then click the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL
  • On the Edit menu, point to New, and then click DWORD Value. Type SendTrustedIssuerList, and then press ENTER to name the registry entry.
  • Right-click SendTrustedIssuerList, and then click Modify.
  • In the Value data box, type 0 if that value is not already displayed, and then click OK.
  • Exit Registry Editor
Method 2:
  • Click Start, click Run, type mmc, and then click OK.
  • On the File menu, click Add/Remove Snap-in, and then click Add.
  • In the Add Standalone Snap-in dialog box, click Certificates, and then click Add.
  • Click Computer account, click Next, and then click Finish.
  • Click Close, and then click OK.
  • Under Console Root in the Microsoft Management Console (MMC) snap-in, expand Certificates (Local Computer), expand Trusted Root Certification Authorities, and then click Certificates.
  • Remove trusted root certificates that you do not have to have. To do this, right-click a certificate, click Delete, and then click Yes to confirm the removal of the certificate.
NOTE: Use caution when removing certificates here. Some certificates are required by Windows. Use the above recommended method if you don't know what to remove. Be careful!

Microsoft Reference: http://support.microsoft.com/kb/933430

Monday, August 16, 2010

Error occurred in the step. Approving object (Exchange 2010 RTM)

I recently came across a scenario where I needed to bulk import PST's into various user's mailboxes. Since the client wasn't running Exchange 2010 SP1 I had to run through the typical hoops of installing Outlook 2010 on one of the servers.

I was able to import PST's into most of the user's mailboxes except a few. The error was:

"An error occurred in the step. Approving object".

This error coincided with the "Microsoft Exchange Mailbox Replication" service crashing over and over again. I tried one of the two paths below to resolve it however in some cases the error came back.

First, I found out that if the active copy of the database wasn't on the server that had Outlook 2010 installed it didn't work. Quickly moving the active copy to the server hosting Outlook resolved the issue.

Secondly I ran the command "FIXMAPI" which resolved it in one case too.

Sadly this didn't resolve the issue for many of my customer's PST files. A call to Microsoft revealed a private hotfix which was installed on one of the Exchange 2010 RTM servers. The mailbox import command was issued again but this time with the "-MRSServer" switch which pointed at the server which had the hotfix installed. This prevented the MRS service from crashing on the import but resulted in a failure to import the data.

Even with SP1 on the Exchange 2010 servers we've encountered many issues with PST files. Here are some examples of the methods used:

  1. Tried native "Import-Mailbox" cmdlet.
  2. Ran a "SCANPST.EXE" to repair any corruption and try step 1 again.
  3. Use the "Import-Mailbox" cmdlet and use the "-MRSServer" switch to point it at a CAS server with the mailbox server role which is also hosting an active copy of the mailbox and try again.
  4. Run the "FIXMAPI" command at the command prompt on the CAS/MB server hosting the active copy of the mailbox and try again.
  5. If all the above fails, try to open the PST in Outlook 2003 or 2010 and grant yourself full access to the mailbox. Importing the PST file into the mailbox this way can often result in success.
  6. We used Lucid8's "DigiScope" product if step 5 didn't work.
  7. If all else failed we gave up on the effort and informed the user their data wasn't importable.

Cheers!

Wednesday, August 4, 2010

Unable to set AlternateWitnessServer to $null

I was working on a project for a client recently where I wanted to use the two Exchange 2010 values called "AlternateWitnessServer" and "AlternateWitnessDirectory" to be a file share and server in a DR facility. I later found out that these settings aren't supported in the RTM version of Exchange 2010 even though they can be populated.

So being the tidy person I am I tried to remove them, or rather set them to a "$null" value as follows:

Set-DatabaseAvailabilityGroup -Identity -AlternateWitnessServer $null -AlternateWitnessDirectory $null

I was met wtih an error message which I don't have at the moment. I later found out that this is a bug in both the RTM and SP1 builds of Exchange 2010. The bug has been pushed off to a post SP1 fix sometime later this year.

If you happen to set these values by accident, the guidance in the mean time is to make them equal to the primary FSW. In other words, make these values the same as the regular WitnessServer and WitnessDirectory entries.

In Exchange 2010 SP1 you can set these values but only when the DAG is configured in DAC mode.
Cheers!