Friday, December 17, 2010

Lync 2010: The following UM IP gateways did not respond as expected to a SIP OPTIONS request.

I recently set up Lync 2010 at our offices and ran into a snag with Exchange 2010 SP1 UM voicemail integration. The symptom was the following error in Exchange 2010:


The following UM IP gateways did not respond as expected to a SIP OPTIONS request. 

...along with the Lync client not being able to call voicemail or users being able to leave a message for the Lync user.

Turns out the UM IP gateway in Exchange 2010 which is set up by running the ExchUCUtil.ps1 script doesn't have a port listed in the configuration. This can be viewed by running: Get-UMIPGateway | fl

To resolve it, run Set-UMIPGateway -Port 5061


Cheers!

Thursday, November 25, 2010

Junk Mail settings in Exchange 2010?

In a recent project I was facing a difficult challenge of trying to sort out how Exchange 2010 handles junk/spam mail. Previous versions exposed GUI features to fine tune "spam confidence levels" (SCL) to avoid issues with false positives, or too much real junk.

In Exchange 2010 this appears to be handled mostly at the Edge server level: http://technet.microsoft.com/en-us/library/bb123559.aspx

There is one setting however in the  "Organizational Configuration" which needs to be tweaked particularly if you're using a hosted spam filtering solution like many of us do today.

To change the SCL level in the organization use PowerShell:


Set-OrganizationConfig -SCLJunkThreshold n

The default is a level of 4. Valid ranges are from 0 to 9. The lower the number the more likely you'll get false positives. So if you're finding legitimate mail ending up in the Junk Mail folder with Exchange 2010, try to increase the value to something like 7 or 8.

Cheers.

Wednesday, October 27, 2010

Outlook 2003 clients see folders at root of mailbox with Exchange 2010

I recently found an interesting but equally strange issue at a client site where we did an Exchange 2010 deployment. The first part of the user migration involved people who all had cached mode turned on for Outlook 2003. The second, and much larger portion of the project involved migrating users who didn't use cached mode (a.k.a online mode).

After migrating a few users over in a pre-pilot, we noticed some people complain about folders appearing at the root of their mailbox folder structure. When viewed in OWA, they weren't there. The project had undergone a Symantec Enterprise Vault 'un-vaulting' process so we quickly assumed it had something to do with that task which caused this abnormality. To make things more troublesome we noticed that if you closed Outlook and re-opened it, a different set of folders would appear, sometimes a larger or smaller set.

After doing some research internally at Microsoft I found several cases of users reporting this issue with a resolution stating it was fixed in SP1 for Exchange 2010. So here is my brief synopsis:


SYMPTOM:
Outlook 2003 users in non-cached mode (a.k.a. “Online mode”) show nested subfolders in the root of the user’s mailbox. Closing an re-opening Outlook produces mixed results with some cases the view having a different set of folders showing up. This appears to not affect Outlook 2003 clients in cached mode, 2007/2010 clients, and OWA users.

CAUSE:
An issue exists with Exchange 2010 RTM (pre-SP1).

RESOLUTION:
Resolved in Service Pack 1 for Exchange 2010.


Cheers!

Monday, September 13, 2010

Sunday, September 12, 2010

Microsoft "Lync 2010" and "Lync Server 2010"?

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=1772A5AD-9688-4861-8387-EC30411BF455

Now when you read the above link information it's clear that a product called "Microsoft Lync Server 2010" is on it's way. What could this be? I'll give you a hint....check the Microsoft web site tomorrow (September 13th, 2010).

UPDATE 1: If you type in www.microsoft.com/lync it points you to a web page (but with an error): http://www.microsoft.com/en-us/lync/default.aspx

UPDATE 2: It's official! Microsoft Lync 2010 is now on the web site http://www.microsoft.com/lync

Cheers!

Saturday, September 4, 2010

Outlook profiles don't update when you move a mailbox (Exchange 2010)

So consider this....

You're performing a migration from a previous version of Exchange Server to the new 2010 platform. You move a mailbox, the person opens Outlook and poof......the profile gets updated. Now let's say the server you've just migrated the account to is an Exchange 2010 all-in-one (hub/mb/cas) server and you move the mailbox to another all-in-one server in a different site. No problem right? Well in the words of a well known Kazakhstan wiseman.......not so much!

Here's the problem. Your Outlook clients will NEVER update their profiles again as long as they can reach the Exchange 2010 CAS role they were moved to originally. This applies to all current Outlook clients including Outlook 2010!

So here's my story:
We were working on a client project where we completed a cross-forest migration from Exchange 2003 to 2010. We built a 3 server all-in-one Exchange 2010 in Canada which would eventually be the final location of the USA mailboxes. To make the move from the USA to Canada possible we set up a local Exchange 2010 server first. This allowed us to perform a cross-forest move locally first, then an online mailbox move over time. The online mailbox move feature in Exchange 2010 worked great. We used the "-SuspendWhenReadyToComplete" switch so we wouldn't interfere with end user connections and resumed the move request at night which flipped the mailbox over. Well we didn't get far before someone noticed Outlook wasn't pointed at the new environment in Canada.

Prior to the migration of any data, we set up our CAS Array in Canada and made sure the RpcClientAccessServer property of each database in the DAG pointed to the new FQDN we created. One would think the Outlook client would check this attribute on first connect to see which CAS server they should connect to but it does not.

Here are a couple ideas to resolve this issue which failed:

1. Running a script to update the Outlook profile using a ".prf" file. Unfortunately this causes the Outlook client running in cached mode to re-cache their mailbox. For an organization with slow WAN links and large mailboxes this can be disastrous. No dice.

2. Create a host entry on the servers in Canada pointing them to the IP of the USA server then updating the DNS record used by clients to have them point to the CAS array. This was less desirable due to the 'jiggery pokery' of meddling with DNS. It didn't work anyway...

3. Disable MAPI clients for a user using "Set-CasMailbox -identity -MAPIEnabled:$false". Have the user launch Outlook. Enable MAPI. Done! That worked.....but impossible to do for several hundred users individually.

I contacted a few Microsoft Exchange team individuals and quickly found out there was only one way to fix this. For Outlook 2003 clients you need to update the profile manually or run a PRF file to update it (causing a re-sync of the OST). For Outlook 2007/2010 clients, a "Repair" of the profile will cause it to wake up and talk to the right server.

So here we have it. Not the greatest solution. We currently have a high severity case open with Microsoft on the issue (as do others). I'll update my post as I know more.

Crap.

Wednesday, September 1, 2010

Communications Server 14 Voodoo (Aastra phones)

I come across these interesting tidbits which I like to share with the 800 or so visitors I get weekly.

Aastra sent me a pair of IP phones to help with a customer demo and I've had a few issues setting them up. I suppose it helps if you read the #$@#$ manual but working with beta software/hardware doesn't always yield the same easy to read, or readily available docs.



Anyway, here are a few tidbits to help you along the way:

DHCPUtil.exe: This utility will help you configure your DHCP options for the Aries (Wave 14) IP phones. You need to configure several new DHCP options in order for the phone to work properly. You will find this utility in the \support folder of your W14 media. Using the following syntax, the utility will configure the options for you (highly recommended): DHCPUtil.exe -sipserver -webserver -RunConfigScript

NOTE: You need to run the above utility on your Windows DHCP server. If it's not an x64 OS, you need to install the "vcredist_x86" from the W14 media.

How to do a hard reset: Hold down the "#" + "4" + backspace keys and plug in the phone. Keep them held down until you see a screen asking if you want to reset the phone.

How to test the phone's bootstrap process: On your Communications Server front-end server, open PowerShell and run: "Test-CsPhoneBootStrap -PhoneOrExt -PIN .

Cheers.

Friday, August 27, 2010

Can't move active mailbox database copy (failed content index catalog)

I've noticed the RTM version of Exchange 2010 suffers from corrupt search catalogs every now and then. You'll notice this by Event ID #123 in the Application log of the server hosting a 'passive' copy of a database. This can be further validated by running:

Get-MailboxDatabaseCopyStatus -server "SERVERNAME"

From the output of the above command you will notice the ContentIndexState shows "Failed" for the database copy. The end result here is that you can't mount the database unless you update the search catalog from another healthy copy in the DAG. If you try to activate the copy you will see an error such as:
"An Active Manager operation failed. Error: The database action failed. Error: An error occurred while trying to validate the specified database copy for possible activation. Error: Database copy 'DB01' on server 'Server01.contoso.com' has content index catalog files in the following state: 'Failed'.

To resolve this, run the following PowerShell command and be sure to specify the database name and servername hosting the failed copy:

Update-MailboxDatabaseCopy "DATABASE"\"SERVERNAME" -CatalogOnly

To confirm the catalog index has been fixed, re-run the Get-MailboxDatabaseCopyStatus command again. Notice the state is now "Healthy". Try the Move-ActiveMailboxDatabase command again and it should work.

What if I only have one copy and I need to activate it?
If this is the case, you can issue the following command to move the database copy and make it active without validating the content index:

Move-ActiveMailboxDatabase "DATABASE" -ActivateOnServer "SERVERNAME" -SkipClientExperienceChecks

I encountered the above situation when performing a DR datacenter switchover at a customer site and this resolved not being able to mount the database.

Cheers!

Monday, August 23, 2010

Event ID: 36885 and the Trusted Root Certificates (OCS2007/CS2010)

Some of you may have needed (or wanted) to install the latest Root Certificate Update from May 2010 on your OCS Edge server and noticed afterward you can't communicate with the outside world or you've seen intermittent issues with connectivity (Audio/Video/Desktop Sharing/Live Meeting/etc).

May 2010 Root Certificate Updates: http://www.microsoft.com/downloads/details.aspx?familyid=E4F9B573-66D7-4DDA-95D5-26C7D0F6C652&displaylang=en

The latest update adds quite a few new issuers to your local trusted certificate store of the OS. This causes issues with applications like OCS because there appears to be a limit with the number of certificates sent by the server. The remaining list is truncated and if your issuer is on the remainder, you get no connectivity, or in some cases, connectivity with some partners and none with others.

Am I affected?
Enable logging for 'schannel' events as follows:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders\Schannel


Value name: EventLogging
Value type: REG_DWORD
Value data: 0x3

Look for EventID: 36885 in Event Viewer. The description should read as follows:
"When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted."
How do I fix it?
There are two ways to resolve this issue:

Method 1: (recommended)
  • Click Start, click Run, type regedit, and then click OK.
  • Locate and then click the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL
  • On the Edit menu, point to New, and then click DWORD Value. Type SendTrustedIssuerList, and then press ENTER to name the registry entry.
  • Right-click SendTrustedIssuerList, and then click Modify.
  • In the Value data box, type 0 if that value is not already displayed, and then click OK.
  • Exit Registry Editor
Method 2:
  • Click Start, click Run, type mmc, and then click OK.
  • On the File menu, click Add/Remove Snap-in, and then click Add.
  • In the Add Standalone Snap-in dialog box, click Certificates, and then click Add.
  • Click Computer account, click Next, and then click Finish.
  • Click Close, and then click OK.
  • Under Console Root in the Microsoft Management Console (MMC) snap-in, expand Certificates (Local Computer), expand Trusted Root Certification Authorities, and then click Certificates.
  • Remove trusted root certificates that you do not have to have. To do this, right-click a certificate, click Delete, and then click Yes to confirm the removal of the certificate.
NOTE: Use caution when removing certificates here. Some certificates are required by Windows. Use the above recommended method if you don't know what to remove. Be careful!

Microsoft Reference: http://support.microsoft.com/kb/933430

Monday, August 16, 2010

Error occurred in the step. Approving object (Exchange 2010 RTM)

I recently came across a scenario where I needed to bulk import PST's into various user's mailboxes. Since the client wasn't running Exchange 2010 SP1 I had to run through the typical hoops of installing Outlook 2010 on one of the servers.

I was able to import PST's into most of the user's mailboxes except a few. The error was:

"An error occurred in the step. Approving object".

This error coincided with the "Microsoft Exchange Mailbox Replication" service crashing over and over again. I tried one of the two paths below to resolve it however in some cases the error came back.

First, I found out that if the active copy of the database wasn't on the server that had Outlook 2010 installed it didn't work. Quickly moving the active copy to the server hosting Outlook resolved the issue.

Secondly I ran the command "FIXMAPI" which resolved it in one case too.

Sadly this didn't resolve the issue for many of my customer's PST files. A call to Microsoft revealed a private hotfix which was installed on one of the Exchange 2010 RTM servers. The mailbox import command was issued again but this time with the "-MRSServer" switch which pointed at the server which had the hotfix installed. This prevented the MRS service from crashing on the import but resulted in a failure to import the data.

Even with SP1 on the Exchange 2010 servers we've encountered many issues with PST files. Here are some examples of the methods used:

  1. Tried native "Import-Mailbox" cmdlet.
  2. Ran a "SCANPST.EXE" to repair any corruption and try step 1 again.
  3. Use the "Import-Mailbox" cmdlet and use the "-MRSServer" switch to point it at a CAS server with the mailbox server role which is also hosting an active copy of the mailbox and try again.
  4. Run the "FIXMAPI" command at the command prompt on the CAS/MB server hosting the active copy of the mailbox and try again.
  5. If all the above fails, try to open the PST in Outlook 2003 or 2010 and grant yourself full access to the mailbox. Importing the PST file into the mailbox this way can often result in success.
  6. We used Lucid8's "DigiScope" product if step 5 didn't work.
  7. If all else failed we gave up on the effort and informed the user their data wasn't importable.

Cheers!

Wednesday, August 4, 2010

Unable to set AlternateWitnessServer to $null

I was working on a project for a client recently where I wanted to use the two Exchange 2010 values called "AlternateWitnessServer" and "AlternateWitnessDirectory" to be a file share and server in a DR facility. I later found out that these settings aren't supported in the RTM version of Exchange 2010 even though they can be populated.

So being the tidy person I am I tried to remove them, or rather set them to a "$null" value as follows:

Set-DatabaseAvailabilityGroup -Identity -AlternateWitnessServer $null -AlternateWitnessDirectory $null

I was met wtih an error message which I don't have at the moment. I later found out that this is a bug in both the RTM and SP1 builds of Exchange 2010. The bug has been pushed off to a post SP1 fix sometime later this year.

If you happen to set these values by accident, the guidance in the mean time is to make them equal to the primary FSW. In other words, make these values the same as the regular WitnessServer and WitnessDirectory entries.

In Exchange 2010 SP1 you can set these values but only when the DAG is configured in DAC mode.
Cheers!

Thursday, July 22, 2010

Unable to save all attachments in Outlook 2003/2007 when using Exchange 2010

Just a quick update here. If you're using Outlook 2003 or 2007 with Exchange 2010 you may have noticed you don't have the ability to save all attachments in a mail message. Rumors about the subject seem to indicate a fix for Outlook 2007 is coming in early August 2010 in the form of a rollup or a "CU". For those of you using Outlook 2003.....sorry to say you're out of luck.

A fix for this issue will not be back-ported to Office 2003.

Tuesday, July 20, 2010

An exception occurred and was handled by Exchange ActiveSync (Multi-hop Proxying)

I came across some interesting yet conflicting information about how to set up a multi-site Exchange 2010 environment with an HQ location and branch office. In my scenario I have a branch office server hosting CAS/HT/MB roles in Exchange 2010. My HQ office has a multi-site DAG with 3 Exchange 2010 servers also hosting CAS/HT/MB roles.

In the HQ site I have a CAS Array established 'outlook.contoso.com' where I point all my autodiscover services  for each server in the organization.....or so I thought. As it turns out, the Microsoft article (http://technet.microsoft.com/en-us/library/bb310763.aspx) does a great job of outlining not only the settings you need for non-internet facing CAS servers, but HOW it all works. I had been given advice from a reputable source who indicated the non-internet facing CAS server (branch server) needed to have the Exchange ActiveSync virtual directory set to the CAS Array fqdn and not the server name. This turned out to be incorrect and resulted in the following error:


Clients were unable to sync their devices after making the change. It took a while to figure out but I did have to re-trace my steps on what changes I had made.

So, just remember, for your non-internet facing CAS servers, keep the virtual directory URL's set to the server's fqdn.

For reference, this can be changed using PowerShell as follows:

Set-ActiveSyncVirtualDirectory -Identity "name" -InternalURL https://.contoso.com/Microsoft-Server-ActiveSync

Cheers!

Saturday, May 22, 2010

Why I hate the Google Nexus One

I suppose it's my own fault. I was crossing the river on my quad and had my iPhone in my pocket when i broke through the ice, flipped onto my back, and sunk up to my chest in freezing cold water. My iPhone was in the pocket of my waterproof pants which unfortunately had a sizable hole permitting the icy cold Athabasca water to penetrate and destroy the one techno-toy I can't live without.

After a few days of trying to dry out the phone and seeing nasty water marks under the screen I was finally able to boot it and get to the home screen. I quickly found out everything worked! Well except for the home button. Now if you think about it, the home button holds a significant place on the iPhone; try living without it!

So off I went in search of a new phone. The smartphone market in Canada is rather bland. We often have to wait months and sometimes longer (in the case of the iPhone) to get what the USA can. I read Engadget from time to time and see all sorts of wonderful prototypes and interesting phones we can't get here. After reading a few reviews of the Android OS and stopping by the Google site I noticed that day that the Nexus One had been released for customers on the Rogers network here in Canada. Woooohoooo! So on a Wednesday afternoon I stepped through the quick order process and even tagged the back with my name and company to see how the quality was. I did read up on the hardware specifications and support for ActiveSync which appeared to be fully supported. By late late Friday I received the phone.

Nice packaging. They've obviously modeled the hardware and packaging after the iPhone. Copy cats.

I charged the phone and quickly began to configure my Google account. The phone locked up solid after 10 minutes. Shit. What the? Okay, take out the battery. Boot the phone again. Everything is good for another 30 minutes until I start to configure ActiveSync. Now I've never seen this done before but Google seems to have forgotten to finish what they started when it comes to support for ActiveSync on the Nexus One. No Calendar or Global Address List support. Are you kidding me? $550+ bones for a phone that pushes email and my personal contacts? No way. I've just been had!

Okay, maybe I'm demanding too much from Google here but I hardly think so. This is Android 2.x already. Get with it. I have to say I was seriously pissed off. To be a corporate mail user and not have access to my Calendar and GAL? I'm lost without it. The 3rd party add on applications you have to buy to get full ActiveSync support are garbage too. I've tried the Google Calendar Sync application which by the way doesn't work with Outlook 2010....unless you have a hacked EXE copy....which I do....and it still sucks. I only get syncing for new calendar items. I'm missing meetings all the time. I can't view, create or accept meeting invitations at all.

So what about the rest of the phone? Well the screen is terrible viewing it in sunlight. When you press an image on the screen like a letter on the soft keyboard or an icon, it's almost as if the image on the screen should be higher up. If they literally shifted up the touch sensor or shifted down the image on the screen, things would line up better. I hate typing on it. The iPhone uses a predictive typing and touch proximity adjustment algorithm to help you NOT press the wrong key when spelling a word. It actually makes the sensor area around english spelling characters bigger so you have a better chance of hitting the right key all on the fly as you type. The Nexus One does not. I'd have to say this single feature is almost as much of a killer for me as is the ActiveSync half-support rubbish.

I own a new Ford vehicle with the Microsoft Sync system which works pretty good most of the time. I was able to sync my wife's iPhone and my own just fine. There are the odd scenarios in which the software on either side has a few bugs but nothing too bad. With the Google Nexus One, if it connects via Bluetooth for hands-free it will magically start to play music. I can see the icon on the phone show a 'play' symbol which is confirmed if I change the input on my stereo to the media feature. I can't understand why my phone thinks I want to play music when I'm not asking it to or when I'm on the phone. This really kills the battery by the way.....more on this later. So I turn off my truck and all of a sudden the Google Nexus One starts blaring music out the external speaker! Then the phone locks up again....while playing music. So I'm now fiddling with buttons and the back cover to take out the battery. I'm just about to chuck it across the parking lot when I realize I have several emails to check up on and phone calls to make. Now as funny as this may seem, it really pisses me off. This is clearly a software issue and should have been well tested and resolved in version 1.0.1 of the platform.

The phone is unresponsive to touch sometimes. The battery is lucky to last an 8 hour day with me placing 2-3 calls and receiving 3-4 calls per day. With the Bluetooth issues mentioned above I almost always turn it off which gives me maybe another hour or two in the day.

Hardware and software issues aside, Google has a lot of catching up to do. Apple has iTunes which offers movie rentals (in Canada too!), music downloads (and yes they have the Dead Milkmen!), and an amazing app store with an interface which offers over the air downloads and previews, etc. The Nexus One has Amazon MP3. I almost laughed myself out of my plane seat when I tried several times to find a few of my favorite albums. No other platform in the world has mastered the integration between online media services and hardware devices like Apple. They've done an amazing job here and all others will be judged in comparison to their success.

So my recent trip to Redmond, WA was fun. I managed to track down an HTC HD2. More on this one later but for now I'm carrying two phones and still feeling like combined they're years behind the iPhone.

Sunday, May 9, 2010

Favorite Exchange 2010 SP1 features - Part 1

There is much to talk about but only so much we can say at this point. Exchange Server 2010 Service Pack 1 adds a few nice to have features people have been asking for. Some are nice to have while others are imperative to operating a large-scale mail environment.

In this post I'll talk about a few of the optimized visual enhancements in Outlook Web App 2010.

THEMES:
In previous versions of OWA you could customize the look slightly by changing the color to a few simple selections. In Exchange 2010 SP1 you have quite a few built-in themes to meet the needs of just about anyone. Here are a few samples:


The themes are selectable through the Options button in the top right corner and don't require switching screens or clicking through multiple menus. The themes apply instantly and change text color along with the top banner image to give Outlook Web App the style you're looking for.

IMPROVED UI:
Outlook Web App SP1 now simplifies the reading pane and action buttons so the user can see more of the content and less wasted white space. The end result is a better viewing experience on size constrained displays such as Netbooks.



PERFORMANCE:
Improvements to performance have been made all over. Moving from message to message, deleting content, viewing calendar data give OWA 2010 SP1 a fresh feel. Official performance improvement numbers haven't been released yet but the general feel is quite positive.

Well that's all I can say for now. Check back again soon for more.

Wednesday, April 21, 2010

DSC_E_NO_SUITABLE_CDC error with Exchange Server 2010 (SACL right)

I'm in the middle of a project at a client site where we've completed both the initial preparation tasks and installation of various server roles for an Exchange 2010 implementation. Somewhere between last Tuesday and Friday our three server Exchange 2010 stopped working. The symptoms included:

  • Booting the server would take anywhere from 30 minutes to 1 hour while waiting for "Applying Computer Settings" to finish.
  • Once the CTRL-ALT-DEL screen was available and you authenticated, it would take anywhere from 30 minutes to 1 hour while waiting for the "Applying User Settings" screen to go away.
  • The Microsoft Exchange Information Store service would be stuck in a "starting" state.
  • The Microsoft Exchange Transport service would also be stuck in a "starting" state.
  • The event viewer would show MAD.EXE with an error code of 0x80040a02 (DSC_E_NO_SUITABLE_CDC).
  • Event Viewer would also show MAD.EXE. All Domain Controllers in use are not responding.
  • Event Viewer would show MSEXCHANGEADTOPOLOGYSERVICE.EXE. Topology discovery failed.

As it turns out the organization may have modified their Default Domain Controllers GPO. The modification was in a setting under Computer Configuration\Policies\Windows Settings\Security Settings\Local Policies\User Rights Assignment. The setting they've changed is "Manage auditing and security log" which contained the global group called "Exchange Enterprise Servers" among others. The key issue here is that it DIDN'T contain the "Exchange Servers" Universal Group.

Over time the DC's will re-apply GPO's and overwrite the security put in place by the setup.com /prepareAD process during setup. I was able to replicate this issue in my lab environment by removing the Universal Group, breaking Exchange 2010, then adding it back to the GPO, and fixing it. The Microsoft TechNet article which discusses the AD preparation tasks http://technet.microsoft.com/en-ca/library/bb125224.aspx shows the process making a change to the "Manage Audit and Security Log" value on the very last bullet on the bottom of the page.

One of the things you can do to determine if you have the same issue is to look for Event ID 2080. This event will show up in a healthy or unhealthy environment and will show you if the SACL right attribute is equal to "1" which is necessary.

The following image shows a "healthy" Event ID 2080 where the SACL right property on the first DC it found is equal to 1. If you count backward from the OS Version, Netlogon, Critical Data attributes (1, 7, 1, then you find the 1 assigned to SACL right).


In an unhealthy situation you will see a zero (0) in the SACL right column.

A word about IPv6 and Exchange Server 2007 and 2010
The biggest red herring I found when troubleshooting this one from articles others had posted was related to IPv6. I see quite a few people suggesting IPv6 is required for Exchange 2007 and 2010. This is NOT true. As a matter of fact, if the server hosting Exchange 2007 or 2010 is a DC, then IPv6 must be enabled otherwise simply uncheck the checkbox in TCP/IP properties on all connected interfaces. You don't need to buggar with the registry to "really disable it"....just uncheck the checkbox.

Cheers.

Thursday, March 25, 2010

Load balancing Exchange 2010 OWA

In a recent post: http://jasonshave.blogspot.com/2010/03/i-been-spending-much-of-my-time-la.html I talked about how to load balance an array of Exchange 2010 CAS servers using a Citrix NetScaler VPX.

In this post I wanted to take a step back and offer a more simple configuration for those of you wishing to load balance basic HTTP/S traffic between your Exchange 2010 CAS servers. One of the primary reasons for this is to provide high availability for internal (or external) clients. The types of connectivity you can expect the NetScaler to see includes both direct and indirect traffic. Direct traffic would be a user opening a web browser and navigating to your load balanced OWA servers. Indirect traffic would be Outlook 2007 or 2010 connecting to the Autodiscover service connection point (SCP).

STEP 1: Setting up HTTP Services
If you followed my previous article on how to set up servers and services, then this should be easy for you. If you haven't, go back and read those steps.

  • Create a service for each server you wish to load balance to. I've given mine a name of "svc_http_jcsexchcal" for my Calgary server and "svc_http_jcsexchedm" for my Edmonton server and so on.
  • The service should have HTTP as the protocol and 80 as the port number.
  • The monitor can be left at TCP for now.
STEP 2: Setting up HTTPS Services
You'll need to perform the same steps for the SSL services with a few differences.
  • When you create the service use the same naming convention but alter it accordingly (i.e. svc_https_jcsexchedm).
  • The service should have a protocol of SSL_BRIDGE and a port of 443.
  • The monitor should have both TCP and HTTPS.

STEP 3: Setting up an HTTP Virtual Server (redirect)
You may be asking yourself at this point why I need HTTP services? You'll need them for your redirect from HTTP to HTTPS. You want to make sure a person is redirected to the SSL site when they type in the base URL.

  • Create your virtual server with the same IP as your CAS array if you wish and give it a proper name such as "vs_http_webmail".
  • Set your protocol to HTTP and port to 80.
  • Add your HTTP services (i.e. svc_http_jcsexchedm and so on)
  • Click on the Advanced tab and type in the full path of OWA into the redirect URL (i.e. https://webmail.lvsedmtest.ca/owa)
  • Click on the Disable button to make sure the vserver is in a "down" state. You want to do this because the redirect won't work otherwise.
STEP 4: Setting up an HTTPS Virtual Server
This is the virtual server actually accepting connections, both direct and indirect.
  • Create another virtual server with the same IP as your CAS array and HTTP vserver.
  • Give it a proper name such as "vs_https_webmail".
  • Set your protocol to SSL_BRIDGE and port to 443.
  • Add your SSL_BRIDGE services to the vserver (i.e. svc_https_jcsexchedm and so on).
  • Click on the Method and Persistence tab and be sure to set the LB Method to Round Robin and your Persistence to SOURCEIP with a timeout of 15min and mask of 255.255.255.255.
At this point you should have your two new services with the vs_http_webmail service in a "out of service" state. This is normal. The effective state will be "down". Your vs_https_webmail service should be "up" and "up" for both state and effective state. Try it out by browsing to your OWA URL and see how things work. Try disabling services, pausing VM's, and disconnecting network cables to simulate various failures. Record your results and convey the expectations to management and your user community. They will ask the questions anway....."what happens if......".


STEP 5: Setting up Exchange 2010
We need to make changes to AD and Exchange 2010 so clients can find your newly created vserver(s). First we need to modify the SCP's for each CAS server and make sure all the internal URL's point to the NetScaler's VIP.

The services we're going to modify include:

  • Autodiscover
  • Exchange Web Services
  • Offline Address Book
  • OWA Virtual Directory
Autodiscover:
  • Run the following on your E2010 server using PowerShell (Get-ClientAccessServer |fl auto*). This will spit out the Service Connection Point (SCP) settings which probably point to the server fqdn.
  • As an example run (Set-ClientAccessServer jcsexchedm -AutoDiscoverServiceInternalUri https://webmail.lvsedmtest.ca/autodiscover/autodiscover.xml). You can run the Get-ClientAccessServer command again to view the changes.
  • Run the above command for each CAS server you have.
Exchange Web Services:
  • Run the following on your E2010 server using PowerShell (Get-WebServicesVirtualDirectory |fl Server, InternalUrl). This will spit out the defined URL for each CAS server.
  • As an example, run (Set-WebServicesVirtualDirectory -identity jcsexchedm\ews* -InternalUrl https://webmail.lvsedmtest.ca/EWS/Exchange.asmx)
  • Repeat for each CAS server.
Offline Address Book:
For this one I just leave things as is. You don't typically need to load balance the OAB. If you run a (Get-OABVirtualDirectory |fl Server, InternalUrl) you'll notice the default is to use HTTP and not HTTPS. I typically leave each server hosting the OAB.

OWA Virtual Directory:
This is key to Outlook Web App when you have an environment with a mixed environment of Exchange 2007 and 2010. These settings help with redirection and proxy connections coming from OWA clients. For example, if you have an Exchange 2007 environment with users homed on it and you've installed your first E2010 CAS server you should be pointing them to the new box. You should also have a legacy URL for your 2007 CAS servers (i.e. legacy.domain.com). Using PowerShell you would modify the InternalUrl parameter to be "https://legacy.domain.com/owa" for each 2007 CAS server so that your E2010 CAS servers know where to send users with a 2007 mailbox to when they authenticate through OWA.
  • Run (Get-OwaVirtualDirectory |fl Server, InternalUrl) to see how things are set up in your environment. Following the same logic here you want to make sure all the URL's for each object on each server point to the same name.
  • You need to switch things up here a bit and add the -identity parameter to the PowerShell command ensuring your modifying the servers you actually want to change. As an example run (Set-OwaVirtualDirectory -identity JCSEXCHEDM\* -InternalUrl https://webmail.lvsedmtest.ca/owa).
  • Repeat for each E2010 CAS server you have.
One thing to note here is that we've only configured the InternalUrl parameter for the various services. Each of them have an ExternalUrl parameter as well with the exception of Autodiscover. If you're using ISA or TMG to front your OWA/EWS/Autodiscover connectivity (which you SHOULD be doing), then be sure to modify the ExternalUrl's as well. Quick example: Set-OwaVirtualDirectory -identity JCSEXCHEDM\* -ExternalUrl https://webmail.lvsedmtest.ca/owa. In my environment the "webmail.lvsedmtest.ca" name resolves to my NetScaler's vserver VIP when I'm inside the network and the same name resolves to my public IP bound to my TMG server. This is called "split horizon DNS" and is essential if you don't want to confuse your users with different URL's for inside vs. outside connectivity.

Anyway, that's enough for now. Enjoy.

Tuesday, March 16, 2010

Can't see call logs or voice mail on Tanjay (CX700) with Exchange 2007 and 2010?

We had an issue with our Exchange 2010 and 2007 environment recently where some users were migrated to our Exchange 2010 servers and activated for Unified Messaging and their call logs and voice mail weren't showing up on the Polycom CX700 (Tanjay) phones.

During our involvement with the Exchange 2010 TAP program we implemented a single all-in-one server. In order to keep everything working and reduce the potential for outages we decided to keep our 'autodiscover.domain.com' record pointing at the existing Exchange 2007 environment. Everything seemed to work fine except the CX700's. It turned out to be resolved by pointing the Autodiscover record at our Exchange 2010 environment which didn't cause an issue for our UM users on 2007 with CX700's either.

This was pretty much our own fault since the deployment guidance from Microsoft suggests migrating to a 2010 CAS server out of the gate. Anyway, live and learn....

Wednesday, March 3, 2010

Load balancing Exchange 2010 CAS servers (CAS Array)

I've been spending much of my time lately working on Exchange 2010 designs for customers here in Canada. One of the common design scenarios we propose is a multi-server DAG with all roles collocated on the same server. When installing multiple Exchange 2010 servers you'll notice that each collocated (MB/HT/CAS) server has a default database. I typically remove the database and create my own with it's own name and file location rather than moving and renaming the default....personal preference. Either way, when you create a database or use the default database on an E2010 MB server there is an attribute called "RPCClientAccessServer" tied to the database which needs some special attention if you intend on load balancing the CAS server role. I'll come back to this in a few moments...

You may know by now that the CAS server in E2010 actually takes on a more critical role than before. In previous versions of Exchange (2007-) the CAS role basically handled IIS (web) traffic for Outlook Web Access users and that's about it. Now with E2010 the CAS role actually handles MAPI/RPC traffic. This means your Outlook 2003/2007/2010 client traffic on the LAN will not connect directly to the database server, but rather the CAS server. Where this becomes more of an impact in deployments is when you want to provide redundancy and DR capabilities to Outlook clients. The clients are configured to "talk" to a specific server initially....and if you have lets say three all-in-one E2010 servers with users in mailboxes on each system, their client settings in Outlook are going to show the specific name of the E2010 server hosting their mailbox. So what if that server goes down? Will the client connect to another server hosting a replica of the database? The answer is no....because you haven't created an RPC Array or set up anything else.

First off if you're wanting to load balance between multiple collocated servers in a Database Availability Group (DAG), you need a hardware (or virtualized) load balancer. Personally I prefer the Citrix NetScaler VPX since it works with VMWare vSphere 4 and the basic model is free and downloadable as a virtual appliance. You can't use NLB since the DAG is using Windows Clustering components and collocation of those technologies isn't supported.

Setting up your hardware load balancer

Let's walk through the setup of the load balancer first. With your hardware load balancer you're going to define a name and IP used by clients to connect to E2010....let's say "webmail.lvsedmtest.ca". This name needs a DNS record on your corporate DNS server and you need to pick an IP address.....let's say 10.10.10.252. Once you've created your DNS host record for the name, we need to configure the load balancer. In my example I'm load balancing to 4 all-in-one servers (see image). Each server and the IP address has been defined in the NetScaler VPX UI.

First, create/define your servers and IP's. These will be later linked to your services which are then bound to the Virtual Server.


Now I need to create a monitor to keep track of the RPC services for each server. Later I'm going to bind this monitor to each RPC service for each server. I use the format of "mon_" for monitor, "rpc_" for the type of monitor, then "cas" for what I'm monitoring. The monitor name is "mon_rpc_cas" and has no specific IP....but rather it has port 135 listed as the port to check to determine if it's operational (up) or not (down).


Now I'm going to create RPC services for each server I need to load balance to. Each service name contains the format of "svc_" then the type "rpc_" then the server name "jcsexchcal" so the entire name looks like "svc_rpc_jcsexchcal". The first service name I've created is linked to the "jcsexchcal.lvsedmtest.local" server (defined earlier) and has both a PING monitor but also the mon_rpc_cas monitor tied to it. This step is critical otherwise your services won't operate in an up/down state properly. You need to repeat this step for each server/service you want to load balance to.


Next I need to create a "Virtual Server" which contains all my services, the IP address, and load balancing attributes (i.e. round robin vs. least connection). I've chosen the name of "vs_" for virtual server, then "rpc_" for the type of data I'm load balancing, then "webmail" for the DNS host name I'm load balancing so the entire name looks like "vs_rpc_webmail". My IP address is 10.10.10.252 which is linked in DNS to "webmail.lvsedmtest.ca". Each service in the UI should show "UP" in the state column by this time by the way! You will want to make sure you click on the "Method and Persistence" tab to set the timeout value to 15min to ensure connections persist with the same CAS server. This will prevent odd re-login issues with OWA.


Great. Now you have a load balanced RPC cluster that can serve up traffic for Outlook clients. Now back to Exchange since we're not quite done there yet.

Setting up the Exchange CAS Array

The CAS Array is a new feature in E2010 which needs some PowerShell hands-on to create and configure. When you define the CAS Array a "site" parameter is specified which is used to determine which CAS servers are a member of the array. You don't actually pick the CAS servers when you create the array. I use the "New-ClientAccessArray" command as follows:


New-ClientAccessArray -fqdn webmail.lvsedmtest.ca -site Default-First-Site-Name -name "CAS Array 1"

At this point if you create any new databases on CAS servers in that site, the "RPCClientAccessServer" attribute will be set to the CAS Array fqdn. I mentioned this at the beginning as an important point because any existing databases you create or databases created during Exchange setup will have the attribute set to the server in which they were created. You will need to change this attribute using the following PowerShell command:

Set-MailboxDatabase database01 -RpcClientAccessServer webmail.lvsedmtest.ca


To view the RPCClientAccessServer attribute currently set on all databases:

Get-MailboxDatabase |fl rpc*,name


This means if you have users within the database you don't have to move them....just update the attribute to be "webmail.lvsedmtest.ca" and their Outlook clients will update too.


Performing a connection status on Outlook will show a TCP/IP connection (not RPC/HTTPS) to the fqdn of your CAS Arrray!



Cheers!